diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000..63825a6ba0 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,2 @@ +*.pdf filter=lfs diff=lfs merge=lfs -text +*.epub filter=lfs diff=lfs merge=lfs -text diff --git a/.nojekyll b/.nojekyll new file mode 100644 index 0000000000..e69de29bb2 diff --git a/README.md b/README.md new file mode 100644 index 0000000000..6f0a0d53cb --- /dev/null +++ b/README.md @@ -0,0 +1,7 @@ +# GitHub Pages Cache + +Nothing to see here. The contents of this branch are essentially a cache that's not intended to be viewed on github.com. + +You can view the actual documentation as it's intended to be viewed at [https://netris.io/docs/](https://netris.io/docs/) + +If you're looking to update our documentation, check the relevant development branch's ['docs'](https://github.com/netrisai/docs). diff --git a/en/2.9/.buildinfo b/en/2.9/.buildinfo new file mode 100644 index 0000000000..a790c9905e --- /dev/null +++ b/en/2.9/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: 87b438ab61e542501fac234e0ef63265 +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/en/2.9/.doctrees/SoftGate-agent-installation.doctree b/en/2.9/.doctrees/SoftGate-agent-installation.doctree new file mode 100644 index 0000000000..1edfeca6ef Binary files /dev/null and b/en/2.9/.doctrees/SoftGate-agent-installation.doctree differ diff --git a/en/2.9/.doctrees/accounts.doctree b/en/2.9/.doctrees/accounts.doctree new file mode 100644 index 0000000000..ac21f15bf0 Binary files /dev/null and b/en/2.9/.doctrees/accounts.doctree differ diff --git a/en/2.9/.doctrees/concepts.doctree b/en/2.9/.doctrees/concepts.doctree new file mode 100644 index 0000000000..252f5854a9 Binary files /dev/null and b/en/2.9/.doctrees/concepts.doctree differ diff --git a/en/2.9/.doctrees/controller-initial-configuration.doctree b/en/2.9/.doctrees/controller-initial-configuration.doctree new file mode 100644 index 0000000000..98c065daf8 Binary files /dev/null and b/en/2.9/.doctrees/controller-initial-configuration.doctree differ diff --git a/en/2.9/.doctrees/controller-installation.doctree b/en/2.9/.doctrees/controller-installation.doctree new file mode 100644 index 0000000000..c50bc2f585 Binary files /dev/null and b/en/2.9/.doctrees/controller-installation.doctree differ diff --git a/en/2.9/.doctrees/controller-k8s-installation.doctree b/en/2.9/.doctrees/controller-k8s-installation.doctree new file mode 100644 index 0000000000..c132014648 Binary files /dev/null and b/en/2.9/.doctrees/controller-k8s-installation.doctree differ diff --git a/en/2.9/.doctrees/environment.pickle b/en/2.9/.doctrees/environment.pickle new file mode 100644 index 0000000000..a9f31ceb20 Binary files /dev/null and b/en/2.9/.doctrees/environment.pickle differ diff --git a/en/2.9/.doctrees/index.doctree b/en/2.9/.doctrees/index.doctree new file mode 100644 index 0000000000..40d3001e9e Binary files /dev/null and b/en/2.9/.doctrees/index.doctree differ diff --git a/en/2.9/.doctrees/network-policies.doctree b/en/2.9/.doctrees/network-policies.doctree new file mode 100644 index 0000000000..e121686e6f Binary files /dev/null and b/en/2.9/.doctrees/network-policies.doctree differ diff --git a/en/2.9/.doctrees/release-notes.doctree b/en/2.9/.doctrees/release-notes.doctree new file mode 100644 index 0000000000..5d91068096 Binary files /dev/null and b/en/2.9/.doctrees/release-notes.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox1/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox1/configurations.doctree new file mode 100644 index 0000000000..6ede946957 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox1/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox1/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox1/creating-services.doctree new file mode 100644 index 0000000000..2382707fce Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox1/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox1/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox1/index.doctree new file mode 100644 index 0000000000..33949d04d1 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox1/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree new file mode 100644 index 0000000000..3b538e7f9f Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox1/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox1/sandbox-info.doctree new file mode 100644 index 0000000000..769440aa1a Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox1/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox10/Configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox10/Configurations.doctree new file mode 100644 index 0000000000..bd9709c48c Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox10/Configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox10/Diagram.doctree b/en/2.9/.doctrees/sandbox/Sandbox10/Diagram.doctree new file mode 100644 index 0000000000..b88957e7b5 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox10/Diagram.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox10/Sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox10/Sandbox-info.doctree new file mode 100644 index 0000000000..a127103f3f Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox10/Sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox10/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox10/creating-services.doctree new file mode 100644 index 0000000000..fb20d3c551 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox10/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox10/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox10/index.doctree new file mode 100644 index 0000000000..6e635f2cfe Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox10/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox2/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox2/configurations.doctree new file mode 100644 index 0000000000..948dbd8db5 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox2/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox2/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox2/creating-services.doctree new file mode 100644 index 0000000000..13569fed31 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox2/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox2/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox2/index.doctree new file mode 100644 index 0000000000..76d642047b Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox2/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree new file mode 100644 index 0000000000..24cbb4c41e Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox2/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox2/sandbox-info.doctree new file mode 100644 index 0000000000..cf7896522b Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox2/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox3/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox3/configurations.doctree new file mode 100644 index 0000000000..cca7e73ab8 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox3/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox3/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox3/creating-services.doctree new file mode 100644 index 0000000000..0d32cb0dad Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox3/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox3/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox3/index.doctree new file mode 100644 index 0000000000..8e12ca4258 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox3/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree new file mode 100644 index 0000000000..48f6004af6 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox3/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox3/sandbox-info.doctree new file mode 100644 index 0000000000..0fb769de8c Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox3/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox4/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox4/configurations.doctree new file mode 100644 index 0000000000..513fc43b17 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox4/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox4/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox4/creating-services.doctree new file mode 100644 index 0000000000..1aa155e6af Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox4/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox4/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox4/index.doctree new file mode 100644 index 0000000000..693e9205c1 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox4/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree new file mode 100644 index 0000000000..f1d4411791 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox4/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox4/sandbox-info.doctree new file mode 100644 index 0000000000..127fa708b3 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox4/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox5/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox5/configurations.doctree new file mode 100644 index 0000000000..adabe190fd Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox5/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox5/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox5/creating-services.doctree new file mode 100644 index 0000000000..cabcb3cd87 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox5/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox5/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox5/index.doctree new file mode 100644 index 0000000000..0e44e8e609 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox5/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree new file mode 100644 index 0000000000..e6d2af6dee Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox5/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox5/sandbox-info.doctree new file mode 100644 index 0000000000..de20c00d83 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox5/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox6/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox6/configurations.doctree new file mode 100644 index 0000000000..62cbf799cb Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox6/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox6/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox6/creating-services.doctree new file mode 100644 index 0000000000..240cf70be4 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox6/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox6/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox6/index.doctree new file mode 100644 index 0000000000..06406a4b94 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox6/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree new file mode 100644 index 0000000000..76d61b8aea Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox6/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox6/sandbox-info.doctree new file mode 100644 index 0000000000..719c2bd2a7 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox6/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox7/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox7/configurations.doctree new file mode 100644 index 0000000000..9c93d277c4 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox7/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox7/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox7/creating-services.doctree new file mode 100644 index 0000000000..224d1fe6a5 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox7/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox7/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox7/index.doctree new file mode 100644 index 0000000000..9a2c33b4e1 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox7/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree new file mode 100644 index 0000000000..b485f04a77 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox7/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox7/sandbox-info.doctree new file mode 100644 index 0000000000..243e1be9b9 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox7/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox8/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox8/configurations.doctree new file mode 100644 index 0000000000..6956db4c95 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox8/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox8/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox8/creating-services.doctree new file mode 100644 index 0000000000..d3c3e73f44 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox8/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox8/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox8/index.doctree new file mode 100644 index 0000000000..10d662010a Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox8/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree new file mode 100644 index 0000000000..c5bb5619ba Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox8/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox8/sandbox-info.doctree new file mode 100644 index 0000000000..bd9684559a Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox8/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox9/configurations.doctree b/en/2.9/.doctrees/sandbox/Sandbox9/configurations.doctree new file mode 100644 index 0000000000..f2d1f5d8e4 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox9/configurations.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox9/creating-services.doctree b/en/2.9/.doctrees/sandbox/Sandbox9/creating-services.doctree new file mode 100644 index 0000000000..829ac941a0 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox9/creating-services.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox9/index.doctree b/en/2.9/.doctrees/sandbox/Sandbox9/index.doctree new file mode 100644 index 0000000000..fd37bdd9a1 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox9/index.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree b/en/2.9/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree new file mode 100644 index 0000000000..e7a919e6b5 Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree differ diff --git a/en/2.9/.doctrees/sandbox/Sandbox9/sandbox-info.doctree b/en/2.9/.doctrees/sandbox/Sandbox9/sandbox-info.doctree new file mode 100644 index 0000000000..c65bee25fd Binary files /dev/null and b/en/2.9/.doctrees/sandbox/Sandbox9/sandbox-info.doctree differ diff --git a/en/2.9/.doctrees/services.doctree b/en/2.9/.doctrees/services.doctree new file mode 100644 index 0000000000..c0c7e58150 Binary files /dev/null and b/en/2.9/.doctrees/services.doctree differ diff --git a/en/2.9/.doctrees/switch-agent-installation.doctree b/en/2.9/.doctrees/switch-agent-installation.doctree new file mode 100644 index 0000000000..78386e60f4 Binary files /dev/null and b/en/2.9/.doctrees/switch-agent-installation.doctree differ diff --git a/en/2.9/.doctrees/visibility.doctree b/en/2.9/.doctrees/visibility.doctree new file mode 100644 index 0000000000..194d9acf1e Binary files /dev/null and b/en/2.9/.doctrees/visibility.doctree differ diff --git a/en/2.9/.nojekyll b/en/2.9/.nojekyll new file mode 100644 index 0000000000..e69de29bb2 diff --git a/en/2.9/SoftGate-agent-installation.html b/en/2.9/SoftGate-agent-installation.html new file mode 100644 index 0000000000..83efb6cd7a --- /dev/null +++ b/en/2.9/SoftGate-agent-installation.html @@ -0,0 +1,581 @@ + + + + + + + + + + + Netris SoftGate agent installation — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris SoftGate agent installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Netris SoftGate agent installation

+
+

Minimal hardware requirements

+
    +
  • 2 x Intel Silver CPU

  • +
  • 96 GB RAM

  • +
  • 300 GB HDD

  • +
  • Nvidia Mellanox Connect-X 5 SmartNIC card

  • +
+
+
+

BIOS configuration

+

The following are some recommendations on BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference:

+
    +
  • Before starting consider resetting all BIOS settings to their defaults.

  • +
  • Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report.

  • +
  • Select Performance as the CPU Power and Performance policy.

  • +
  • Disable Turbo Boost to ensure the performance scaling increases with the number of cores.

  • +
  • Set memory frequency to the highest available number, NOT auto.

  • +
  • Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d.

  • +
  • Disable Hyper-Threading.

  • +
+
+
+

Software installation

+

Requires freshly installed Ubuntu Linux 18.04 and network connectivity with your Netris Controller over the out-of-band management network.

+
    +
  1. Set environment variables to use Netris Controller as a proxy.

  2. +
+
export http_proxy=http://<Your Netris Controller address>:3128 && export https_proxy=http://<Your Netris Controller address>:3128
+
+echo -e 'Acquire::http::Proxy "http://<Your Netris Controller address>:3128";\nAcquire::https::Proxy "http://<Your Netris Controller address>:3128";' | sudo tee -a /etc/apt/apt.conf.d/netris-proxy
+
+
+
    +
  1. Config the apt for Mellanox repository.

  2. +
+
wget -qO - https://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add -
+
+wget http://linux.mellanox.com/public/repo/mlnx_ofed/5.0-2.1.8.0/ubuntu18.04/mellanox_mlnx_ofed.list -O /tmp/mellanox_mlnx_ofed.list && sudo mv /tmp/mellanox_mlnx_ofed.list /etc/apt/sources.list.d/
+
+
+
    +
  1. Config the apt for Netris repository.

  2. +
+
wget -qO - http://repo.netris.ai/repo/public.key | sudo apt-key add -
+
+echo "deb http://repo.netris.ai/repo/ bionic main" | sudo tee /etc/apt/sources.list.d/netris.list
+
+
+
    +
  1. Install Mellanox drivers

  2. +
+
sudo apt-get update && sudo apt-get install mlnx-ofed-dpdk
+
+
+
    +
  1. Install Netris agent package and dependencies, including specific Linux Kernel version.

  2. +
+
sudo apt-get install netris-dpdk-mlnx
+
+
+
    +
  1. Configure Management IP address

  2. +
+

Configure out of band management IP address. In case Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+       address <Management IP address/prefix length>
+       up ip ro add <Controller address> via <Management network gateway> #delete this line if Netris Controller is located in the same network with the SoftGate node.
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Initialize the SoftGate

  2. +
+
+
netris-setup parameters, described below.
+
+
+
–auth - Authentication key, “6878C6DD88224981967F67EE2A73F092” is the default value, we strongly recommend to change this string in your controller as described in Controller initial configuration section.
+
–controller - IP address or domain name of Netris Controller.
+
–hostname - Specify the hostname for the current switch, this hostname should match the name defined for particular switch in the Controller..
+
–lo - IP address for the loopback interface, as it is defined in the controller.
+
–node-prio - brief explanation of node priority goes here
+

+
Run netris-setup.
+
+
sudo /opt/netris/bin/netris-setup --lo=<SoftGate loopback IP address as defined in controller>  --controller=<Netris Controller IP or FQDN> --hostname=<node name as defined in controller> --auth=<authentication key> --node-prio=<node priority 1/2>
+
+
+

Example: Running netris-setup

+
netris@ubuntu:~$ sudo /opt/netris/bin/netris-setup --lo=10.254.97.33  --controller=10.254.97.10 --hostname=softgate1 --auth=6a284d55148f81728f932b28e9d020736c8f78e1950b3d576f6e679d90516df1 --node-prio=1
+* Setup Hostname
+* Setup Hosts
+* Setup Keepalived
+* Setup Collectd
+* Setup Loopback
+* Get CPU List
+* Setup FRR BGP Daemon
+* Setup Netris Agent Config
+* Setup DPDK Router Config
+* Setup DPDK Router Systemd Unit
+└── └── * Setup Grub Config
+* Update Grub
+└──
+
+
+
+
*** ATTENTION: You must reboot SoftGate to complete the installation
+ +

+
+
    +
  1. Reboot the server

  2. +
+
sudo reboot
+
+
+

When server boots up, you should see it’s heartbeat status in Net→Inventory

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/_images/ACL_active.png b/en/2.9/_images/ACL_active.png new file mode 100644 index 0000000000..6ba98a50b8 Binary files /dev/null and b/en/2.9/_images/ACL_active.png differ diff --git a/en/2.9/_images/ACL_approval.png b/en/2.9/_images/ACL_approval.png new file mode 100644 index 0000000000..6345dea408 Binary files /dev/null and b/en/2.9/_images/ACL_approval.png differ diff --git a/en/2.9/_images/ACL_rule.png b/en/2.9/_images/ACL_rule.png new file mode 100644 index 0000000000..81c08c11db Binary files /dev/null and b/en/2.9/_images/ACL_rule.png differ diff --git a/en/2.9/_images/BGP_neighbor.png b/en/2.9/_images/BGP_neighbor.png new file mode 100644 index 0000000000..cdde4813cc Binary files /dev/null and b/en/2.9/_images/BGP_neighbor.png differ diff --git a/en/2.9/_images/BGP_neighbors_listing.png b/en/2.9/_images/BGP_neighbors_listing.png new file mode 100644 index 0000000000..de768abbc4 Binary files /dev/null and b/en/2.9/_images/BGP_neighbors_listing.png differ diff --git a/en/2.9/_images/BGP_route.png b/en/2.9/_images/BGP_route.png new file mode 100644 index 0000000000..b2e702fc7e Binary files /dev/null and b/en/2.9/_images/BGP_route.png differ diff --git a/en/2.9/_images/EVPN_routing.png b/en/2.9/_images/EVPN_routing.png new file mode 100644 index 0000000000..c9196538a8 Binary files /dev/null and b/en/2.9/_images/EVPN_routing.png differ diff --git a/en/2.9/_images/IP_allocation.png b/en/2.9/_images/IP_allocation.png new file mode 100644 index 0000000000..beb8255a39 Binary files /dev/null and b/en/2.9/_images/IP_allocation.png differ diff --git a/en/2.9/_images/IP_pool_assignment.png b/en/2.9/_images/IP_pool_assignment.png new file mode 100644 index 0000000000..38ae3e311b Binary files /dev/null and b/en/2.9/_images/IP_pool_assignment.png differ diff --git a/en/2.9/_images/IPv4_Prefix.png b/en/2.9/_images/IPv4_Prefix.png new file mode 100644 index 0000000000..4726a74662 Binary files /dev/null and b/en/2.9/_images/IPv4_Prefix.png differ diff --git a/en/2.9/_images/IPv6_Prefix.png b/en/2.9/_images/IPv6_Prefix.png new file mode 100644 index 0000000000..c4ab1a9fb0 Binary files /dev/null and b/en/2.9/_images/IPv6_Prefix.png differ diff --git a/en/2.9/_images/ISP_Iris.png b/en/2.9/_images/ISP_Iris.png new file mode 100644 index 0000000000..638bc3a1df Binary files /dev/null and b/en/2.9/_images/ISP_Iris.png differ diff --git a/en/2.9/_images/MAC_listing.png b/en/2.9/_images/MAC_listing.png new file mode 100644 index 0000000000..abe057a8ac Binary files /dev/null and b/en/2.9/_images/MAC_listing.png differ diff --git a/en/2.9/_images/NATIP_address.png b/en/2.9/_images/NATIP_address.png new file mode 100644 index 0000000000..631f1b8bbd Binary files /dev/null and b/en/2.9/_images/NATIP_address.png differ diff --git a/en/2.9/_images/NetSubnets_listing.png b/en/2.9/_images/NetSubnets_listing.png new file mode 100644 index 0000000000..31fbe09a32 Binary files /dev/null and b/en/2.9/_images/NetSubnets_listing.png differ diff --git a/en/2.9/_images/Port_Forwarding.png b/en/2.9/_images/Port_Forwarding.png new file mode 100644 index 0000000000..e079057c1a Binary files /dev/null and b/en/2.9/_images/Port_Forwarding.png differ diff --git a/en/2.9/_images/ROH_instance.png b/en/2.9/_images/ROH_instance.png new file mode 100644 index 0000000000..d5e65969cc Binary files /dev/null and b/en/2.9/_images/ROH_instance.png differ diff --git a/en/2.9/_images/ROH_listing.png b/en/2.9/_images/ROH_listing.png new file mode 100644 index 0000000000..d03f35a800 Binary files /dev/null and b/en/2.9/_images/ROH_listing.png differ diff --git a/en/2.9/_images/SiteMesh_listing.png b/en/2.9/_images/SiteMesh_listing.png new file mode 100644 index 0000000000..7142f87e52 Binary files /dev/null and b/en/2.9/_images/SiteMesh_listing.png differ diff --git a/en/2.9/_images/SiteMesh_modes.png b/en/2.9/_images/SiteMesh_modes.png new file mode 100644 index 0000000000..5edfe756f0 Binary files /dev/null and b/en/2.9/_images/SiteMesh_modes.png differ diff --git a/en/2.9/_images/Site_Mesh.png b/en/2.9/_images/Site_Mesh.png new file mode 100644 index 0000000000..f9fb780b4d Binary files /dev/null and b/en/2.9/_images/Site_Mesh.png differ diff --git a/en/2.9/_images/Spine1.png b/en/2.9/_images/Spine1.png new file mode 100644 index 0000000000..9871609ae1 Binary files /dev/null and b/en/2.9/_images/Spine1.png differ diff --git a/en/2.9/_images/TCAM.png b/en/2.9/_images/TCAM.png new file mode 100644 index 0000000000..0bba84f117 Binary files /dev/null and b/en/2.9/_images/TCAM.png differ diff --git a/en/2.9/_images/VNET_listing.png b/en/2.9/_images/VNET_listing.png new file mode 100644 index 0000000000..1fbd26925b Binary files /dev/null and b/en/2.9/_images/VNET_listing.png differ diff --git a/en/2.9/_images/VNETs.png b/en/2.9/_images/VNETs.png new file mode 100644 index 0000000000..925532de5f Binary files /dev/null and b/en/2.9/_images/VNETs.png differ diff --git a/en/2.9/_images/V_NET.png b/en/2.9/_images/V_NET.png new file mode 100644 index 0000000000..74733e071a Binary files /dev/null and b/en/2.9/_images/V_NET.png differ diff --git a/en/2.9/_images/action_permit.png b/en/2.9/_images/action_permit.png new file mode 100644 index 0000000000..c478300133 Binary files /dev/null and b/en/2.9/_images/action_permit.png differ diff --git a/en/2.9/_images/allocation.png b/en/2.9/_images/allocation.png new file mode 100644 index 0000000000..81a7647856 Binary files /dev/null and b/en/2.9/_images/allocation.png differ diff --git a/en/2.9/_images/anycast_IPv4_address.png b/en/2.9/_images/anycast_IPv4_address.png new file mode 100644 index 0000000000..d1e4f29c99 Binary files /dev/null and b/en/2.9/_images/anycast_IPv4_address.png differ diff --git a/en/2.9/_images/approve_reject.png b/en/2.9/_images/approve_reject.png new file mode 100644 index 0000000000..a3054f9912 Binary files /dev/null and b/en/2.9/_images/approve_reject.png differ diff --git a/en/2.9/_images/assignment.png b/en/2.9/_images/assignment.png new file mode 100644 index 0000000000..ce828c7378 Binary files /dev/null and b/en/2.9/_images/assignment.png differ diff --git a/en/2.9/_images/assignment2.png b/en/2.9/_images/assignment2.png new file mode 100644 index 0000000000..fbac0bb80d Binary files /dev/null and b/en/2.9/_images/assignment2.png differ diff --git a/en/2.9/_images/community.png b/en/2.9/_images/community.png new file mode 100644 index 0000000000..53fd553d3f Binary files /dev/null and b/en/2.9/_images/community.png differ diff --git a/en/2.9/_images/create_link.png b/en/2.9/_images/create_link.png new file mode 100644 index 0000000000..572869cb6f Binary files /dev/null and b/en/2.9/_images/create_link.png differ diff --git a/en/2.9/_images/createboard.png b/en/2.9/_images/createboard.png new file mode 100644 index 0000000000..8040723586 Binary files /dev/null and b/en/2.9/_images/createboard.png differ diff --git a/en/2.9/_images/credentials.png b/en/2.9/_images/credentials.png new file mode 100644 index 0000000000..edbc78148f Binary files /dev/null and b/en/2.9/_images/credentials.png differ diff --git a/en/2.9/_images/defaultroute.png b/en/2.9/_images/defaultroute.png new file mode 100644 index 0000000000..1a4d407173 Binary files /dev/null and b/en/2.9/_images/defaultroute.png differ diff --git a/en/2.9/_images/edit_switch_port.png b/en/2.9/_images/edit_switch_port.png new file mode 100644 index 0000000000..711980ab9e Binary files /dev/null and b/en/2.9/_images/edit_switch_port.png differ diff --git a/en/2.9/_images/globalIP.png b/en/2.9/_images/globalIP.png new file mode 100644 index 0000000000..bae8a860ce Binary files /dev/null and b/en/2.9/_images/globalIP.png differ diff --git a/en/2.9/_images/graphboard.png b/en/2.9/_images/graphboard.png new file mode 100644 index 0000000000..b777f11b69 Binary files /dev/null and b/en/2.9/_images/graphboard.png differ diff --git a/en/2.9/_images/hairpin.png b/en/2.9/_images/hairpin.png new file mode 100644 index 0000000000..8bfa508087 Binary files /dev/null and b/en/2.9/_images/hairpin.png differ diff --git a/en/2.9/_images/hairpin_topology.png b/en/2.9/_images/hairpin_topology.png new file mode 100644 index 0000000000..95b3c4fe5d Binary files /dev/null and b/en/2.9/_images/hairpin_topology.png differ diff --git a/en/2.9/_images/hardware_health.png b/en/2.9/_images/hardware_health.png new file mode 100644 index 0000000000..df70fc2ecc Binary files /dev/null and b/en/2.9/_images/hardware_health.png differ diff --git a/en/2.9/_images/installOS.png b/en/2.9/_images/installOS.png new file mode 100644 index 0000000000..cd36377e22 Binary files /dev/null and b/en/2.9/_images/installOS.png differ diff --git a/en/2.9/_images/inventory_heartbeat.png b/en/2.9/_images/inventory_heartbeat.png new file mode 100644 index 0000000000..8e884600a3 Binary files /dev/null and b/en/2.9/_images/inventory_heartbeat.png differ diff --git a/en/2.9/_images/inventory_profile.png b/en/2.9/_images/inventory_profile.png new file mode 100644 index 0000000000..9761caabe1 Binary files /dev/null and b/en/2.9/_images/inventory_profile.png differ diff --git a/en/2.9/_images/leaf1_spine1.png b/en/2.9/_images/leaf1_spine1.png new file mode 100644 index 0000000000..8b9b4a3ff0 Binary files /dev/null and b/en/2.9/_images/leaf1_spine1.png differ diff --git a/en/2.9/_images/listing_L3.png b/en/2.9/_images/listing_L3.png new file mode 100644 index 0000000000..9d05242700 Binary files /dev/null and b/en/2.9/_images/listing_L3.png differ diff --git a/en/2.9/_images/listing_L4.png b/en/2.9/_images/listing_L4.png new file mode 100644 index 0000000000..fdaf235f72 Binary files /dev/null and b/en/2.9/_images/listing_L4.png differ diff --git a/en/2.9/_images/listing_kubenet.png b/en/2.9/_images/listing_kubenet.png new file mode 100644 index 0000000000..98335713ec Binary files /dev/null and b/en/2.9/_images/listing_kubenet.png differ diff --git a/en/2.9/_images/load-balancer.png b/en/2.9/_images/load-balancer.png new file mode 100644 index 0000000000..4ace2e1371 Binary files /dev/null and b/en/2.9/_images/load-balancer.png differ diff --git a/en/2.9/_images/loadbalancer_listing.png b/en/2.9/_images/loadbalancer_listing.png new file mode 100644 index 0000000000..8c388b11ef Binary files /dev/null and b/en/2.9/_images/loadbalancer_listing.png differ diff --git a/en/2.9/_images/netris_controller_diagram.png b/en/2.9/_images/netris_controller_diagram.png new file mode 100644 index 0000000000..610a8ae396 Binary files /dev/null and b/en/2.9/_images/netris_controller_diagram.png differ diff --git a/en/2.9/_images/new_VNET.png b/en/2.9/_images/new_VNET.png new file mode 100644 index 0000000000..5fa3e54493 Binary files /dev/null and b/en/2.9/_images/new_VNET.png differ diff --git a/en/2.9/_images/new_hardware.png b/en/2.9/_images/new_hardware.png new file mode 100644 index 0000000000..f6afc05f61 Binary files /dev/null and b/en/2.9/_images/new_hardware.png differ diff --git a/en/2.9/_images/new_kubenet.png b/en/2.9/_images/new_kubenet.png new file mode 100644 index 0000000000..b6df06c237 Binary files /dev/null and b/en/2.9/_images/new_kubenet.png differ diff --git a/en/2.9/_images/password.png b/en/2.9/_images/password.png new file mode 100644 index 0000000000..a27b7725a9 Binary files /dev/null and b/en/2.9/_images/password.png differ diff --git a/en/2.9/_images/permission_group.png b/en/2.9/_images/permission_group.png new file mode 100644 index 0000000000..e4d1ad6ab0 Binary files /dev/null and b/en/2.9/_images/permission_group.png differ diff --git a/en/2.9/_images/ping.png b/en/2.9/_images/ping.png new file mode 100644 index 0000000000..c8e9e64e47 Binary files /dev/null and b/en/2.9/_images/ping.png differ diff --git a/en/2.9/_images/request_L4.png b/en/2.9/_images/request_L4.png new file mode 100644 index 0000000000..8e73cf8be8 Binary files /dev/null and b/en/2.9/_images/request_L4.png differ diff --git a/en/2.9/_images/route-map.png b/en/2.9/_images/route-map.png new file mode 100644 index 0000000000..717a9bb9c4 Binary files /dev/null and b/en/2.9/_images/route-map.png differ diff --git a/en/2.9/_images/sandbox-l4lb-kubeapi.png b/en/2.9/_images/sandbox-l4lb-kubeapi.png new file mode 100644 index 0000000000..6a592476df Binary files /dev/null and b/en/2.9/_images/sandbox-l4lb-kubeapi.png differ diff --git a/en/2.9/_images/sandbox-l4lbs.png b/en/2.9/_images/sandbox-l4lbs.png new file mode 100644 index 0000000000..c002c0fed7 Binary files /dev/null and b/en/2.9/_images/sandbox-l4lbs.png differ diff --git a/en/2.9/_images/sandbox-podinfo-prov.png b/en/2.9/_images/sandbox-podinfo-prov.png new file mode 100644 index 0000000000..079871e59a Binary files /dev/null and b/en/2.9/_images/sandbox-podinfo-prov.png differ diff --git a/en/2.9/_images/sandbox-podinfo-ready.png b/en/2.9/_images/sandbox-podinfo-ready.png new file mode 100644 index 0000000000..d6919ecf9d Binary files /dev/null and b/en/2.9/_images/sandbox-podinfo-ready.png differ diff --git a/en/2.9/_images/sandbox_topology.png b/en/2.9/_images/sandbox_topology.png new file mode 100644 index 0000000000..94276f44c5 Binary files /dev/null and b/en/2.9/_images/sandbox_topology.png differ diff --git a/en/2.9/_images/sandbox_topology_n.png b/en/2.9/_images/sandbox_topology_n.png new file mode 100644 index 0000000000..2743b42a5c Binary files /dev/null and b/en/2.9/_images/sandbox_topology_n.png differ diff --git a/en/2.9/_images/saveasnormal.png b/en/2.9/_images/saveasnormal.png new file mode 100644 index 0000000000..6f5cd5bda3 Binary files /dev/null and b/en/2.9/_images/saveasnormal.png differ diff --git a/en/2.9/_images/siteDefault.png b/en/2.9/_images/siteDefault.png new file mode 100644 index 0000000000..f8b1cf2688 Binary files /dev/null and b/en/2.9/_images/siteDefault.png differ diff --git a/en/2.9/_images/softgate_diagram.png b/en/2.9/_images/softgate_diagram.png new file mode 100644 index 0000000000..a6ade1fd5f Binary files /dev/null and b/en/2.9/_images/softgate_diagram.png differ diff --git a/en/2.9/_images/softgate_node.png b/en/2.9/_images/softgate_node.png new file mode 100644 index 0000000000..2cf048d951 Binary files /dev/null and b/en/2.9/_images/softgate_node.png differ diff --git a/en/2.9/_images/static_route.png b/en/2.9/_images/static_route.png new file mode 100644 index 0000000000..a022e230d4 Binary files /dev/null and b/en/2.9/_images/static_route.png differ diff --git a/en/2.9/_images/subnet_listing.png b/en/2.9/_images/subnet_listing.png new file mode 100644 index 0000000000..fc7a6a62be Binary files /dev/null and b/en/2.9/_images/subnet_listing.png differ diff --git a/en/2.9/_images/switch_port.png b/en/2.9/_images/switch_port.png new file mode 100644 index 0000000000..8f0e537af5 Binary files /dev/null and b/en/2.9/_images/switch_port.png differ diff --git a/en/2.9/_images/switch_port_statuses.png b/en/2.9/_images/switch_port_statuses.png new file mode 100644 index 0000000000..c93ae7594d Binary files /dev/null and b/en/2.9/_images/switch_port_statuses.png differ diff --git a/en/2.9/_images/telescope.png b/en/2.9/_images/telescope.png new file mode 100644 index 0000000000..7cfc53fcc7 Binary files /dev/null and b/en/2.9/_images/telescope.png differ diff --git a/en/2.9/_images/tenants.png b/en/2.9/_images/tenants.png new file mode 100644 index 0000000000..eb4172733d Binary files /dev/null and b/en/2.9/_images/tenants.png differ diff --git a/en/2.9/_images/topology_manager.png b/en/2.9/_images/topology_manager.png new file mode 100644 index 0000000000..ce7bb1effe Binary files /dev/null and b/en/2.9/_images/topology_manager.png differ diff --git a/en/2.9/_images/uninstallOS.png b/en/2.9/_images/uninstallOS.png new file mode 100644 index 0000000000..eb2e1965c0 Binary files /dev/null and b/en/2.9/_images/uninstallOS.png differ diff --git a/en/2.9/_images/updateONIE.png b/en/2.9/_images/updateONIE.png new file mode 100644 index 0000000000..093c50c2de Binary files /dev/null and b/en/2.9/_images/updateONIE.png differ diff --git a/en/2.9/_images/user_role.png b/en/2.9/_images/user_role.png new file mode 100644 index 0000000000..d285d9b688 Binary files /dev/null and b/en/2.9/_images/user_role.png differ diff --git a/en/2.9/_images/users.png b/en/2.9/_images/users.png new file mode 100644 index 0000000000..2f90077728 Binary files /dev/null and b/en/2.9/_images/users.png differ diff --git a/en/2.9/_images/waiting_approval.png b/en/2.9/_images/waiting_approval.png new file mode 100644 index 0000000000..ac769acb93 Binary files /dev/null and b/en/2.9/_images/waiting_approval.png differ diff --git a/en/2.9/_sources/SoftGate-agent-installation.rst.txt b/en/2.9/_sources/SoftGate-agent-installation.rst.txt new file mode 100644 index 0000000000..ad18da35df --- /dev/null +++ b/en/2.9/_sources/SoftGate-agent-installation.rst.txt @@ -0,0 +1,139 @@ +.. meta:: + :description: Netris SoftGate Agent Installation + +*********************************** +Netris SoftGate agent installation +*********************************** +Minimal hardware requirements +============================= +* 2 x Intel Silver CPU +* 96 GB RAM +* 300 GB HDD +* Nvidia Mellanox Connect-X 5 SmartNIC card + +BIOS configuration +================== +The following are some recommendations on BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference: + +* Before starting consider resetting all BIOS settings to their defaults. +* Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report. +* Select Performance as the CPU Power and Performance policy. +* Disable Turbo Boost to ensure the performance scaling increases with the number of cores. +* Set memory frequency to the highest available number, NOT auto. +* Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d. +* Disable Hyper-Threading. + +Software installation +===================== +Requires freshly installed Ubuntu Linux 18.04 and network connectivity with your Netris Controller over the out-of-band management network. + +1. Set environment variables to use Netris Controller as a proxy. + +.. code-block:: shell-session + + export http_proxy=http://:3128 && export https_proxy=http://:3128 + + echo -e 'Acquire::http::Proxy "http://:3128";\nAcquire::https::Proxy "http://:3128";' | sudo tee -a /etc/apt/apt.conf.d/netris-proxy + +2. Config the apt for Mellanox repository. + +.. code-block:: shell-session + + wget -qO - https://www.mellanox.com/downloads/ofed/RPM-GPG-KEY-Mellanox | sudo apt-key add - + + wget http://linux.mellanox.com/public/repo/mlnx_ofed/5.0-2.1.8.0/ubuntu18.04/mellanox_mlnx_ofed.list -O /tmp/mellanox_mlnx_ofed.list && sudo mv /tmp/mellanox_mlnx_ofed.list /etc/apt/sources.list.d/ + +3. Config the apt for Netris repository. + +.. code-block:: shell-session + + wget -qO - http://repo.netris.ai/repo/public.key | sudo apt-key add - + + echo "deb http://repo.netris.ai/repo/ bionic main" | sudo tee /etc/apt/sources.list.d/netris.list + +4. Install Mellanox drivers + +.. code-block:: shell-session + + sudo apt-get update && sudo apt-get install mlnx-ofed-dpdk + +5. Install Netris agent package and dependencies, including specific Linux Kernel version. + +.. code-block:: shell-session + + sudo apt-get install netris-dpdk-mlnx + +6. Configure Management IP address + +Configure out of band management IP address. In case Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + up ip ro add via #delete this line if Netris Controller is located in the same network with the SoftGate node. + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +7. Initialize the SoftGate + +| netris-setup parameters, described below. + +| **--auth** - Authentication key, “6878C6DD88224981967F67EE2A73F092” is the default value, we strongly recommend to change this string in your controller as described in Controller initial configuration section. +| **--controller** - IP address or domain name of Netris Controller. +| **--hostname** - Specify the hostname for the current switch, this hostname should match the name defined for particular switch in the Controller.. +| **--lo** - IP address for the loopback interface, as it is defined in the controller. +| **--node-prio - brief explanation of node priority goes here** +| +| Run netris-setup. + +.. code-block:: shell-session + + sudo /opt/netris/bin/netris-setup --lo= --controller= --hostname= --auth= --node-prio= + +Example: Running netris-setup + +.. code-block:: shell-session + + netris@ubuntu:~$ sudo /opt/netris/bin/netris-setup --lo=10.254.97.33 --controller=10.254.97.10 --hostname=softgate1 --auth=6a284d55148f81728f932b28e9d020736c8f78e1950b3d576f6e679d90516df1 --node-prio=1 + * Setup Hostname + * Setup Hosts + * Setup Keepalived + * Setup Collectd + * Setup Loopback + * Get CPU List + * Setup FRR BGP Daemon + * Setup Netris Agent Config + * Setup DPDK Router Config + * Setup DPDK Router Systemd Unit + └── └── * Setup Grub Config + * Update Grub + └── + +| *** ATTENTION: You must reboot SoftGate to complete the installation +| netris@ubuntu:~$ +| + +8. Reboot the server + +.. code-block:: shell-session + + sudo reboot + +When server boots up, you should see it’s heartbeat status in Net→Inventory + + diff --git a/en/2.9/_sources/accounts.rst.txt b/en/2.9/_sources/accounts.rst.txt new file mode 100644 index 0000000000..eb264dbe69 --- /dev/null +++ b/en/2.9/_sources/accounts.rst.txt @@ -0,0 +1,64 @@ +.. meta:: + :description: Netris Controller User Account Management + +######## +Accounts +######## + +The accounts section is for the management of user accounts, access permissions, and tenants. + +Users +===== +Description of User account fields: + +* **Username** - Unique username. +* **Full Name** - Full Name of the user. +* **E-mail** - The email address of the user. Also used for system notifications and for password retrieval. +* **E-mail CC** - Send copies of email notifications to this address. +* **Phone Number** - User’s phone number. +* **Company** - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers. +* **Position** - Position within the company. +* **User Role** - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate. +* **Permission Group** - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used) +* **+Tenant** - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used) + +Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin. + +.. image:: images/users.png + :align: center + +**Password**: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side. + +Example: Listing of user accounts. + + +.. image:: images/password.png + :align: center + +Tenants +======= +IP addresses and Switch Ports are network resources that can be assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs) as part of DevOps/NetOps pipeline. + +A Tenant has just two fields, the unique name and custom description. + +Example: Adding a tenant. + +.. image:: images/tenants.png + :align: center + +Permission Groups +================= +Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections. + +Example: Permission Group. + +.. image:: images/permission_group.png + :align: center + +User Roles +========== +Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username. + +.. image:: images/user_role.png + :align: center + diff --git a/en/2.9/_sources/concepts.rst.txt b/en/2.9/_sources/concepts.rst.txt new file mode 100644 index 0000000000..4ced571122 --- /dev/null +++ b/en/2.9/_sources/concepts.rst.txt @@ -0,0 +1,44 @@ +.. meta:: + :description: Netris Core Concepts + +######## +Concepts +######## + +Introduction to Netris +====================== +Netris is an automatic netops software for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network's health and either apply software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation. + + +What is Netris Controller +========================= +Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud. + +Diagram: High level Netris architecture. + +.. image:: images/netris_controller_diagram.png + :align: center + +* **Controller HA.** We highly recommend running more than one copy of the controller for database replication. +* **Multiple sites.** Netris is designed to operate multiple sites with just a single controller with HA. +* **What if the controller is unreachable.** Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers' core operation will not be affected. + +Netris Switch Agent +=================== +Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet. + +Netris SoftGate +=============== +Netris SoftGate is automatic configuration software and reference architecture for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), and site-to-site VPN function on a regular x86 server with a SmartNIC card. + +Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2. + +The server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities. + +Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels. + + +Diagram: Netris SoftGate high level architecture + +.. image:: images/softgate_diagram.png + :align: center diff --git a/en/2.9/_sources/controller-initial-configuration.rst.txt b/en/2.9/_sources/controller-initial-configuration.rst.txt new file mode 100644 index 0000000000..5c1734167a --- /dev/null +++ b/en/2.9/_sources/controller-initial-configuration.rst.txt @@ -0,0 +1,162 @@ +.. meta:: + :description: Netris Controller Initial Configuration + +******************************** +Controller initial configuration +******************************** + +Definitions +=========== + +* **User** - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is ``netris``, with password ``newNet0ps``. +* **Tenant** - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request & manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs) DevOps/NetOps pipeline. +* **Permission Group** - List of permissions on a per section basis can be attached individually to a User or a User Role. +* **User Role** - Group of user permissions and tenants for role-based access control. +* **Site** - Each separate deployment (each data center) should be defined as a Site. All network units and resources are attached to a site. Netris Controller comes with a "default" site preconfigured. Site entry defines global attributes such as; AS numbers, default ACL policy, Site Mesh (site to site VPN) type. +* **Subnet** - IPv4/IPv6 address resources linked to Sites and Tenants. +* **Switch Port** - Physical ports of all switches attached to the system. Switch port objects represent statuses, take basic parameters, and are assigned to Tenants. +* **Inventory** - This is an inventory of all network units that are operated using Netris Agent. +* **E-BGP** - Is for defining all External BGP peers (iBGP and eBGP). + +Subnets +======= +It is required to define at least two subnets to get started. One subnet is for the management interfaces, another for the loopback addresses. Every network unit managed with Netris should have at least one management IP and at least one loopback IP. Loopback IP addresses are used for network unit identification by network protocols and by Netris Agent/Controller. There’s no need for defining any IP addresses for the switch-to-switch links. Netris is using IPv6 link-local addresses for all switch-to-switch communication. + +Example: (IP addresses used are just examples, please replace them following your IP planning.) +------------------------------------------------------------------------------------------------ +In NET->Subnets section of the Netris Controller GUI, you can add new subnet entries. Subnets are of 2 types of allocation and assignment. Allocations are the large blocks of IP resources assigned to the organization. Assignments are IP blocks that are smaller blocks inside the allocation and can be used by services or policies that yet to be defined. + +1. Adding a new allocation. In this example, 10.0.0.0/8 is used as a large block of allocation. You can add as many allocations as required. + +.. image:: images/allocation.png + :align: center + +2. Adding two new assignments. + +* 10.254.96.0/24 (netManagement) assigned to the tenant “Admin” and available for the site “Default”. +* 10.254.97.0/24 (netLoopbacks) assigned to the tenant “Admin” and available for the site “Default”. + +.. image:: images/assignment.png + :align: center + +.. image:: images/assignment2.png + :align: center + +Screenshot: Listing of the Subnets section after adding the new objects. + +.. image:: images/subnet_listing.png + :align: center +| +| +Inventory Profiles +================== +Inventory profiles define access security, timezone, DNS, NTP settings profiles for network switches and SoftGate nodes. +To create a new Inventory profile, click +Add under the Net→Inventory Profiles section. + +Fields descriptions: +-------------------- +* **Name** - Profile name. +* **Description** - Free text description. +* **Allow SSH from IPv4** - List of IPv4 subnets allowed to ssh (one address per line) +* **Allow SSH from IPv6** - List of IPv6 subnets allowed to ssh (one address per line) +* **Timezone** - Devices using this inventory profile will adjust their system time to the selected timezone. +* **NTP servers** - List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate. +* **DNS servers** - List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate. + +**Example:** In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup). + +.. image:: images/inventory_profile.png + :align: center + +Adding Switches to Topology +=========================== +You need to define every switch in the Net→Topology section. To add a switch, please go to Net→Topology and click +Add. + +* **Name** - Descriptive name. +* **Owner Tenant** - Tenant(typically Admin) who administers this node. +* **Description** - Free text description. +* **Hardware Type** - For switches: Spine Switch or Leaf Switch. +* **NOS*** - Network operating system. Cumulus Linux, Ubuntu SwitchDev (Nvidia Mellanox only), SONiC (not for production use yet) +* **Site*** - The site where the switch belongs. +* **Inventory Profile** - Reference to Timezone, DNS, NTP, and Security features profile. +* **IP Address*** - IPv4 address for the loopback interface. +* **Management IP address** - IPv4 address for the out of band management interface. +* **Zero-touch provisioning** - Automatically install the NOS. (Experimental in this version) +* **MAC address** - Out of band management interface MAC address used for zero-touch provisioning. (Experimental in this version) +* **The number of ports** - It is required for the topology manager. WIll be synced to the real number of Switch Ports when Netris Switch Agent establishes the very first connection with the Netris Controller. + +**Example:** Adding a spine switch w/ Cumulus Linux. + +.. image:: images/new_hardware.png + :align: center + +Tip: You can drag/move the units to your desired positions and click “Save positions”. + +Note: Repeat this process to define all your switches. + +Topology Manager +================ +The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents will configure the underlying network devices according to this topology dynamically and start watching against potential failures. + +To define the links, right-click on the spine switch, then click create a link. Select the “from port,” then “to device” and “port.” See the example below. + +.. image:: images/create_link.png + :align: center + +All links require definition in the topology manager. Topology links can also be described through a .yaml file when using Kubernetes CRD. (a GUI wizard is planned to be available in v2.10). + +.. image:: images/topology_manager.png + :align: center + +Now when network units and links are defined, your network is automatically configured as long as physical connectivity is in place and Netris Agents can communicate with Netris Controller. + +Hairpin (Cumulus only) +====================== +With Cumulus Linux only, we need to loop two ports on spine switches (hairpin cable) in the current release, usually two upstream (higher capacity) ports. We are planning to lift this requirement in the next Netris release (v2.10). + +To define what ports will be used as a hairpin, navigate to Net→Switch Ports, or right-click on the spine switch, click Ports in Net-->Topology. + +Example: Accessing Switch Ports from Net→Topology + +.. image:: images/switch_port.png + :align: center + +For each spine switch, find the two ports that you are going to connect (loop/hairpin) and configure one port as a “hairpin **l2**” and another port as “hairpin **l3**”. The order doesn’t matter. The system needs to know which ports you have dedicated for the hairpin/loop on each spine switch. (do not do this for non-Cumulus switches) +| +| +Example: Editing Switch Port from Net→Switch Ports. + +.. image:: images/edit_switch_port.png + :align: center + +Example: Setting port types to “hairpin l2” and “hairpin l3”. + +.. image:: images/hairpin.png + :align: center + +Screenshot: Hairpin visualized in Net→Topology + +.. image:: images/hairpin_topology.png + :align: center + +Adding SoftGate nodes to Topology +================================= +Every SoftGate node first needs to be defined in Netris Controller. +To add a SoftGate node, please go to Net→Topology and click +Add. + +* **Name** - Descriptive name. +* **Owner Tenant** - Tenant(typically Admin), who administers this node. +* **Description** - Free text description. +* **Hardware Type** - NFV node. +* **Site** - The data center where the current SoftGate node belongs. +* **Inventory Profile** - Profile describing the timezone, DNS, NTP, and Security features. +* **IP Address** - IPv4 address for the loopback interface. +* **Management IP address** - IPv4 address for the out of band management interface. +* **NFV Node Port** - A physical port on a spine switch where the SoftGate node's first SmartNIC port is connected. Typically each spine switch has one SoftGate node connected to it. +* **+NAT address** - Public IP addresses to be used as global IP for SNAT/DNAT. (check Enabling NAT section of Network Policies chapter) +* **+NAT address pool** - Public IP address subnets to be used as rolling global IP addresses for SNAT. (check Enabling NAT section of Network Policies chapter) + +Example: Adding a SoftGate Node to Topology. + +.. image:: images/softgate_node.png + :align: center diff --git a/en/2.9/_sources/controller-installation.rst.txt b/en/2.9/_sources/controller-installation.rst.txt new file mode 100644 index 0000000000..3d896cf1d8 --- /dev/null +++ b/en/2.9/_sources/controller-installation.rst.txt @@ -0,0 +1,212 @@ +.. meta:: + :description: Netris Controller Virtual Machine Installation + +************************************** +On-prem Netris Controller installation +************************************** +Netris Controller can be hosted in Netris cloud, installed locally as a VM, or deployed as a Kubernetes application. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime. + +KVM virtual machine +=================== +| Minimal system requirements for the VM: +| CPU - 8 Core +| RAM - 16 Gb +| Disk - 100Gb +| Network - 1 virtual NIC + +Installation steps for KVM hypervisor +===================================== +If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04) + +.. code-block:: shell-session + + sudo apt-get install virt-manager + +Netris Controller Installation steps +==================================== + +1. Download the Netris Controller image. (contact Netris support for repository access permissions). + +.. code-block:: shell-session + + cd /var/lib/libvirt/images + + sudo wget http://img.netris.ai/netris-controller.qcow2 + +2. Download vm definition file. + +.. code-block:: shell-session + + cd /etc/libvirt/qemu + + sudo wget http://img.netris.ai/netris-controller.xml + +3. Define the KVM virtual machine + +.. code-block:: shell-session + + sudo virsh define netris-controller.xml + +.. note:: + + Netris controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below network interface configuration exam + +Example: Network configuration on host (hypervisor) machine. + +.. note:: + + replace , and + with the correct NIC and IP for your host machine. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + #Physical NIC connected to the management network + auto + iface inet static + address 0.0.0.0/0 + + #bridge interface + auto br-mgmt + iface br-mgmt inet static + address + gateway + bridge-ports + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +4. Set the virtual machine to autostart and start it. + +.. code-block:: shell-session + + sudo virsh autostart netris-controller + +.. code-block:: shell-session + + sudo virsh start netris-controller + +Accessing the Netris Controller +=============================== +By default, Netris Controller will obtain an IP address from a **DHCP** server. + +Below steps describe how to configure a **static IP** address for the Netris Controller. + +1. Connecting to the VM console. + +default credentials. **login**: ``netris`` **password**: ``newNet0ps`` + +.. code-block:: shell-session + + sudo virsh console netris-controller + +.. note:: + + Do not forget to change the default password (using passwd command). + +2. Setting a static IP address. + +Edit network configuration file. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +Example: IP configuration file. + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +Reload the network config. + +.. code-block:: shell-session + + sudo ifreload -a + +.. note:: + + Make sure Netris Controller has Internet access. + +3. Reboot the controller + +.. code-block:: shell-session + + sudo reboot + +After reboot, the Netris Controller GUI should be accessible using a browser. Use ``netris/newNet0ps`` credentials. + +.. image:: images/credentials.png + :align: center + +Don’t forget to change the default password by clicking your login name in the top right corner and then clicking “Change Password”. + +Security hardening +================== +| Recommended for production use. +| + +Changing the default GRPC authentication key. +--------------------------------------------- + +Connect to the Netris Controller CLI (SSH or Console) + +Tip: You can generate a random and secure key using sha256sum. + +.. code-block:: shell-session + + echo "" | sha256sum + +example: + +.. code-block:: shell-session + + netris@iris:~$ echo "" | sha256sum + 6a284d55148f81728f932b28e9d020736c8f78e1950b3d576f6e679d90516df1 - + +Set your newly generated secure key into Netris Controller. + +.. code-block:: shell-session + + sudo /opt/telescope/netris-set-auth.sh --key + +Please store the auth key in a safe place as it will be required every time when installing Netris Agent for the switches and SoftGates. + +Replacing the SSL certificate +------------------------------ + +1. Replace below file with your SSL certificate file. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.cert.pem; + +2. Replace below file with your SSL private key. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.key.pem; + +3. Restart Nginx service. + +.. code-block:: shell-session + + systemctl restart nginx.service diff --git a/en/2.9/_sources/controller-k8s-installation.rst.txt b/en/2.9/_sources/controller-k8s-installation.rst.txt new file mode 100644 index 0000000000..320dd395d6 --- /dev/null +++ b/en/2.9/_sources/controller-k8s-installation.rst.txt @@ -0,0 +1,567 @@ + +.. meta:: + :description: Netris Controller Installation on Kubernetes + +Netris Controller Helm Chart +============================ + + +* Installs the automatic NetOps platform `Netris Controller `_ + +Prerequisites +------------- + + +* Kubernetes 1.12+ +* Helm 3.1+ +* PV provisioner support in the underlying infrastructure + +Get Repo Info +------------- + +Add the Netris Helm repository: + +.. code-block:: + + helm repo add netrisai https://netrisai.github.io/charts + helm repo update + +Installing the Chart +-------------------- + +In order to install the Helm chart, you must follow these steps: + +Create the namespace for netris-controller: + +.. code-block:: + + kubectl create namespace netris-controller + +Generate strong auth key + +.. code-block:: + + export mystrongauthkey=$(date |base64 | md5sum | base64 | head -c 32) + +Install helm chart with netris-controller + +.. code-block:: + + helm install netris-controller netrisai/netris-controller \ + --namespace netris-controller \ + --set app.ingress.hosts={my.domain.com} \ + --set netris.authKey=$mystrongauthkey + +Uninstalling the Chart +---------------------- + +To uninstall/delete the ``netris-controller`` helm release: + +.. code-block:: + + helm uninstall netris-controller + +Configuration +------------- + +The following table lists the configurable parameters of the netris-controller chart and their default values. + +Common parameters +^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``nameOverride`` + - String to partially override common.names.fullname template with a string (will prepend the release name) + - ``nil`` + * - ``fullnameOverride`` + - String to fully override common.names.fullname template with a string + - ``nil`` + * - ``serviceAccount.create`` + - Create a serviceAccount for the deployment + - ``true`` + * - ``serviceAccount.name`` + - Use the serviceAccount with the specified name + - ``""`` + * - ``serviceAccount.annotations`` + - Annotations to add to the service account + - ``{}`` + * - ``podAnnotations`` + - Pod annotations + - ``{}`` + * - ``podSecurityContext`` + - Pod Security Context + - ``{}`` + * - ``securityContext`` + - Containers security context + - ``{}`` + * - ``resources`` + - CPU/memory resource requests/limits + - ``{}`` + * - ``nodeSelector`` + - Node labels for pod assignment + - ``{}`` + * - ``tolerations`` + - Node tolerations for pod assignment + - ``[]`` + * - ``affinity`` + - Node affinity for pod assignment + - ``{}`` + + +Netris-Controller common parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``netris.webLogin`` + - Netris Controller GUI default login + - ``netris`` + * - ``netris.webPassword`` + - Netris Controller GUI default password + - ``newNet0ps`` + * - ``netris.authKey`` + - Netris Controller agents authentication key + - ``mystrongkey`` + + +Netris-Controller app parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``app.replicaCount`` + - Number of replicas in app deployment + - ``1`` + * - ``app.image.repository`` + - Image repository + - ``netrisai/xcaas-xcaas-web`` + * - ``app.image.tag`` + - Image tag. Overrides the image tag whose default is the chart appVersion + - ``""`` + * - ``app.image.pullPolicy`` + - Image pull policy + - ``IfNotPresent`` + * - ``app.imagePullSecrets`` + - Reference to one or more secrets to be used when pulling images + - ``[]`` + * - ``app.service.type`` + - Kubernetes service type + - ``ClusterIP`` + * - ``app.service.port`` + - Kubernetes port where service is expose + - ``80`` + * - ``app.service.portName`` + - Name of the port on the service + - ``http`` + * - ``app.ingress.enabled`` + - Enables Ingress + - ``true`` + * - ``app.ingress.annotations`` + - Ingress annotations (values are templated) + - ``{ kubernetes.io/ingress.class: nginx }`` + * - ``app.ingress.labels`` + - Custom labels + - ``{}`` + * - ``app.ingress.path`` + - Ingress accepted path + - ``/`` + * - ``app.ingress.pathType`` + - Ingress type of path + - ``Prefix`` + * - ``app.ingress.hosts`` + - Ingress accepted hostnames + - ``["chart-example.local"]`` + * - ``app.ingress.tls`` + - Ingress TLS configuration + - ``[]`` + * - ``app.autoscaling.enabled`` + - Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics + - ``false`` + * - ``app.autoscaling.minReplicas`` + - Default min replicas for autoscaling + - ``1`` + * - ``app.autoscaling.maxReplicas`` + - Default max replicas for autoscaling + - ``100`` + * - ``app.autoscaling.targetCPUUtilizationPercentage`` + - The desired target CPU utilization for autoscaling + - ``80`` + + +Netris-Controller grpc parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``grpc.replicaCount`` + - Number of replicas in grpc deployment + - ``1`` + * - ``grpc.image.repository`` + - Image repository + - ``netrisai/xcaas-agent-api-server`` + * - ``grpc.image.tag`` + - Image tag. Overrides the image tag whose default is the chart appVersion + - ``""`` + * - ``grpc.image.pullPolicy`` + - Image pull policy + - ``IfNotPresent`` + * - ``grpc.imagePullSecrets`` + - Reference to one or more secrets to be used when pulling images + - ``[]`` + * - ``grpc.service.type`` + - Kubernetes service type + - ``ClusterIP`` + * - ``grpc.service.port`` + - Kubernetes port where service is expose + - ``443`` + * - ``grpc.service.portName`` + - Name of the port on the service + - ``grpc`` + * - ``grpc.autoscaling.enabled`` + - Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics + - ``false`` + * - ``grpc.autoscaling.minReplicas`` + - Default min replicas for autoscaling + - ``1`` + * - ``grpc.autoscaling.maxReplicas`` + - Default max replicas for autoscaling + - ``100`` + * - ``grpc.autoscaling.targetCPUUtilizationPercentage`` + - The desired target CPU utilization for autoscaling + - ``80`` + + +Netris-Controller telescope parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``telescope.replicaCount`` + - Number of replicas in telescope deployment + - ``1`` + * - ``telescope.image.repository`` + - Image repository + - ``netrisai/xcaas-telescope-go`` + * - ``telescope.image.tag`` + - Image tag. Overrides the image tag whose default is the chart appVersion + - ``""`` + * - ``telescope.image.pullPolicy`` + - Image pull policy + - ``IfNotPresent`` + * - ``telescope.imagePullSecrets`` + - Reference to one or more secrets to be used when pulling images + - ``[]`` + * - ``telescope.service.type`` + - Kubernetes service type + - ``ClusterIP`` + * - ``telescope.service.port`` + - Kubernetes port where service is expose + - ``80`` + * - ``telescope.service.portName`` + - Name of the port on the service + - ``ws`` + * - ``telescope.service.securePort`` + - Kubernetes secure port where service is expose + - ``443`` + * - ``telescope.service.securePortName`` + - Name of the secure port on the service + - ``wss`` + * - ``telescope.autoscaling.enabled`` + - Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics + - ``false`` + * - ``telescope.autoscaling.minReplicas`` + - Default min replicas for autoscaling + - ``1`` + * - ``telescope.autoscaling.maxReplicas`` + - Default max replicas for autoscaling + - ``100`` + * - ``telescope.autoscaling.targetCPUUtilizationPercentage`` + - The desired target CPU utilization for autoscaling + - ``80`` + + +Netris-Controller k8s-watcher parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``k8s-watcher.replicaCount`` + - Number of replicas in k8s-watcher deployment + - ``1`` + * - ``k8s-watcher.image.repository`` + - Image repository + - ``netrisai/xcaas-kuberis-k8-api-agent`` + * - ``k8s-watcher.image.tag`` + - Image tag. Overrides the image tag whose default is the chart appVersion + - ``""`` + * - ``k8s-watcher.image.pullPolicy`` + - Image pull policy + - ``IfNotPresent`` + * - ``k8s-watcher.imagePullSecrets`` + - Reference to one or more secrets to be used when pulling images + - ``[]`` + * - ``k8s-watcher.autoscaling.enabled`` + - Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics + - ``false`` + * - ``k8s-watcher.autoscaling.minReplicas`` + - Default min replicas for autoscaling + - ``1`` + * - ``k8s-watcher.autoscaling.maxReplicas`` + - Default max replicas for autoscaling + - ``100`` + * - ``k8s-watcher.autoscaling.targetCPUUtilizationPercentage`` + - The desired target CPU utilization for autoscaling + - ``80`` + + +Netris-Controller telescope-notifier parameters +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``telescope-notifier.replicaCount`` + - Number of replicas in telescope-notifier deployment + - ``1`` + * - ``telescope-notifier.image.repository`` + - Image repository + - ``netrisai/xcaas-xcaas-notifier`` + * - ``telescope-notifier.image.tag`` + - Image tag. Overrides the image tag whose default is the chart appVersion + - ``""`` + * - ``telescope-notifier.image.pullPolicy`` + - Image pull policy + - ``IfNotPresent`` + * - ``telescope-notifier.imagePullSecrets`` + - Reference to one or more secrets to be used when pulling images + - ``[]`` + * - ``telescope-notifier.autoscaling.enabled`` + - Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics + - ``false`` + * - ``telescope-notifier.autoscaling.minReplicas`` + - Default min replicas for autoscaling + - ``1`` + * - ``telescope-notifier.autoscaling.maxReplicas`` + - Default max replicas for autoscaling + - ``100`` + * - ``telescope-notifier.autoscaling.targetCPUUtilizationPercentage`` + - The desired target CPU utilization for autoscaling + - ``80`` + + +Mariadb parameters +^^^^^^^^^^^^^^^^^^ + +*Using default values* `from `_ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``mariadb.image.repository`` + - MariaDB image name. We extended bitnami's mariadb image with own plugin + - ``netrisai/netris-mariadb`` + * - ``mariadb.image.tag`` + - MariaDB image tag. (only 10.1 is supported) + - ``10.1`` + * - ``mariadb.initdbScriptsConfigMap`` + - ConfigMap with the initdb scripts. + - ``netris-controller-initdb`` + * - ``mariadb.auth.database`` + - Name for a database to create + - ``netris`` + * - ``mariadb.auth.username`` + - Name for a user to create + - ``netris`` + * - ``mariadb.auth.password`` + - Password for the new user + - ``changeme`` + * - ``mariadb.auth.rootPassword`` + - Password for the root user + - ``changeme`` + + +*Auth from existing secret not supported at the momment* + +Mongodb parameters +^^^^^^^^^^^^^^^^^^ + +*Using default values* `from `_ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``mongodb.useStatefulSet`` + - Use StatefulSet instead of Deployment when deploying standalone + - ``true`` + * - ``mongodb.initdbScriptsConfigMap`` + - ConfigMap with the initdb scripts. + - ``netris-controller-initdb-mongodb`` + * - ``mongodb.auth.database`` + - Name for a database to create + - ``netris`` + * - ``mongodb.auth.username`` + - Name for a user to create + - ``netris`` + * - ``mongodb.auth.password`` + - Password for the new user + - ``changeme`` + * - ``mongodb.auth.rootPassword`` + - Password for the root user + - ``changeme`` + + +*Auth from existing secret not supported at the momment* + +Redis parameters +^^^^^^^^^^^^^^^^ + +*Using default values* `from `_ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``redis.cluster.enabled`` + - Use master-slave topology + - ``false`` + * - ``redis.usePassword`` + - Use password + - ``false`` + + +*Auth not supported at the momment* + +Smtp parameters +^^^^^^^^^^^^^^^ + +*Using default values* `from `_ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``smtp.config.DISABLE_IPV6`` + - Disable IPv6 + - ``1`` + * - ``smtp.config.RELAY_NETWORKS`` + - Relay networks. Change if your CNI use other subnets + - ``:172.16.0.0/12:10.0.0.0/8:192.168.0.0/16`` + + +HAproxy parameters +^^^^^^^^^^^^^^^^^^ + +*Using default values* `from `_ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``haproxy.enabled`` + - Enable HAProxy. Used for exposing netris agents ports from single loadbalancer ip. Disable if you can't have type:LoadBalancer service in cluster + - ``true`` + * - ``haproxy.service.type`` + - Kubernetes service type + - ``LoadBalancer`` + + +Graphite parameters +^^^^^^^^^^^^^^^^^^^ + +*Using default values* `from `_ + +.. list-table:: + :header-rows: 1 + + * - Parameter + - Description + - Default + * - ``graphite.configMaps`` + - Netris-Controller supported graphite's config files + - ``see in values.yaml`` + * - ``graphite.service.type`` + - Kubernetes service type + - ``ClusterIP`` + + +Usage +----- + +Specify each parameter using the --set key=value[,key=value] argument to helm install. For example, + +.. code-block:: + + helm install netris-controller netrisai/netris-controller \ + --namespace netris-controller \ + --set app.ingress.hosts={my.domain.com} \ + --set netris.authKey=$mystrongauthkey \ + --set mariadb.auth.rootPassword=my-root-password \ + --set mariadb.auth.password=my-password \ + --set mongodb.auth.rootPassword=my-root-password \ + --set mongodb.auth.password=my-password + +The above command sets netris-controller application ingress host to ``my.domain.com`` and sets generated netris.authKey. Additionally, it sets MariaDB and MongoDB root account password to ``my-root-password`` and user account password to ``my-password``. + +Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example, + +.. code-block:: + + helm install netris-controller netrisai/netris-controller --namespace netris-controller -f values.yaml + +After installation use ``EXTERNAL-IP`` of haproxy service as ``--controller`` parameter in `netris-setup `_ + +.. code-block:: + + kubectl get svc -nnetris-controller |grep haproxy + +and ``$mystrongauthkey`` as ``--auth`` parameter in `netris-setup `_ + +.. code-block:: + + echo $mystrongauthkey + +Also you can see overrides values from helm get values + +.. code-block:: + + helm get values netris-controller diff --git a/en/2.9/_sources/index.rst.txt b/en/2.9/_sources/index.rst.txt new file mode 100644 index 0000000000..10cabb380e --- /dev/null +++ b/en/2.9/_sources/index.rst.txt @@ -0,0 +1,49 @@ +.. Read the Docs Template documentation master file, created by + sphinx-quickstart on Tue Aug 26 14:19:49 2014. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to Netris documentation! +================================================== + +Netris is the automatic NetOps platform that runs the physical network and provides cloud-like user experience for NetOps and DevOps engineers. + +Contents: + +.. toctree:: + :maxdepth: 2 + :caption: Getting Started + + concepts + controller-installation + controller-k8s-installation + controller-initial-configuration + switch-agent-installation + SoftGate-agent-installation + +.. toctree:: + :maxdepth: 4 + :caption: Network Policies + + network-policies + +.. toctree:: + :maxdepth: 4 + :caption: Network Services + + services + +.. toctree:: + :maxdepth: 4 + :caption: Operations + + visibility + accounts + + +.. toctree:: + :maxdepth: 4 + :caption: Updates + + release-notes + diff --git a/en/2.9/_sources/network-policies.rst.txt b/en/2.9/_sources/network-policies.rst.txt new file mode 100644 index 0000000000..cbbd7cbfff --- /dev/null +++ b/en/2.9/_sources/network-policies.rst.txt @@ -0,0 +1,310 @@ +.. meta:: + :description: Netris Network Policies & Protocol Configuration + +######### +Basic BGP +######### + +BGP neighbors can be declared in the Net→E-BGP section. Netris will automatically generate and inject the right configuration to meet your requirements as declared. See below description of E-BGP neighbor declaration fields. + +* **Name** - Name for BGP session. +* **Description** - Free description. +* **Site** - Selects the site (data center) where this BGP session should be terminated on. +* **NFV Node** - Only if SoftGate nodes are in use, define on which node BGP session should be terminated on. +* **Neighbor AS** - Autonomous System number of the remote side. (Local AS is defined at Net→Sites section) +* **Terminate on switch** - Typically used for setups without SoftGate, for connecting with upstream routers. Instructs the system to terminate the BGP session directly on the switch. +* **Switch port** - Switch Port for the physical cable to the BGP neighbor. (any port on the fabric). Optionally can bind to a V-NET service, typically used for peering with IXPs or systems like GGC (Google Global Cache). +* **VLAN ID** - Optionally tag with a VLAN ID. (usually untagged) +* **IP Version** - IPv4 / IPv6 +* **Local IP** - BGP peering IP address on Netris controlled side. +* **Remote IP** - BGP peering IP address on the remote end. +* **State** - Administrative state. (Enabled/Disabled) +* **Advanced** - Advanced policy settings are described in the next section. + +Example: Declare a basic BGP neighbor. + +.. image:: images/BGP_neighbor.png + :align: center + + +############ +Advanced BGP +############ +BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies. + +Click Advanced to expand the BGP neighbor add/edit window. + +* **Neighbor address** - IP address of the neighbor when peering with the loopback IP address instead of the interface IP address. (aka Multihop). +* **Update source** - When Multihop BGP peering is used, it allows the operator to choose one of the loopback IP addresses of the SoftGate node as a BGP speaker source IP address. +* **BGP password** - Password for the BGP session. +* **Allowas-in** - Define the number of allowed occurrences of the self AS number in the received BGP NLRI to consider it valid. (normally 0) +* **Default Originate** - Originate default route to the current neighbor. +* **Prefix Inbound Max** - Drop the BGP session if the number of received prefixes exceeds this max limit. For switch termination maximum allowed is 1000 prefixes, while SoftGate termination can handle up to one million prefixes. +* **Inbound Route-Map** - Apply BGP policies described in a route-map for inbound BGP updates. +* **Outbound Route-Map** - Apply BGP policies described in a route-map for outbound BGP updates. +* **Local Preference** - Set local preference for all inbound routes for the current neighbor. +* **Weight** - Set weight for all inbound routes for the current neighbor. +* **Prepend Inbound(times)** - How many times to prepend self AS number for inbound routes. +* **Prepend Outbound(times)** - How many times to prepend self AS number for outbound routes. +* **Prefix List Inbound** - List of IP addresses prefixes to permit or deny inbound. +* **Prefix List Outbound** - List of IP addresses prefixes to permit or deny outbound. +* **Send BGP Community** - List of BGP communities to send to the current neighbor. + +BGP objects +----------- +| Under Net→E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy. +| Supported objects are: + +* IPv4 Prefix +* IPv6 Prefix +* AS-PATH +* Community +* Extended Community +* Large Community + +IPv4 Prefix. +^^^^^^^^^^^^ +| Rules defined one per line. +| Each line in IPv4 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv4 prefix (mandatory). +* Length - Possible values are: le , ge or ge le . + +Example: Creating an IPv4 Prefix list. + +.. image:: images/IPv4_Prefix.png + :align: center + +IPv6 Prefix. +^^^^^^^^^^^^ +| Rules defined one per line. +| Each line in IPv6 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv6 prefix (mandatory). +* Keyword - Possible values are: le , ge or ge le . + +Example: Creating an IPv6 Prefix list. + +.. image:: images/IPv6_Prefix.png + :align: center + +Community. +^^^^^^^^^^ +| Community field has two parts: + +* Action - Possible values: permit or deny (mandatory). +* Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive). + +Example: Creating community. + +.. image:: images/community.png + :align: center + +BGP route-maps +-------------- +| Under the Net→E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound. +| Description of route-map fields: + +* **Sequence Number** - Automatically assigned a sequence number. Drag and move sequences to organize the order. +* **Description** - Free description. +* **Policy** - Permit or deny the routes which match below all match clauses within the current sequence. +* **+Match** - Rules for route matching. + + * **Type** - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag. + * **Object** - Select an object from the list. + +* **Action** - Action when all match clauses are met. + + * **Action type** - Define whether to manipulate a particular BGP attribute or go to another sequence. + * **Attribute** - The attribute to be manipulated. + * **Value** - New attribute value. + +Example: route-map + +.. image:: images/route-map.png + :align: center + +####################### +Routes (static routing) +####################### +Located under Net→Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. +We recommend using the Routes only if BGP is not supported by the remote end. + +| Typical use cases for Routes +* To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported. +* Temporary interconnection with the old network for a migration. +* Routing a subnet behind a VM hypervisor machine for an internal VM network. +* Specifically routing traffic destined to a particular prefix through an out-of-band management network. + +| Add new static route fields description: +* **Prefix** - Route destination to match. +* **Next-Hop** - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network. +* **Description** - Free description. +* **Site*** - Site where Route belongs. +* **State** - Administrative (enable/disable) state of the Route. +* **+Apply to** - Limit the scope to particular units. It’s typically used for Null routes. + + +Example: Default route pointing to a Next-Hop that belongs to one of V-NETs. + +.. image:: images/defaultroute.png + :align: center + +Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network. + +.. image:: images/static_route.png + :align: center + +Screenshot: This Shows that my back route is actually applied on leaf1 and spine1. + +.. image:: images/leaf1_spine1.png + :align: center + +### +NAT +### +Netris SoftGate nodes are required to support NAT (Network Address Translation). + +Enabling NAT +------------ +To enable NAT for a given site, you first need to attach NAT IP addresses and/or NAT IP pool resources to SoftGate nodes. NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need, assuming it's configured as an allocation under Net→Subnets section. + +1. Allocate a public IP subnet for NAT under Net→Subnets. + +Example: Adding an IP allocation under Net→Subnets. + +.. image:: images/IP_allocation.png + :align: center + +2. Attach NAT IP addresses and/or NAT IP Pools to just one SoftGate node. Other SoftGate Nodes on the same site will automatically add the same NAT IP/Pool resources for proper consistency and high availability. + +Example: Adding NAT IP addresses and NAT IP Address Pools to a SoftGate node. + +.. image:: images/NATIP_address.png + :align: center + +Defining NAT rules +------------------ +NAT rules are defined under Net→NAT. + +NAT rule fields described: + +* **Name** - Unique name. +* **Protocol** + + * **All** - Match any IP protocol. + * **TCP** - Match TCP traffic and ports. + * **UDP** - Match UDP traffic and ports + * **ICMP** - Match ICMP traffic. + +* **Action** + + * **SNAT** - Replace the source IP address with specified NAT IP. + * **DNAT** - Replace the destination IP address and/or destination port with specified NAT IP. + * **ACCEPT** - Silently forward, typically used to add an exemption to broader SNAT or DNAT rule. + +* **Source** + + * **Address** - Source IP address to match. + * **From port** - Source ports to match starting with this value (TCP/UDP) + * **To port** - Source ports to much up to this value (TCP/UDP) + +* **Destination** + + * **Address** - Destination IP address to match. In the case of DNAT it should be one of the predefined NAT IP addresses. + * **Port** - For DNAT only, to match a single destination port. + * **From port** - For SNAT/ACCEPT only. Destination ports to match starting with this value (TCP/UDP) + * **To port** - For SNAT/ACCEPT only. Destination ports to much up to this value (TCP/UDP) + +* **NAT IP** - The global IP address for SNAT to be visible on Public Internet. The internal IP address for DNAT to replace the original destination address with. +* **Status** - Administrative state (enable/disable). +* **Comment** - Free optional comment. + +Example: SNAT all hosts on 10.0.0.0/8 to the Internet using 198.51.100.65 as a global IP. + +.. image:: images/globalIP.png + :align: center + +Example: Port forwarding. DNAT the traffic destined to 198.51.100.66:80 to be forwarded to the host 10.0.4.10 on port tcp/1080. + +.. image:: images/Port_Forwarding.png + :align: center + +SiteMesh +======== +SiteMesh is a Netris service for site-to-site interconnects over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site. + +Edit Net->Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below. + +* **Disabled** - Do not participate in SiteMesh. +* **Hub** - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites) +* **Spoke** - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites) +* **Dynamic Spoke** - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites) + +Screenshot: Site Mesh parameter editing a Site under Net→Sites. + +.. image:: images/Site_Mesh.png + :align: center + +You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge. + +.. image:: images/SiteMesh_modes.png + :align: center + +Check the Net→Site Mesh section for the listing of tunnel statuses. + +Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh) + +.. image:: images/SiteMesh_listing.png + :align: center + +############# +Looking Glass +############# +The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Net→Looking Glass then selecting the device from the top-left dropdown menu. + +Looking Glass controls described for IPv4/IPv6 protocol families. + +* **BGP Summary** - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes. +* **BGP Route** - Lookup the BGP table (RIB) for the given address. +* **Route** - Lookup switch routing table for the given address. +* **Traceroute** - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address. +* **Ping** - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address. + +Example: Spine1: listing BGP neighbors and number of received prefixes. + +.. image:: images/Spine1.png + :align: center + +Example: BGP Route - looking up my leaf1 switch’s loopback address from spine1’s perspective. Spine1 is load balancing between two available paths. + +.. image:: images/BGP_route.png + :align: center + +Example: Ping. + +.. image:: images/ping.png + :align: center + +| Looking Glass controls described for the EVPN family. +* **BGP Summary** - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received. +* **VNI** - List VNIs learned. +* **BGP EVPN** - List detailed EVPN routing information optionally for the given route distinguisher. +* **MAC table** - List MAC address table for the given VNI. + +Example: Listing of adjacent BGP neighbors and number of EVPN prefixes received. + +.. image:: images/BGP_neighbors_listing.png + :align: center + +Example: Listing MAC addresses on VNI 2. + +.. image:: images/MAC_listing.png + :align: center + +Example: EVPN routing information listing for a specified route distinguisher. + +.. image:: images/EVPN_routing.png + :align: center diff --git a/en/2.9/_sources/release-notes.rst.txt b/en/2.9/_sources/release-notes.rst.txt new file mode 100644 index 0000000000..f8b17164ca --- /dev/null +++ b/en/2.9/_sources/release-notes.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Netris Release Notes + +############# +Release notes +############# +* DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGateperformance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on. +* L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, wenow support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates withKubernetes providing cloud-like load balancer service (type: load-balancer). +* Kubenet​ - a network service purpose-built for Kubernetes cluster nodes.Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed tocomplement Kubernetes CNI networking with modern physical networking. +* API logs​ - Comprehensive logging of all API calls sent to Netris Controller withthe ability to search by various attributes, sort by each column, and filter bymethod type. +* SiteMesh​ - a Netris service for automatically configuring site-to-siteinterconnect over the public Internet. SiteMesh supports configuration forWireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a fewclicks, services in one site get connectivity to services in other sites over a meshof WireGuard tunnels. +* Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loopcable. Removed the need for IP address reservation for V-NET, switchingentirely to the anycast default gateway. +* Controller distributions​ - Netris controller, is now available in threedeployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application.3) Managed/Hosted in the cloud. +* Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes. +* Switch/SoftGate agents​ - New installer with easy initial config tool. Support forIP and FQDN as a controller address. Authentication key. +* GUI​ - Improved Net→Topology section, becoming the main and required placefor defining the network topology. All sections got a column organizer, so everyuser can order and hide/show columns to their comfort. diff --git a/en/2.9/_sources/sandbox/Sandbox1/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox1/configurations.rst.txt new file mode 100644 index 0000000000..666d6747c8 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox1/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s1-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox1.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.24 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox1/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox1/creating-services.rst.txt new file mode 100644 index 0000000000..fe1a86f68c --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox1/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s1-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s1-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox1.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s1-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox1.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1012``. + 10. In the **Local IP** field, type in ``45.38.161.22`` + 11. In the **Remote IP** field, type in ``45.38.161.21``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.0/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s1-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox1.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **45.38.161.6/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s1-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox1.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox1/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox1/index.rst.txt new file mode 100644 index 0000000000..0f3903929c --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox1/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox1 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt new file mode 100644 index 0000000000..63e78b0282 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox1 # sandbox name + 166.88.17.24 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1011 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1012 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.0/28 # customer public subnet + 45.38.161.10 # second usable ip address in load-balancer subnet + 45.38.161.11 # third usable ip address in load-balancer subnet + 45.38.161.18/30 # isp1-customer bgp peer local ip + 45.38.161.17/30 # isp1-customer bgp peer remote ip + 45.38.161.22/30 # isp2-customer bgp peer local ip + 45.38.161.21/30 # isp2-customer bgp peer remote ip + 45.38.161.6/32 # customer v-net nat ip + s1-pre-configured # LINKS + s1-learn-by-doing # LINKS + s1-e-bgp # LINKS + s1-v-net # LINKS + s1-nat # LINKS + s1-acl # LINKS + s1-k8s # LINKS + +.. _s1-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox1.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox1.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.10 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.10:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.10 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.10 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.10 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.10 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.10 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.11 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.11 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.11 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.11 + + SRV05-NYC + curl 45.38.161.11 + + SRV05-NYC + curl 45.38.161.11 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1012 + localIP: 45.38.161.22/30 + remoteIP: 45.38.161.21/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.0/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 45.38.161.22/30 45.38.161.21/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 45.38.161.22/30 45.38.161.21/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 45.38.161.22/30 45.38.161.21/30 7m59s + sandbox1-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox1-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox1-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 45.38.161.22/30 45.38.161.21/30 8m41s + sandbox1-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox1-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox1-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.10 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox1/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox1/sandbox-info.rst.txt new file mode 100644 index 0000000000..15e7ff5ea7 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox1/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox1.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.24 -p 30061 + srv02: ssh demo@166.88.17.24 -p 30062 + srv03: ssh demo@166.88.17.24 -p 30063 + srv04: ssh demo@166.88.17.24 -p 30064 + srv05: ssh demo@166.88.17.24 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1011 + IP customer: 45.38.161.18/30 + IP Iris: 45.38.161.17/30 + + Neighbor AS: 65007 + Vlan: 1012 + IP customer: 45.38.161.22/30 + IP Iris: 45.38.161.21/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 45.38.161.0/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox10/Configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox10/Configurations.rst.txt new file mode 100644 index 0000000000..5a698de606 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox10/Configurations.rst.txt @@ -0,0 +1,62 @@ +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the `step-by-step instructions ` + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting **https://sandbox10.netris.ai** and navigating to **Services > V-Net**, you will find a V-Net service named "**V-Net Example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net Example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select switch** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=33.4 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=27.1 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.702 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=1.37 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.609 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4008ms + rtt min/avg/max/mdev = 0.609/12.660/33.422/14.545 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the "**V-Net (Ethernet/Vlan/VXlan)**" section below. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**ISP1 Example**" configured as example with ISP1. This ensures communication of the inside network with the Internet. You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the "**ISP1 Example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with ISP2 yourself, please refer to the step-by-step instructions in the "**E-BGP (Exterior Border Gateway Protocol)**" section below. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network with the public **1.1.1.1** IP address. You can examine the particular settings of the NAT service by clicking **Edit** from the Actions menu indicated by three vertical dots on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.22 -p 23064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the "**NAT (Network Address Translation)**" section below. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network with the public **1.1.1.1** IP address is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the "**ACL (Access Control List)**" section below. diff --git a/en/2.9/_sources/sandbox/Sandbox10/Diagram.rst.txt b/en/2.9/_sources/sandbox/Sandbox10/Diagram.rst.txt new file mode 100644 index 0000000000..041ad3c041 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox10/Diagram.rst.txt @@ -0,0 +1,8 @@ +******** +Diagram +******** + +.. image:: ../images/sandbox_topology.png + :align: center + +Management Subnet: **10.254.45.0/24 (X=45)** Loopback Subnet: **10.254.46.0/24 (X=46)** diff --git a/en/2.9/_sources/sandbox/Sandbox10/Sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox10/Sandbox-info.rst.txt new file mode 100644 index 0000000000..51673ba188 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox10/Sandbox-info.rst.txt @@ -0,0 +1,85 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions. https://netris.ai/slack + +The credentials should be in the email response to your sandbox request. + +This environment includes: + +* Netris Controller: A cloud-hosted Netris controller, loaded with examples. +* Switching fabric: Two spine switches and four leaf switches, all Netris-operated. +* SoftGate: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* Linux servers: Four Linux servers, with root access where you can run any applications for your tests. +* Kubernetes cluster: A 3 node Kubernetes cluster, integrated with Netris controller, feel free to deploy any applications for your tests. +* ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + +Netris GUI +========== +http://sandbox10.netris.ai + +Linux servers +============= + +Example Netris services pre-configured: + srv01, srv02, srv03 - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + srv01, srv02 - are behing Anycast L3 load balancer. + srv04 - is consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.22 -p 23061 + srv02: ssh demo@166.88.17.22 -p 23062 + srv03: ssh demo@166.88.17.22 -p 23063 + srv04: ssh demo@166.88.17.22 -p 23064 + + +Kubernetes cluster +================== +This sandbox provides an up and running Kubernetes cluster. You can deploy any application that needs to expose a TCP port, or you can deploy your favorite ingress controller that needs to expose it's TCP port. Netris will process type:load-balancer automatically using it's L4 Load Balancer service. + +To access built-in Kubernetes cluster navigate to Services-->Kubenet in Netris GUI. You'll find a pre-configured example Kubenet service. Kubenet is a network service purpose built for serving to Kubernetes nodes. Edit the service and copy "Kubeconfig" content from the example service into your ".kube/config" on your local machine. Now you should be able to kubectl the sandbox cluster. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP Iris. + +ISP settings: + +.. code-block:: shell-session + + Customer subnet: 50.117.59.208/28 + + (pre-configured example) + Vlan: 801 + IP customer: 50.117.59.122/30 + IP Iris: 50.117.59.121/30 + + Vlan: 802 + IP customer: 50.117.59.126/30 + IP Iris: 50.117.59.125/30 + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + Public subnet: 50.117.59.208/28 diff --git a/en/2.9/_sources/sandbox/Sandbox10/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox10/creating-services.rst.txt new file mode 100644 index 0000000000..110579c661 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox10/creating-services.rst.txt @@ -0,0 +1,126 @@ +************************** +Learn by Creating Services +************************** +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv04-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv04-nyc** server by typing ``ssh demo@166.88.17.22 -p 23064``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth2 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting **http://sandbox10.netris.ai** and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv04-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can find the required port by typing "``srv04``" in the Search field. + + 7. Select the port named **swp2(sw21-nyc-swp2 (srv04))@sw21-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with "**ISP1**". + +Optionally you can configure an E-BGP session to ISP 2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting **https://sandbox10.netris.ai** and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``ISP2 Customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``100``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can find the required port by typing "``ISP2``" in the Search field. + + 8. For the **VLAN ID** field, unselect ``Untag`` and type in ``802``. + 9. In the **Local IP** field, type in ``50.117.59.126`` + 10. In the **Remote IP** field, type in ``50.117.59.127``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 50.117.59.208/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv04-nyc** server to be able to communicate with **8.8.8.8**. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.22 -p 23064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 8.8.8.8`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with public IP **8.8.8.8**. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting **https://sandbox10.netris.ai** and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.213/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 8.8.8.8: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the **8.8.8.8** address is reachable. + +ACL (Access Control List) +========================= +Now that **srv02-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.22 -p 23064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 8.8.8.8`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting **https://sandbox10.netris.ai** and navigate to **Net > Sites**. + 2. Click **Edit** from the Actions menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 8.8.8.8`` command, indicating that the **8.8.8.8** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv02-nyc** server to communicate with **8.8.8.8**. + +* Back in the web browser: + + 1. Navigate to **Services > ACL** + 2. Click **+Add** to define a new ACL + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +| +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 8.8.8.8`` command have resumed, indicating that the **srv02-nyc** server can communicate with **8.8.8.8** once again. diff --git a/en/2.9/_sources/sandbox/Sandbox10/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox10/index.rst.txt new file mode 100644 index 0000000000..5eec9ac4aa --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox10/index.rst.txt @@ -0,0 +1,9 @@ +Sandbox10 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + Sandbox-info + creating-services diff --git a/en/2.9/_sources/sandbox/Sandbox2/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox2/configurations.rst.txt new file mode 100644 index 0000000000..add958bdad --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox2/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s2-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox2.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.190 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox2/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox2/creating-services.rst.txt new file mode 100644 index 0000000000..02a9b57479 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox2/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s2-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s2-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox2.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s2-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox2.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1022``. + 10. In the **Local IP** field, type in ``45.38.161.30`` + 11. In the **Remote IP** field, type in ``45.38.161.29``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.32/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s2-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox2.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **45.38.161.38/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s2-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox2.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox2/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox2/index.rst.txt new file mode 100644 index 0000000000..dfcf08d016 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox2/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox2 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt new file mode 100644 index 0000000000..a16e449700 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox2 # sandbox name + 166.88.17.190 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1021 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1022 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.32/28 # customer public subnet + 45.38.161.42 # second usable ip address in load-balancer subnet + 45.38.161.43 # third usable ip address in load-balancer subnet + 45.38.161.26/30 # isp1-customer bgp peer local ip + 45.38.161.25/30 # isp1-customer bgp peer remote ip + 45.38.161.30/30 # isp2-customer bgp peer local ip + 45.38.161.29/30 # isp2-customer bgp peer remote ip + 45.38.161.38/32 # customer v-net nat ip + s2-pre-configured # LINKS + s2-learn-by-doing # LINKS + s2-e-bgp # LINKS + s2-v-net # LINKS + s2-nat # LINKS + s2-acl # LINKS + s2-k8s # LINKS + +.. _s2-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox2.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox2.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.42 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.42:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.42 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.42 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.42 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.42 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.42 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.43 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.43 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.43 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.43 + + SRV05-NYC + curl 45.38.161.43 + + SRV05-NYC + curl 45.38.161.43 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1022 + localIP: 45.38.161.30/30 + remoteIP: 45.38.161.29/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.32/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 45.38.161.30/30 45.38.161.29/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 45.38.161.30/30 45.38.161.29/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 45.38.161.30/30 45.38.161.29/30 7m59s + sandbox2-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox2-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox2-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 45.38.161.30/30 45.38.161.29/30 8m41s + sandbox2-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox2-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox2-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.42 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox2/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox2/sandbox-info.rst.txt new file mode 100644 index 0000000000..55f3696f32 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox2/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox2.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.190 -p 30061 + srv02: ssh demo@166.88.17.190 -p 30062 + srv03: ssh demo@166.88.17.190 -p 30063 + srv04: ssh demo@166.88.17.190 -p 30064 + srv05: ssh demo@166.88.17.190 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1021 + IP customer: 45.38.161.26/30 + IP Iris: 45.38.161.25/30 + + Neighbor AS: 65007 + Vlan: 1022 + IP customer: 45.38.161.30/30 + IP Iris: 45.38.161.29/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 45.38.161.32/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox3/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox3/configurations.rst.txt new file mode 100644 index 0000000000..78eb8a85b2 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox3/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s3-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox3.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.189 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox3/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox3/creating-services.rst.txt new file mode 100644 index 0000000000..636561f0c5 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox3/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s3-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s3-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox3.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s3-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox3.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1032``. + 10. In the **Local IP** field, type in ``45.38.161.70`` + 11. In the **Remote IP** field, type in ``45.38.161.69``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.48/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s3-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox3.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **45.38.161.54/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s3-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox3.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox3/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox3/index.rst.txt new file mode 100644 index 0000000000..4d717deebf --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox3/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox3 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt new file mode 100644 index 0000000000..ccf4ba9a5c --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox3 # sandbox name + 166.88.17.189 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1031 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1032 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.48/28 # customer public subnet + 45.38.161.58 # second usable ip address in load-balancer subnet + 45.38.161.59 # third usable ip address in load-balancer subnet + 45.38.161.66/30 # isp1-customer bgp peer local ip + 45.38.161.65/30 # isp1-customer bgp peer remote ip + 45.38.161.70/30 # isp2-customer bgp peer local ip + 45.38.161.69/30 # isp2-customer bgp peer remote ip + 45.38.161.54/32 # customer v-net nat ip + s3-pre-configured # LINKS + s3-learn-by-doing # LINKS + s3-e-bgp # LINKS + s3-v-net # LINKS + s3-nat # LINKS + s3-acl # LINKS + s3-k8s # LINKS + +.. _s3-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox3.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox3.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.58 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.58:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.58 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.58 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.58 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.58 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.58 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.59 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.59 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.59 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.59 + + SRV05-NYC + curl 45.38.161.59 + + SRV05-NYC + curl 45.38.161.59 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1032 + localIP: 45.38.161.70/30 + remoteIP: 45.38.161.69/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.48/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 45.38.161.70/30 45.38.161.69/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 45.38.161.70/30 45.38.161.69/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 45.38.161.70/30 45.38.161.69/30 7m59s + sandbox3-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox3-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox3-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 45.38.161.70/30 45.38.161.69/30 8m41s + sandbox3-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox3-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox3-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.58 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox3/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox3/sandbox-info.rst.txt new file mode 100644 index 0000000000..141289cd96 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox3/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox3.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.189 -p 30061 + srv02: ssh demo@166.88.17.189 -p 30062 + srv03: ssh demo@166.88.17.189 -p 30063 + srv04: ssh demo@166.88.17.189 -p 30064 + srv05: ssh demo@166.88.17.189 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1031 + IP customer: 45.38.161.66/30 + IP Iris: 45.38.161.65/30 + + Neighbor AS: 65007 + Vlan: 1032 + IP customer: 45.38.161.70/30 + IP Iris: 45.38.161.69/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 45.38.161.48/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox4/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox4/configurations.rst.txt new file mode 100644 index 0000000000..94bad1944c --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox4/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s4-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox4.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.188 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox4/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox4/creating-services.rst.txt new file mode 100644 index 0000000000..3d122d2ba7 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox4/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s4-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s4-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox4.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s4-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox4.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1042``. + 10. In the **Local IP** field, type in ``45.38.161.78`` + 11. In the **Remote IP** field, type in ``45.38.161.77``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.80/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s4-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox4.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **45.38.161.86/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s4-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox4.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox4/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox4/index.rst.txt new file mode 100644 index 0000000000..6017417ef8 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox4/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox4 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt new file mode 100644 index 0000000000..76858c5ef9 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox4 # sandbox name + 166.88.17.188 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1041 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1042 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.80/28 # customer public subnet + 45.38.161.90 # second usable ip address in load-balancer subnet + 45.38.161.91 # third usable ip address in load-balancer subnet + 45.38.161.74/30 # isp1-customer bgp peer local ip + 45.38.161.73/30 # isp1-customer bgp peer remote ip + 45.38.161.78/30 # isp2-customer bgp peer local ip + 45.38.161.77/30 # isp2-customer bgp peer remote ip + 45.38.161.86/32 # customer v-net nat ip + s4-pre-configured # LINKS + s4-learn-by-doing # LINKS + s4-e-bgp # LINKS + s4-v-net # LINKS + s4-nat # LINKS + s4-acl # LINKS + s4-k8s # LINKS + +.. _s4-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox4.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox4.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.90 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.90:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.90 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.90 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.90 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.90 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.90 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.91 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.91 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.91 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.91 + + SRV05-NYC + curl 45.38.161.91 + + SRV05-NYC + curl 45.38.161.91 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1042 + localIP: 45.38.161.78/30 + remoteIP: 45.38.161.77/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.80/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 45.38.161.78/30 45.38.161.77/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 45.38.161.78/30 45.38.161.77/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 45.38.161.78/30 45.38.161.77/30 7m59s + sandbox4-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox4-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox4-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 45.38.161.78/30 45.38.161.77/30 8m41s + sandbox4-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox4-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox4-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.90 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox4/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox4/sandbox-info.rst.txt new file mode 100644 index 0000000000..12955859d1 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox4/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox4.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.188 -p 30061 + srv02: ssh demo@166.88.17.188 -p 30062 + srv03: ssh demo@166.88.17.188 -p 30063 + srv04: ssh demo@166.88.17.188 -p 30064 + srv05: ssh demo@166.88.17.188 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1041 + IP customer: 45.38.161.74/30 + IP Iris: 45.38.161.73/30 + + Neighbor AS: 65007 + Vlan: 1042 + IP customer: 45.38.161.78/30 + IP Iris: 45.38.161.77/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 45.38.161.80/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox5/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox5/configurations.rst.txt new file mode 100644 index 0000000000..74b27d98a3 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox5/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s5-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox5.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.187 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox5/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox5/creating-services.rst.txt new file mode 100644 index 0000000000..ae9e23922f --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox5/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s5-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s5-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox5.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s5-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox5.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1052``. + 10. In the **Local IP** field, type in ``50.117.59.86`` + 11. In the **Remote IP** field, type in ``50.117.59.85``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.128/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s5-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox5.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.134/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s5-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox5.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox5/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox5/index.rst.txt new file mode 100644 index 0000000000..b140b5ceae --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox5/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox5 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt new file mode 100644 index 0000000000..0fc959f667 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox5 # sandbox name + 166.88.17.187 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1051 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1052 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.128/28 # customer public subnet + 50.117.59.138 # second usable ip address in load-balancer subnet + 50.117.59.139 # third usable ip address in load-balancer subnet + 50.117.59.82/30 # isp1-customer bgp peer local ip + 50.117.59.81/30 # isp1-customer bgp peer remote ip + 50.117.59.86/30 # isp2-customer bgp peer local ip + 50.117.59.85/30 # isp2-customer bgp peer remote ip + 50.117.59.134/32 # customer v-net nat ip + s5-pre-configured # LINKS + s5-learn-by-doing # LINKS + s5-e-bgp # LINKS + s5-v-net # LINKS + s5-nat # LINKS + s5-acl # LINKS + s5-k8s # LINKS + +.. _s5-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox5.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox5.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.138 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.138:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.138 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.138 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.138 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.138 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.138 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.139 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.139 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.139 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.139 + + SRV05-NYC + curl 50.117.59.139 + + SRV05-NYC + curl 50.117.59.139 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1052 + localIP: 50.117.59.86/30 + remoteIP: 50.117.59.85/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.128/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.86/30 50.117.59.85/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.86/30 50.117.59.85/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.86/30 50.117.59.85/30 7m59s + sandbox5-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox5-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox5-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.86/30 50.117.59.85/30 8m41s + sandbox5-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox5-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox5-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.138 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox5/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox5/sandbox-info.rst.txt new file mode 100644 index 0000000000..3eb48dfbc8 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox5/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox5.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.187 -p 30061 + srv02: ssh demo@166.88.17.187 -p 30062 + srv03: ssh demo@166.88.17.187 -p 30063 + srv04: ssh demo@166.88.17.187 -p 30064 + srv05: ssh demo@166.88.17.187 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1051 + IP customer: 50.117.59.82/30 + IP Iris: 50.117.59.81/30 + + Neighbor AS: 65007 + Vlan: 1052 + IP customer: 50.117.59.86/30 + IP Iris: 50.117.59.85/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 50.117.59.128/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox6/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox6/configurations.rst.txt new file mode 100644 index 0000000000..219e0222f8 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox6/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s6-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.186 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox6/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox6/creating-services.rst.txt new file mode 100644 index 0000000000..c5e13300d8 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox6/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s6-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s6-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s6-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1062``. + 10. In the **Local IP** field, type in ``50.117.59.94`` + 11. In the **Remote IP** field, type in ``50.117.59.93``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.144/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s6-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.150/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s6-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox6/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox6/index.rst.txt new file mode 100644 index 0000000000..1481cb647a --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox6/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox6 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt new file mode 100644 index 0000000000..586d4924d3 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox6 # sandbox name + 166.88.17.186 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1061 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1062 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.144/28 # customer public subnet + 50.117.59.154 # second usable ip address in load-balancer subnet + 50.117.59.155 # third usable ip address in load-balancer subnet + 50.117.59.90/30 # isp1-customer bgp peer local ip + 50.117.59.89/30 # isp1-customer bgp peer remote ip + 50.117.59.94/30 # isp2-customer bgp peer local ip + 50.117.59.93/30 # isp2-customer bgp peer remote ip + 50.117.59.150/32 # customer v-net nat ip + s6-pre-configured # LINKS + s6-learn-by-doing # LINKS + s6-e-bgp # LINKS + s6-v-net # LINKS + s6-nat # LINKS + s6-acl # LINKS + s6-k8s # LINKS + +.. _s6-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox6.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox6.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.154 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.154:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.154 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.155 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.155 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.155 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1062 + localIP: 50.117.59.94/30 + remoteIP: 50.117.59.93/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.144/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.94/30 50.117.59.93/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.94/30 50.117.59.93/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.94/30 50.117.59.93/30 7m59s + sandbox6-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox6-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox6-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.94/30 50.117.59.93/30 8m41s + sandbox6-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox6-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox6-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.154 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox6/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox6/sandbox-info.rst.txt new file mode 100644 index 0000000000..e7c71c4572 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox6/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox6.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.186 -p 30061 + srv02: ssh demo@166.88.17.186 -p 30062 + srv03: ssh demo@166.88.17.186 -p 30063 + srv04: ssh demo@166.88.17.186 -p 30064 + srv05: ssh demo@166.88.17.186 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1061 + IP customer: 50.117.59.90/30 + IP Iris: 50.117.59.89/30 + + Neighbor AS: 65007 + Vlan: 1062 + IP customer: 50.117.59.94/30 + IP Iris: 50.117.59.93/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 50.117.59.144/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox7/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox7/configurations.rst.txt new file mode 100644 index 0000000000..1ca22ca096 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox7/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s7-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox7.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.185 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox7/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox7/creating-services.rst.txt new file mode 100644 index 0000000000..be7161745c --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox7/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s7-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s7-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox7.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s7-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox7.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1072``. + 10. In the **Local IP** field, type in ``50.117.59.102`` + 11. In the **Remote IP** field, type in ``50.117.59.101``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.160/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s7-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox7.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.166/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s7-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox7.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox7/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox7/index.rst.txt new file mode 100644 index 0000000000..4a807f75e9 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox7/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox7 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt new file mode 100644 index 0000000000..8c62a07d29 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox7 # sandbox name + 166.88.17.185 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1071 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1072 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.160/28 # customer public subnet + 50.117.59.170 # second usable ip address in load-balancer subnet + 50.117.59.171 # third usable ip address in load-balancer subnet + 50.117.59.98/30 # isp1-customer bgp peer local ip + 50.117.59.97/30 # isp1-customer bgp peer remote ip + 50.117.59.102/30 # isp2-customer bgp peer local ip + 50.117.59.101/30 # isp2-customer bgp peer remote ip + 50.117.59.166/32 # customer v-net nat ip + s7-pre-configured # LINKS + s7-learn-by-doing # LINKS + s7-e-bgp # LINKS + s7-v-net # LINKS + s7-nat # LINKS + s7-acl # LINKS + s7-k8s # LINKS + +.. _s7-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox7.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox7.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.170 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.170:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.170 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.170 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.170 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.170 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.170 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.171 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.171 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.171 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.171 + + SRV05-NYC + curl 50.117.59.171 + + SRV05-NYC + curl 50.117.59.171 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1072 + localIP: 50.117.59.102/30 + remoteIP: 50.117.59.101/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.160/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.102/30 50.117.59.101/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.102/30 50.117.59.101/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.102/30 50.117.59.101/30 7m59s + sandbox7-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox7-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox7-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.102/30 50.117.59.101/30 8m41s + sandbox7-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox7-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox7-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.170 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox7/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox7/sandbox-info.rst.txt new file mode 100644 index 0000000000..8e11de4214 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox7/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox7.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.185 -p 30061 + srv02: ssh demo@166.88.17.185 -p 30062 + srv03: ssh demo@166.88.17.185 -p 30063 + srv04: ssh demo@166.88.17.185 -p 30064 + srv05: ssh demo@166.88.17.185 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1071 + IP customer: 50.117.59.98/30 + IP Iris: 50.117.59.97/30 + + Neighbor AS: 65007 + Vlan: 1072 + IP customer: 50.117.59.102/30 + IP Iris: 50.117.59.101/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 50.117.59.160/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox8/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox8/configurations.rst.txt new file mode 100644 index 0000000000..455cfa30e2 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox8/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s8-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox8.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.42.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.41.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.41.64 + PING 192.168.41.64 (192.168.41.64) 56(84) bytes of data. + 64 bytes from 192.168.41.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.41.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.41.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.41.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.41.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.41.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.41.0/24** network with the public **1.1.1.1** IP address. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.29 -p 22864``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.41.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox8/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox8/creating-services.rst.txt new file mode 100644 index 0000000000..7df5fde7b4 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox8/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s8-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s8-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.29 -p 22865``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.42.1`` is configured as the default gateway, indicated by the "**default via 192.168.42.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.42.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.42.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox8.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.42.0/24(CUSTOMER)`` and IP ``192.168.42.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.42.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s8-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox8.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1082``. + 10. In the **Local IP** field, type in ``50.117.59.110`` + 11. In the **Remote IP** field, type in ``50.117.59.109``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.176/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s8-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.29 -p 22865``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.42.0/24** can communicate with public IP **1.1.1.1**. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox8.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.42.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.182/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s8-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.29 -p 22865``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox8.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.42.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox8/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox8/index.rst.txt new file mode 100644 index 0000000000..42a8419ba2 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox8/index.rst.txt @@ -0,0 +1,10 @@ +Sandbox8 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services diff --git a/en/2.9/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt new file mode 100644 index 0000000000..9898384dac --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt @@ -0,0 +1,630 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox8 # sandbox name + 50.117.59.186 # second usable ip address in load-balancer subnet + 50.117.59.187 # third usable ip address in load-balancer subnet + 192.168.41.64 # srv4 ip address + 192.168.42.65 # srv5 ip address + 192.168.42.1 # vnet-customer gateway + 65007 # Iris AS number bgp peer, SHORT QUERY BE CAREFUL WHILE REPLACING + 1082 # Iris peer vlanid, SHORT QUERY BE CAREFUL WHILE REPLACING + 50.117.59.110/30 # isp2-customer bgp peer local ip + 50.117.59.109/30 # isp2-customer bgp peer remote ip + 50.117.59.176/28 # sandbox public subnet + 192.168.108. # k8s vnet subnet + s8-e-bgp # LINKS + s8-v-net # LINKS + s8-acl # LINKS + s8-k8s # LINKS + +.. _s8-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox8.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox8.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.186 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.186:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.186 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.186 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.186 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.41.64:80 + - 192.168.42.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.186 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.186 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.187 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.187 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.42.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.42.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.187 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.187 + + SRV05-NYC + curl 50.117.59.187 + + SRV05-NYC + curl 50.117.59.187 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.42.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1082 + localIP: 50.117.59.110/30 + remoteIP: 50.117.59.109/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.176/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.110/30 50.117.59.109/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.110/30 50.117.59.109/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.110/30 50.117.59.109/30 7m59s + sandbox8-srv06-nyc-192.168.108.66 enabled 4200070000 192.168.108.1/24 192.168.108.66/24 26s + sandbox8-srv07-nyc-192.168.108.67 enabled 4200070001 192.168.108.1/24 192.168.108.67/24 26s + sandbox8-srv08-nyc-192.168.108.68 enabled 4200070002 192.168.108.1/24 192.168.108.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.110/30 50.117.59.109/30 8m41s + sandbox8-srv06-nyc-192.168.108.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.108.1/24 192.168.108.66/24 68s + sandbox8-srv07-nyc-192.168.108.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.108.1/24 192.168.108.67/24 68s + sandbox8-srv08-nyc-192.168.108.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.108.1/24 192.168.108.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.186 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox8/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox8/sandbox-info.rst.txt new file mode 100644 index 0000000000..634f72dec0 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox8/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox8.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.29 -p 22861 + srv02: ssh demo@166.88.17.29 -p 22862 + srv03: ssh demo@166.88.17.29 -p 22863 + srv04: ssh demo@166.88.17.29 -p 22864 + srv05: ssh demo@166.88.17.29 -p 22865 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1081 + IP customer: 50.117.59.106/30 + IP Iris: 50.117.59.105/30 + + Neighbor AS: 65007 + Vlan: 1082 + IP customer: 50.117.59.110/30 + IP Iris: 50.117.59.109/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.41.0/24 + Loopback subnet: 10.254.42.0/24 + Example subnet: 192.168.41.0/24 + Customer subnet: 192.168.42.0/24 + K8s subnet: 192.168.108.0/24 + Public subnet: 50.117.59.176/28 + diff --git a/en/2.9/_sources/sandbox/Sandbox9/configurations.rst.txt b/en/2.9/_sources/sandbox/Sandbox9/configurations.rst.txt new file mode 100644 index 0000000000..33bfc70f2c --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox9/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s9-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox9.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.19 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/2.9/_sources/sandbox/Sandbox9/creating-services.rst.txt b/en/2.9/_sources/sandbox/Sandbox9/creating-services.rst.txt new file mode 100644 index 0000000000..e8d0a75c8d --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox9/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s9-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s9-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox9.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s9-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox9.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1092``. + 10. In the **Local IP** field, type in ``50.117.59.118`` + 11. In the **Remote IP** field, type in ``50.117.59.117``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.192/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s9-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox9.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.198/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s9-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox9.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/2.9/_sources/sandbox/Sandbox9/index.rst.txt b/en/2.9/_sources/sandbox/Sandbox9/index.rst.txt new file mode 100644 index 0000000000..325e04c6fc --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox9/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox9 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/2.9/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt b/en/2.9/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt new file mode 100644 index 0000000000..0272f1c6e2 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox9 # sandbox name + 166.88.17.19 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1091 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1092 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.192/28 # customer public subnet + 50.117.59.202 # second usable ip address in load-balancer subnet + 50.117.59.203 # third usable ip address in load-balancer subnet + 50.117.59.114/30 # isp1-customer bgp peer local ip + 50.117.59.113/30 # isp1-customer bgp peer remote ip + 50.117.59.118/30 # isp2-customer bgp peer local ip + 50.117.59.117/30 # isp2-customer bgp peer remote ip + 50.117.59.198/32 # customer v-net nat ip + s9-pre-configured # LINKS + s9-learn-by-doing # LINKS + s9-e-bgp # LINKS + s9-v-net # LINKS + s9-nat # LINKS + s9-acl # LINKS + s9-k8s # LINKS + +.. _s9-k8s: + +************************************************************** +Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps +************************************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox9.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox9.netris.ai/' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.202 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.202:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.202 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.202 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.202 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.203 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.203 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.203 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.203 + + SRV05-NYC + curl 50.117.59.203 + + SRV05-NYC + curl 50.117.59.203 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1092 + localIP: 50.117.59.118/30 + remoteIP: 50.117.59.117/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.192/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.118/30 50.117.59.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.118/30 50.117.59.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.118/30 50.117.59.117/30 7m59s + sandbox9-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox9-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox9-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.118/30 50.117.59.117/30 8m41s + sandbox9-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox9-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox9-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.202 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/2.9/_sources/sandbox/Sandbox9/sandbox-info.rst.txt b/en/2.9/_sources/sandbox/Sandbox9/sandbox-info.rst.txt new file mode 100644 index 0000000000..fe7c6e1985 --- /dev/null +++ b/en/2.9/_sources/sandbox/Sandbox9/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +http://sandbox9.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.19 -p 30061 + srv02: ssh demo@166.88.17.19 -p 30062 + srv03: ssh demo@166.88.17.19 -p 30063 + srv04: ssh demo@166.88.17.19 -p 30064 + srv05: ssh demo@166.88.17.19 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1091 + IP customer: 50.117.59.114/30 + IP Iris: 50.117.59.113/30 + + Neighbor AS: 65007 + Vlan: 1092 + IP customer: 50.117.59.118/30 + IP Iris: 50.117.59.117/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 50.117.59.192/28 + diff --git a/en/2.9/_sources/services.rst.txt b/en/2.9/_sources/services.rst.txt new file mode 100644 index 0000000000..29979d7af7 --- /dev/null +++ b/en/2.9/_sources/services.rst.txt @@ -0,0 +1,349 @@ +.. meta:: + :description: Netris Services and Configuration Examples + +##### +V-NET +##### +V-NET is a virtual networking service. V-NETs can be used for Layer-2 (unrouted) or Layer-3 (routed) virtual network segments involving switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization). + + Automatically, Netris will configure a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes. + + +V-Net fields. + +- **Name** - Unique name for the V-NET. +- **Owner** - Tenant, who can make any changes to current V-NET. +- **V-Net state** - Active/Disable state for entire V-NET. +- **VLAN aware** - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible. +- **Guest tenants** - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters. +- **Sites** - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites). +- **IPv4 Gateway** - IPv4 address to be used as a default gateway in this V-NET. Should be configured under Net→Subnets as an assignment, assigned to the owner tenant, and available in the site where V-NET is intended to span. +- **IPv6 Gateway** - IPv6 address to be used as a default gateway in this V-NET. Should be configured under Net→Subnets as an assignment, assigned to the owner tenant, and available in the site or sites where V-NET is intended to span. +- **Port** - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports. + + - **Enabled** - Enable or disable individual Switch Port under current V-NET + - **Port Name** - Switch Port format: (swp)@ + - **VLAN ID / Untag** - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port’s ingress/egress unless VLAN aware mode is used. + - **LAG Mode** - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes). + +Tip: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +Example: Adding a new V-NET. + +.. image:: images/new_VNET.png + :align: center + + +Example: Listing of V-NETs. + +.. image:: images/VNETs.png + :align: center + + +Example: Expanded view of a V-NET listing. + +.. image:: images/VNET_listing.png + :align: center + +####### +Kubenet +####### +Kubenet is a network service purpose-built for Kubernetes cluster nodes. Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters. + +The Gateway and Switch Port part of Kubenet is similar to the V-NET. In fact, it is leveraging a V-NET. Kubeconfig is for granting Netris Controller access to your Kube API. Kubenet therefore, dynamically leverages Netris L4LB and other services based on events that Netris kube-watcher (Kube API integration adapter) watches in your Kube API. + +Description of Kubenet fields. + +- **Name** - Unique name for the Kubenet. +- **Tenant** - Tenant, who can make any changes to current Kubenet. +- **Site** - Site where Kubernetes cluster belongs. +- **State** - Active/Disable state for particular Kubenet service. +- **IPv4** Gateway - IPv4 address to be used as a default gateway for current Kubenet. +- **Port** - Physical Switch Port anywhere on the switch fabric. Switch Port should be assigned to the owner tenant under Net→Switch Ports. + + - **Enabled** - Enable or disable individual Switch Port under current Kubenet. + - **Port Name** - Switch Port format: (swp)@ + - **VLAN ID / Untag** - Specify a VLAN ID for tagging traffic on a per-port basis or set to Untag not to use tagging on a particular port. + +- **Kubeconfig** - After installing the Kubernetes cluster, add your Kube config for granting Netris at least read-only access to the Kube API. + +Tip: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +Example: Adding a new Kubenet service. + +.. image:: images/new_kubenet.png + :align: center + +Once Netris Controller establishes a connection with Kube API, status will reflect on the listing. + + +Screenshot: Listing of Kubenet services. Kube API connection is successful. + +.. image:: images/listing_kubenet.png + :align: center + + +Screenshot: Physical Switch Port statuses. + +.. image:: images/switch_port_statuses.png + :align: center + + +Screenshot: Statuses of on-demand load balancers (type: load-balancer) + +.. image:: images/load-balancer.png + :align: center + + +######################### +ROH (Routing on the Host) +######################### +To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly to their servers. Known as ROH (Routing on the Host). + +In ROH architectures, servers use a routing demon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon is FRR. + +Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. Thus, leveraging the Layer-3 network throughout the entire network down the servers. + +ROH architecture with Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). +For each instance of ROH, you’ll need to create an ROH entry in Netris Controller. + +Description of ROH instance fields: + +- **Name** - Unique name for the ROH instance. +- **Site** - Site where the current ROH instance belongs. +- **Type** - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor. +- **ROH Routing Profile** - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances. + + - **Default route only (a most common choice)** - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch. + - **Default + Aggregate** - Will add prefixes of defined assignments + "Default" profile. + - **Full table** - Will advertise all prefixes available in the routing table of the connected switch. + - **Inherit** - will inherit policy from site objects defined under Net→Sites. + +- **Legacy Mode** - Switch from default zero-config mode to using /30 IP addresses. Use for MS Windows Servers or other OS that doesn’t support FRR. +- **+Port** - Physical Switch Ports anywhere on the network. +- **+IPv4** - IPv4 addresses for the loopback interface. +- **+Inbound Prefix List** - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks. + +Tip: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +Example: Adding an ROH instance. (Yes, you can use A.B.C.0/32 and A.B.C.255/32) + +.. image:: images/ROH_instance.png + :align: center + + +Screenshot: Expanded view of ROH listing. BGP sessions are up, and the expected IP is in fact received from the actual ROH server. Traffic stats are available per port. + +.. image:: images/ROH_listing.png + :align: center + + +############################# +L3 Load Balancer (Anycast LB) +############################# +L3 (Anycast) load balancer is leveraging ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks. + +ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server. + +End-user traffic should be destined to the anycast IP address. Switch fabric will ECMP load balance the traffic towards every server, as well as will hash based on IP/Protocol/Port such that TCP sessions will keep complete between given end-user and server pair. Optionally health checks are available to reroute the traffic away in the event of application failure. + +To configure L3 (Anycast) load balancing, edit an existing ROH instance entry and add an extra IPv4 address, and select Anycast. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances. + + +Example: Adding an Anycast IPv4 address + +.. image:: images/anycast_IPv4_address.png + :align: center + + +Example: Under Services→Load Balancer, you can find the listing of L3 (Anycast) Load Balancers, service statuses, and you can add/remove more ROH instances and/or health checks. + +.. image:: images/listing_L3.png + :align: center + + +Screenshot: L3 (Anycast) Load Balancer listing. + + +.. image:: images/loadbalancer_listing.png + :align: center + + +####################### +L4 Load Balancer (L4LB) +####################### +Netris L4 Load Balancer (L4LB) is leveraging SoftGate(Linux router) nodes for providing Layer-4 load balancing service, including on-demand cloud load balancer with native integration with Kubernetes. + +Enabling L4LB service +--------------------- +L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer). + +The IP address pool for L4LB can be defined in the Net→Subnets section by adding an Allocation and setting the purpose field to ‘load-balancer.’ You can define multiple IP pools for L4LB at any given Site. See the below example. + +Example: Adding a load-balancer IP pool assignment. + +.. image:: images/IP_pool_assignment.png + :align: center + + +Screenshot: Listing of Net→Subnets after adding a load-balancer assignment + +.. image:: images/NetSubnets_listing.png + :align: center + + +Consuming L4LB service +---------------------- +This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section. + +Click +add under Services→L4 Load Balancer to request an L4LB service. + +Add new L4 Load Balancer fields are described below: + +**General fields** + +* **Name*** - Unique name. +* **Protocol*** - TCP or UDP. +* **Tenant*** - Requestor Tenant should have access to the backend IP space. +* **Site*** - Site where L4LB service is being requested for. Backends should belong on this site. +* **State*** - Administrative state. + +**Frontend** + +* **Address*** - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses. +* **Port*** - TCP or UDP port to be exposed. + +**Health-check** + +* **Type*** - Probe backends on service availability. + + * **None** - load balance unconditionally. + * **TCP** - probe backend service availability through TCP connect checks. + * **HTTP** - probe backend service availability through http GET checks. + +* **Timeout(ms)*** - Probe timeout in milliseconds. +* **Request path*** - Http request path. + +**Backend** + +* **+Add** - add a backend host. +* **Address** - IP address of the backend host. +* **Port** - Service port on the backend host. +* **Enabled** - Administrative state of particular backend. + + +Example: Requesting an L4 Load Balancer service. + +.. image:: images/request_L4.png + :align: center + +Example: Listing of L4 Load Balancer services + +.. image:: images/listing_L4.png + :align: center + + +########################## +Access Control Lists (ACL) +########################## +Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access. + +Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches. + +Screenshot: TCAM utilization can be seen under Net→Inventory + +.. image:: images/TCAM.png + :align: center + +Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements. + +ACL Default Policy. +------------------- +The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules. + +Example: Changing “ACL Default Policy” for the site “siteDefault”. + +.. image:: images/siteDefault.png + :align: center + + +ACL rules +--------- +ACL rules can be created, listed, edited, approved under Services→ACL. + +Description of ACL fields. +General + +* **Name*** - Unique name for the ACL entry. +* **Protocol*** - IP protocol to match. + + * All - Any IP protocols. + * IP - Specific IP protocol number. + * TCP - TCP. + * UDP - UDP. + * ICMP ALL - Any IPv4 ICMP protocol. + * ICMP Custom - Custom IPv4 ICMP code. + * ICMPv6 ALL - Any IPv6 ICMP protocol. + * ICMPv6 Custom - Custom IPv6 ICMP code. + +* **Active Until** - Disable this rule at the defined date/time. +* **Action** - Permit or Deny forwarding of matched packets. +* **Established/Reverse** - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination. + +Source/Destination - Source and destination addresses and ports to match. + +* **Source*** IPv4/IPv6 - IPv4/IPv6 address. +* **Ports Type*** + + * Port Range - Match on the port or a port range defined in this window. + * Port Group - Match on a group of ports defined under Services→ ACL Port Group. + +* **From Port*** - Port range starting from. +* **To Port*** - Port range ending with. + +* **Comment** - Descriptive comment, commonly used for approval workflows. + +* **Check button** - Check if Another ACL on the system already permits the described network access. + +Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established). + +.. image:: images/action_permit.png + :align: center + + +Example: “Check” shows that requested access is already provided by a broader ACL rule. + +.. image:: images/ACL_rule.png + :align: center + + +ACL approval workflow +--------------------- +When one Tenant (one team) needs to get network access to resources under the responsibility of another Tenant (another team), an ACL can be created but will activate only after approval of the Tenant responsible for the destination address resources. See the below example. + +Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant. + +.. image:: images/ACL_approval.png + :align: center + +Screenshot: ACL stays in “waiting for approval” state until approved. + +.. image:: images/waiting_approval.png + :align: center + +Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it. + +.. image:: images/approve_reject.png + :align: center + +Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric. + +.. image:: images/ACL_active.png + :align: center + +The sequence order of ACL rules +------------------------------- +1. User-defined Deny Rules +2. User-defined Permit Rules +3. Deny the rest + diff --git a/en/2.9/_sources/switch-agent-installation.rst.txt b/en/2.9/_sources/switch-agent-installation.rst.txt new file mode 100644 index 0000000000..883ff606d0 --- /dev/null +++ b/en/2.9/_sources/switch-agent-installation.rst.txt @@ -0,0 +1,251 @@ +.. meta:: + :description: Netris Switch Agent Installation + +******************************** +Netris switch agent installation +******************************** + +For Cumulus Linux +================= +Requirements: +* Fresh install of Cumulus Linux v. 3.7.(x) - Cumulus 4.X is in the process of validation and will be supported in the next Netris release. + +Configure the OOB Management IP address +--------------------------------------- +Configure out of band management IP address, and in case Netris Controller is not in the same OOB network then configure a route to Netris Controller. No default route or other IP addresses should be configured. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + up ip ro add via #delete this line if Netris Controller is located in the same network with the switch. + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +Configure Cumulus Linux license +------------------------------- + +.. code-block:: shell-session + + sudo cl-license -i + +Copy/paste the Cumulus Linux license string then press ctrl-d. + +Install the Netris Agent +------------------------ +1. Add netris repository using Netris Controller as an http proxy. Replace with your actual Netris Controller address. + +.. note:: + + Netris Controller built-in proxy, by default, permits RFC1918 IP addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16). + If your management network is using IP addresses outside these ranges you will need to configure iptables on the Netris Controller accordingly. + +.. code-block:: shell-session + + export http_proxy=http://:3128 + + wget -qO - http://repo.netris.ai/repo/public.key | sudo apt-key add - + + echo "deb http://repo.netris.ai/repo/ jessie main" | sudo tee /etc/apt/sources.list.d/netris.list + +2. Update the apt + +.. code-block:: shell-session + + echo -e 'Acquire::http::Proxy "http://:3128";\nAcquire::https::Proxy "http://:3128";' | sudo tee -a /etc/apt/apt.conf.d/netris-proxy + + sudo apt update + +3. Install Netris Agent and dependencies + +.. code-block:: shell-session + + sudo apt install netris-sw + +4. Initialize the switch using netris-setup + +Description of netris-setup parameters + +.. code-block:: shell-session + + --auth - Authentication key, "6878C6DD88224981967F67EE2A73F092" is the default key. + --controller - IP address or domain name of Netris Controller. + --hostname - The hostname for the current switch, this hostname should match the name defined in the Controller. + --lo - IP address for the loopback interface, as it is defined in the controller. + --type - Role of the switch in your topology: spine/leaf + +.. code-block:: shell-session + + sudo /opt/netris/bin/netris-setup --auth= --controller= --hostname= --lo= --type= + +5. Reboot the switch + +.. code-block:: shell-session + + sudo reboot + +Once the switch boots up you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and switch color will reflect its health in Net→Topology + +Screenshot: Net→Inventory + +.. image:: images/inventory_heartbeat.png + :align: center + +For Ubuntu SwitchDev +==================== +.. note:: + + Further installation requires a Console and Internet connectivity via management port! + +1. NOS Uninstall + +Fist of all uninstall current NOS using **Uninstall OS** from grub menu: + +.. image:: images/uninstallOS.png + :align: center + +Once the uninstallation is completed, the switch will reboot automatically. + +2. Update ONIE + +Select **Update ONIE** from grub menu: + +.. image:: images/updateONIE.png + :align: center + +In case you don't have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually: + +.. code-block:: shell-session + + onie-discovery-stop + ip addr add dev eth0 + ip route add default via + echo "nameserver 1.1.1.1" > /etc/resolv.conf + +Update ONIE to the supported version. + +.. note:: + + ONIE image available for Mellanox switches only! + +.. code-block:: shell-session + + onie-self-update http://repo.netris.ai/repo/onie-updater-x86_64-mlnx_x86-r0 + +3. NOS Install + +Select **Install OS** from grub menu: + +.. image:: images/installOS.png + :align: center + +In case you don't have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually: + +.. code-block:: shell-session + + onie-discovery-stop + ip addr add dev eth0 + ip route add default via + echo "nameserver 1.1.1.1" > /etc/resolv.conf + +Install Ubuntu-SiwtchDev from the Netris custom image: + +.. code-block:: shell-session + + onie-nos-install http://repo.netris.ai/repo/netris-ubuntu-18.04.1.bin + +Default username/password + +``netris/newNet0ps`` + +Configure the OOB Management IP address +--------------------------------------- +Configure out of band management IP address, and in case Netris Controller is not in the same OOB network then configure a route to Netris Controller. No default route or other IP addresses should be configured. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + up ip ro add via #delete this line if Netris Controller is located in the same network with the switch. + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +Install the Netris Agent +------------------------ +1. Add netris repository using Netris Controller as an http proxy. Replace with your actual Netris Controller address. + +.. note:: + + Netris Controller built-in proxy, by default, permits RFC1918 IP addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16). If your management network is using IP addresses outside these ranges you will need to configure iptables on the Netris Controller accordingly. + +.. code-block:: shell-session + + export http_proxy=http://:3128 + + wget -qO - http://repo.netris.ai/repo/public.key | sudo apt-key add - + + echo "deb http://repo.netris.ai/repo/ bionic main" | sudo tee /etc/apt/sources.list.d/netris.list + +2. Update the apt + +.. code-block:: shell-session + + echo -e 'Acquire::http::Proxy "http://:3128";\nAcquire::https::Proxy "http://:3128";' | sudo tee -a /etc/apt/apt.conf.d/netris-proxy + + sudo apt update + +3. Install Netris Agent and dependencies + +.. code-block:: shell-session + + sudo apt-get update && sudo apt-get install netris-sw + +4. Initialize the switch using netris-setup + +Description of netris-setup parameters + +.. code-block:: shell-session + + --auth - Authentication key, "6878C6DD88224981967F67EE2A73F092" is the default key. + --controller - IP address or domain name of Netris Controller. + --hostname - The hostname for the current switch, this hostname should match the name defined in the Controller. + --lo - IP address for the loopback interface, as it is defined in the controller. + --type - Role of the switch in your topology: spine/leaf + +.. code-block:: shell-session + + sudo /opt/netris/bin/netris-setup --auth= --controller= --hostname= --lo= --type= + +5. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/2.9/_sources/visibility.rst.txt b/en/2.9/_sources/visibility.rst.txt new file mode 100644 index 0000000000..cba565de9d --- /dev/null +++ b/en/2.9/_sources/visibility.rst.txt @@ -0,0 +1,84 @@ +.. meta:: + :description: Netris System Visibility, Monitoring & Telemetry + +********************** +Visibility (Telescope) +********************** + +Graph Boards +================= +You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view. + +To start with Graph Boards, first, you need to add a new Graph Board. + +1. Navigate to Telescope→Graph Boards, open the dropdown menu in the top left corner, then click +Add board. + +.. image:: images/telescope.png + :align: center + +2. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants. + +.. image:: images/createboard.png + :align: center + +Now you can add graphs by clicking +Add graph. + +Description of +Add graph fields: + +- **Title** - Title for the new graph. +- **Type** - Type of data source. + + - Bps - Traffic bits per second. + - Pps - Traffic packets per second. + - Errors - Errors per second. + - Optical - Optical signal statistics/history. + - MAC Count - History of the number of MAC addresses on the port. +- **Function** - Currently, only summing is supported. +- **+Member** - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port. + +Example: Sum of traffic on two ISP(Iris1 + Iris2) links. + +.. image:: images/ISP_Iris.png + :align: center + +Example: Sum of the traffic on all ports under the service called “my V-NET” + +.. image:: images/V_NET.png + :align: center + +Screenshot: Listing of a Graph Board with the explanation of the controls. + +.. image:: images/graphboard.png + :align: center + +API Logs +======== +Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type. + +Dashboard +========= +Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems. + +Telescope→Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner. + +Description of the pie charts. + +* **Hardware Health** - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions. +* **E-BGP** - Statuses of external BGP sessions. +* **LB VIP** - Statuses of Load Balancer frontend / VIP availability. +* **LB Members** - Statuses of Load Balancer backend members. + +By clicking on each title you can see the details of the checks on the right side. + +Screenshot: Dashboard showing details of “Hardware Health.” + +.. image:: images/hardware_health.png + :align: center + +Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state. + +Screenshot: “Save as normal” on selected ports. + +.. image:: images/saveasnormal.png + :align: center + diff --git a/en/2.9/_static/_sphinx_javascript_frameworks_compat.js b/en/2.9/_static/_sphinx_javascript_frameworks_compat.js new file mode 100644 index 0000000000..8549469dc2 --- /dev/null +++ b/en/2.9/_static/_sphinx_javascript_frameworks_compat.js @@ -0,0 +1,134 @@ +/* + * _sphinx_javascript_frameworks_compat.js + * ~~~~~~~~~~ + * + * Compatability shim for jQuery and underscores.js. + * + * WILL BE REMOVED IN Sphinx 6.0 + * xref RemovedInSphinx60Warning + * + */ + +/** + * select a different prefix for underscore + */ +$u = _.noConflict(); + + +/** + * small helper function to urldecode strings + * + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL + */ +jQuery.urldecode = function(x) { + if (!x) { + return x + } + return decodeURIComponent(x.replace(/\+/g, ' ')); +}; + +/** + * small helper function to urlencode strings + */ +jQuery.urlencode = encodeURIComponent; + +/** + * This function returns the parsed url parameters of the + * current request. Multiple values per key are supported, + * it will always return arrays of strings for the value parts. + */ +jQuery.getQueryParameters = function(s) { + if (typeof s === 'undefined') + s = document.location.search; + var parts = s.substr(s.indexOf('?') + 1).split('&'); + var result = {}; + for (var i = 0; i < parts.length; i++) { + var tmp = parts[i].split('=', 2); + var key = jQuery.urldecode(tmp[0]); + var value = jQuery.urldecode(tmp[1]); + if (key in result) + result[key].push(value); + else + result[key] = [value]; + } + return result; +}; + +/** + * highlight a given string on a jquery object by wrapping it in + * span elements with the given class name. + */ +jQuery.fn.highlightText = function(text, className) { + function highlight(node, addItems) { + if (node.nodeType === 3) { + var val = node.nodeValue; + var pos = val.toLowerCase().indexOf(text); + if (pos >= 0 && + !jQuery(node.parentNode).hasClass(className) && + !jQuery(node.parentNode).hasClass("nohighlight")) { + var span; + var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.className = className; + } + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + node.parentNode.insertBefore(span, node.parentNode.insertBefore( + document.createTextNode(val.substr(pos + text.length)), + node.nextSibling)); + node.nodeValue = val.substr(0, pos); + if (isInSVG) { + var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect"); + var bbox = node.parentElement.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute('class', className); + addItems.push({ + "parent": node.parentNode, + "target": rect}); + } + } + } + else if (!jQuery(node).is("button, select, textarea")) { + jQuery.each(node.childNodes, function() { + highlight(this, addItems); + }); + } + } + var addItems = []; + var result = this.each(function() { + highlight(this, addItems); + }); + for (var i = 0; i < addItems.length; ++i) { + jQuery(addItems[i].parent).before(addItems[i].target); + } + return result; +}; + +/* + * backward compatibility for jQuery.browser + * This will be supported until firefox bug is fixed. + */ +if (!jQuery.browser) { + jQuery.uaMatch = function(ua) { + ua = ua.toLowerCase(); + + var match = /(chrome)[ \/]([\w.]+)/.exec(ua) || + /(webkit)[ \/]([\w.]+)/.exec(ua) || + /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) || + /(msie) ([\w.]+)/.exec(ua) || + ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) || + []; + + return { + browser: match[ 1 ] || "", + version: match[ 2 ] || "0" + }; + }; + jQuery.browser = {}; + jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true; +} diff --git a/en/2.9/_static/basic.css b/en/2.9/_static/basic.css new file mode 100644 index 0000000000..eeb0519a69 --- /dev/null +++ b/en/2.9/_static/basic.css @@ -0,0 +1,899 @@ +/* + * basic.css + * ~~~~~~~~~ + * + * Sphinx stylesheet -- basic theme. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +div.section::after { + display: block; + content: ''; + clear: left; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; + word-wrap: break-word; + overflow-wrap : break-word; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar #searchbox form.search { + overflow: hidden; +} + +div.sphinxsidebar #searchbox input[type="text"] { + float: left; + width: 80%; + padding: 0.25em; + box-sizing: border-box; +} + +div.sphinxsidebar #searchbox input[type="submit"] { + float: left; + width: 20%; + border-left: none; + padding: 0.25em; + box-sizing: border-box; +} + + +img { + border: 0; + max-width: 100%; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin: 10px 0 0 20px; + padding: 0; +} + +ul.search li { + padding: 5px 0 5px 20px; + background-image: url(file.png); + background-repeat: no-repeat; + background-position: 0 7px; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li p.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; + margin-left: auto; + margin-right: auto; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable ul { + margin-top: 0; + margin-bottom: 0; + list-style-type: none; +} + +table.indextable > tbody > tr > td > ul { + padding-left: 0em; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- domain module index --------------------------------------------------- */ + +table.modindextable td { + padding: 2px; + border-collapse: collapse; +} + +/* -- general body styles --------------------------------------------------- */ + +div.body { + min-width: 360px; + max-width: 800px; +} + +div.body p, div.body dd, div.body li, div.body blockquote { + -moz-hyphens: auto; + -ms-hyphens: auto; + -webkit-hyphens: auto; + hyphens: auto; +} + +a.headerlink { + visibility: hidden; +} +a.brackets:before, +span.brackets > a:before{ + content: "["; +} + +a.brackets:after, +span.brackets > a:after { + content: "]"; +} + + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { + visibility: visible; +} + +div.body p.caption { + text-align: inherit; +} + +div.body td { + text-align: left; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +img.align-left, figure.align-left, .figure.align-left, object.align-left { + clear: left; + float: left; + margin-right: 1em; +} + +img.align-right, figure.align-right, .figure.align-right, object.align-right { + clear: right; + float: right; + margin-left: 1em; +} + +img.align-center, figure.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +img.align-default, figure.align-default, .figure.align-default { + display: block; + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left; +} + +.align-center { + text-align: center; +} + +.align-default { + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar, +aside.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px; + background-color: #ffe; + width: 40%; + float: right; + clear: right; + overflow-x: auto; +} + +p.sidebar-title { + font-weight: bold; +} +div.admonition, div.topic, blockquote { + clear: left; +} + +/* -- topics ---------------------------------------------------------------- */ +div.topic { + border: 1px solid #ccc; + padding: 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- content of sidebars/topics/admonitions -------------------------------- */ + +div.sidebar > :last-child, +aside.sidebar > :last-child, +div.topic > :last-child, +div.admonition > :last-child { + margin-bottom: 0; +} + +div.sidebar::after, +aside.sidebar::after, +div.topic::after, +div.admonition::after, +blockquote::after { + display: block; + content: ''; + clear: both; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + margin-top: 10px; + margin-bottom: 10px; + border: 0; + border-collapse: collapse; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +table.align-default { + margin-left: auto; + margin-right: auto; +} + +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +th > :first-child, +td > :first-child { + margin-top: 0px; +} + +th > :last-child, +td > :last-child { + margin-bottom: 0px; +} + +/* -- figures --------------------------------------------------------------- */ + +div.figure, figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption, figcaption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number, +figcaption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text, +figcaption span.caption-text { +} + +/* -- field list styles ----------------------------------------------------- */ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.field-name { + -moz-hyphens: manual; + -ms-hyphens: manual; + -webkit-hyphens: manual; + hyphens: manual; +} + +/* -- hlist styles ---------------------------------------------------------- */ + +table.hlist { + margin: 1em 0; +} + +table.hlist td { + vertical-align: top; +} + +/* -- object description styles --------------------------------------------- */ + +.sig { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; +} + +.sig-name, code.descname { + background-color: transparent; + font-weight: bold; +} + +.sig-name { + font-size: 1.1em; +} + +code.descname { + font-size: 1.2em; +} + +.sig-prename, code.descclassname { + background-color: transparent; +} + +.optional { + font-size: 1.3em; +} + +.sig-paren { + font-size: larger; +} + +.sig-param.n { + font-style: italic; +} + +/* C++ specific styling */ + +.sig-inline.c-texpr, +.sig-inline.cpp-texpr { + font-family: unset; +} + +.sig.c .k, .sig.c .kt, +.sig.cpp .k, .sig.cpp .kt { + color: #0033B3; +} + +.sig.c .m, +.sig.cpp .m { + color: #1750EB; +} + +.sig.c .s, .sig.c .sc, +.sig.cpp .s, .sig.cpp .sc { + color: #067D17; +} + + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +:not(li) > ol > li:first-child > :first-child, +:not(li) > ul > li:first-child > :first-child { + margin-top: 0px; +} + +:not(li) > ol > li:last-child > :last-child, +:not(li) > ul > li:last-child > :last-child { + margin-bottom: 0px; +} + +ol.simple ol p, +ol.simple ul p, +ul.simple ol p, +ul.simple ul p { + margin-top: 0; +} + +ol.simple > li:not(:first-child) > p, +ul.simple > li:not(:first-child) > p { + margin-top: 0; +} + +ol.simple p, +ul.simple p { + margin-bottom: 0; +} +dl.footnote > dt, +dl.citation > dt { + float: left; + margin-right: 0.5em; +} + +dl.footnote > dd, +dl.citation > dd { + margin-bottom: 0em; +} + +dl.footnote > dd:after, +dl.citation > dd:after { + content: ""; + clear: both; +} + +dl.field-list { + display: grid; + grid-template-columns: fit-content(30%) auto; +} + +dl.field-list > dt { + font-weight: bold; + word-break: break-word; + padding-left: 0.5em; + padding-right: 5px; +} +dl.field-list > dt:after { + content: ":"; +} + + +dl.field-list > dd { + padding-left: 0.5em; + margin-top: 0em; + margin-left: 0em; + margin-bottom: 0em; +} + +dl { + margin-bottom: 15px; +} + +dd > :first-child { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +dl > dd:last-child, +dl > dd:last-child > :last-child { + margin-bottom: 0; +} + +dt:target, span.highlighted { + background-color: #fbe54e; +} + +rect.highlighted { + fill: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa; +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +.classifier:before { + font-style: normal; + margin: 0 0.5em; + content: ":"; + display: inline-block; +} + +abbr, acronym { + border-bottom: dotted 1px; + cursor: help; +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; + overflow-y: hidden; /* fixes display issues on Chrome browsers */ +} + +pre, div[class*="highlight-"] { + clear: both; +} + +span.pre { + -moz-hyphens: none; + -ms-hyphens: none; + -webkit-hyphens: none; + hyphens: none; + white-space: nowrap; +} + +div[class*="highlight-"] { + margin: 1em 0; +} + +td.linenos pre { + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + display: block; +} + +table.highlighttable tbody { + display: block; +} + +table.highlighttable tr { + display: flex; +} + +table.highlighttable td { + margin: 0; + padding: 0; +} + +table.highlighttable td.linenos { + padding-right: 0.5em; +} + +table.highlighttable td.code { + flex: 1; + overflow: hidden; +} + +.highlight .hll { + display: block; +} + +div.highlight pre, +table.highlighttable pre { + margin: 0; +} + +div.code-block-caption + div { + margin-top: 0; +} + +div.code-block-caption { + margin-top: 1em; + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +table.highlighttable td.linenos, +span.linenos, +div.highlight span.gp { /* gp: Generic.Prompt */ + user-select: none; + -webkit-user-select: text; /* Safari fallback only */ + -webkit-user-select: none; /* Chrome/Safari */ + -moz-user-select: none; /* Firefox */ + -ms-user-select: none; /* IE10+ */ +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + margin: 1em 0; +} + +code.xref, a code { + background-color: transparent; + font-weight: bold; +} + +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.body div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +span.eqno a.headerlink { + position: absolute; + z-index: 1; +} + +div.math:hover a.headerlink { + visibility: visible; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} \ No newline at end of file diff --git a/en/2.9/_static/check-solid.svg b/en/2.9/_static/check-solid.svg new file mode 100644 index 0000000000..92fad4b5c0 --- /dev/null +++ b/en/2.9/_static/check-solid.svg @@ -0,0 +1,4 @@ + + + + diff --git a/en/2.9/_static/clipboard.min.js b/en/2.9/_static/clipboard.min.js new file mode 100644 index 0000000000..54b3c46381 --- /dev/null +++ b/en/2.9/_static/clipboard.min.js @@ -0,0 +1,7 @@ +/*! + * clipboard.js v2.0.8 + * https://clipboardjs.com/ + * + * Licensed MIT © Zeno Rocha + */ +!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1 + + + + diff --git a/en/2.9/_static/copybutton.css b/en/2.9/_static/copybutton.css new file mode 100644 index 0000000000..f1916ec7d1 --- /dev/null +++ b/en/2.9/_static/copybutton.css @@ -0,0 +1,94 @@ +/* Copy buttons */ +button.copybtn { + position: absolute; + display: flex; + top: .3em; + right: .3em; + width: 1.7em; + height: 1.7em; + opacity: 0; + transition: opacity 0.3s, border .3s, background-color .3s; + user-select: none; + padding: 0; + border: none; + outline: none; + border-radius: 0.4em; + /* The colors that GitHub uses */ + border: #1b1f2426 1px solid; + background-color: #f6f8fa; + color: #57606a; +} + +button.copybtn.success { + border-color: #22863a; + color: #22863a; +} + +button.copybtn svg { + stroke: currentColor; + width: 1.5em; + height: 1.5em; + padding: 0.1em; +} + +div.highlight { + position: relative; +} + +/* Show the copybutton */ +.highlight:hover button.copybtn, button.copybtn.success { + opacity: 1; +} + +.highlight button.copybtn:hover { + background-color: rgb(235, 235, 235); +} + +.highlight button.copybtn:active { + background-color: rgb(187, 187, 187); +} + +/** + * A minimal CSS-only tooltip copied from: + * https://codepen.io/mildrenben/pen/rVBrpK + * + * To use, write HTML like the following: + * + *

Short

+ */ + .o-tooltip--left { + position: relative; + } + + .o-tooltip--left:after { + opacity: 0; + visibility: hidden; + position: absolute; + content: attr(data-tooltip); + padding: .2em; + font-size: .8em; + left: -.2em; + background: grey; + color: white; + white-space: nowrap; + z-index: 2; + border-radius: 2px; + transform: translateX(-102%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); +} + +.o-tooltip--left:hover:after { + display: block; + opacity: 1; + visibility: visible; + transform: translateX(-100%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); + transition-delay: .5s; +} + +/* By default the copy button shouldn't show up when printing a page */ +@media print { + button.copybtn { + display: none; + } +} diff --git a/en/2.9/_static/copybutton.js b/en/2.9/_static/copybutton.js new file mode 100644 index 0000000000..2ea7ff3e21 --- /dev/null +++ b/en/2.9/_static/copybutton.js @@ -0,0 +1,248 @@ +// Localization support +const messages = { + 'en': { + 'copy': 'Copy', + 'copy_to_clipboard': 'Copy to clipboard', + 'copy_success': 'Copied!', + 'copy_failure': 'Failed to copy', + }, + 'es' : { + 'copy': 'Copiar', + 'copy_to_clipboard': 'Copiar al portapapeles', + 'copy_success': '¡Copiado!', + 'copy_failure': 'Error al copiar', + }, + 'de' : { + 'copy': 'Kopieren', + 'copy_to_clipboard': 'In die Zwischenablage kopieren', + 'copy_success': 'Kopiert!', + 'copy_failure': 'Fehler beim Kopieren', + }, + 'fr' : { + 'copy': 'Copier', + 'copy_to_clipboard': 'Copier dans le presse-papier', + 'copy_success': 'Copié !', + 'copy_failure': 'Échec de la copie', + }, + 'ru': { + 'copy': 'Скопировать', + 'copy_to_clipboard': 'Скопировать в буфер', + 'copy_success': 'Скопировано!', + 'copy_failure': 'Не удалось скопировать', + }, + 'zh-CN': { + 'copy': '复制', + 'copy_to_clipboard': '复制到剪贴板', + 'copy_success': '复制成功!', + 'copy_failure': '复制失败', + }, + 'it' : { + 'copy': 'Copiare', + 'copy_to_clipboard': 'Copiato negli appunti', + 'copy_success': 'Copiato!', + 'copy_failure': 'Errore durante la copia', + } +} + +let locale = 'en' +if( document.documentElement.lang !== undefined + && messages[document.documentElement.lang] !== undefined ) { + locale = document.documentElement.lang +} + +let doc_url_root = DOCUMENTATION_OPTIONS.URL_ROOT; +if (doc_url_root == '#') { + doc_url_root = ''; +} + +/** + * SVG files for our copy buttons + */ +let iconCheck = ` + ${messages[locale]['copy_success']} + + +` + +// If the user specified their own SVG use that, otherwise use the default +let iconCopy = ``; +if (!iconCopy) { + iconCopy = ` + ${messages[locale]['copy_to_clipboard']} + + + +` +} + +/** + * Set up copy/paste for code blocks + */ + +const runWhenDOMLoaded = cb => { + if (document.readyState != 'loading') { + cb() + } else if (document.addEventListener) { + document.addEventListener('DOMContentLoaded', cb) + } else { + document.attachEvent('onreadystatechange', function() { + if (document.readyState == 'complete') cb() + }) + } +} + +const codeCellId = index => `codecell${index}` + +// Clears selected text since ClipboardJS will select the text when copying +const clearSelection = () => { + if (window.getSelection) { + window.getSelection().removeAllRanges() + } else if (document.selection) { + document.selection.empty() + } +} + +// Changes tooltip text for a moment, then changes it back +// We want the timeout of our `success` class to be a bit shorter than the +// tooltip and icon change, so that we can hide the icon before changing back. +var timeoutIcon = 2000; +var timeoutSuccessClass = 1500; + +const temporarilyChangeTooltip = (el, oldText, newText) => { + el.setAttribute('data-tooltip', newText) + el.classList.add('success') + // Remove success a little bit sooner than we change the tooltip + // So that we can use CSS to hide the copybutton first + setTimeout(() => el.classList.remove('success'), timeoutSuccessClass) + setTimeout(() => el.setAttribute('data-tooltip', oldText), timeoutIcon) +} + +// Changes the copy button icon for two seconds, then changes it back +const temporarilyChangeIcon = (el) => { + el.innerHTML = iconCheck; + setTimeout(() => {el.innerHTML = iconCopy}, timeoutIcon) +} + +const addCopyButtonToCodeCells = () => { + // If ClipboardJS hasn't loaded, wait a bit and try again. This + // happens because we load ClipboardJS asynchronously. + if (window.ClipboardJS === undefined) { + setTimeout(addCopyButtonToCodeCells, 250) + return + } + + // Add copybuttons to all of our code cells + const COPYBUTTON_SELECTOR = 'div.highlight pre'; + const codeCells = document.querySelectorAll(COPYBUTTON_SELECTOR) + codeCells.forEach((codeCell, index) => { + const id = codeCellId(index) + codeCell.setAttribute('id', id) + + const clipboardButton = id => + `` + codeCell.insertAdjacentHTML('afterend', clipboardButton(id)) + }) + +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} + + +var copyTargetText = (trigger) => { + var target = document.querySelector(trigger.attributes['data-clipboard-target'].value); + + // get filtered text + let exclude = '.linenos'; + + let text = filterText(target, exclude); + return formatCopyText(text, '', false, true, true, true, '', '') +} + + // Initialize with a callback so we can modify the text before copy + const clipboard = new ClipboardJS('.copybtn', {text: copyTargetText}) + + // Update UI with error/success messages + clipboard.on('success', event => { + clearSelection() + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_success']) + temporarilyChangeIcon(event.trigger) + }) + + clipboard.on('error', event => { + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_failure']) + }) +} + +runWhenDOMLoaded(addCopyButtonToCodeCells) \ No newline at end of file diff --git a/en/2.9/_static/copybutton_funcs.js b/en/2.9/_static/copybutton_funcs.js new file mode 100644 index 0000000000..dbe1aaad79 --- /dev/null +++ b/en/2.9/_static/copybutton_funcs.js @@ -0,0 +1,73 @@ +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +export function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +export function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} diff --git a/en/2.9/_static/css/badge_only.css b/en/2.9/_static/css/badge_only.css new file mode 100644 index 0000000000..e380325bc6 --- /dev/null +++ b/en/2.9/_static/css/badge_only.css @@ -0,0 +1 @@ +.fa:before{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-style:normal;font-weight:400;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#FontAwesome) format("svg")}.fa:before{font-family:FontAwesome;font-style:normal;font-weight:400;line-height:1}.fa:before,a .fa{text-decoration:inherit}.fa:before,a .fa,li .fa{display:inline-block}li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-.8em}ul.fas li .fa{width:.8em}ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before,.icon-book:before{content:"\f02d"}.fa-caret-down:before,.icon-caret-down:before{content:"\f0d7"}.fa-caret-up:before,.icon-caret-up:before{content:"\f0d8"}.fa-caret-left:before,.icon-caret-left:before{content:"\f0d9"}.fa-caret-right:before,.icon-caret-right:before{content:"\f0da"}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60}.rst-versions .rst-current-version:after{clear:both;content:"";display:block}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}} \ No newline at end of file diff --git a/en/2.9/_static/css/fonts/Roboto-Slab-Bold.woff b/en/2.9/_static/css/fonts/Roboto-Slab-Bold.woff new file mode 100644 index 0000000000..6cb6000018 Binary files /dev/null and b/en/2.9/_static/css/fonts/Roboto-Slab-Bold.woff differ diff --git a/en/2.9/_static/css/fonts/Roboto-Slab-Bold.woff2 b/en/2.9/_static/css/fonts/Roboto-Slab-Bold.woff2 new file mode 100644 index 0000000000..7059e23142 Binary files /dev/null and b/en/2.9/_static/css/fonts/Roboto-Slab-Bold.woff2 differ diff --git a/en/2.9/_static/css/fonts/Roboto-Slab-Regular.woff b/en/2.9/_static/css/fonts/Roboto-Slab-Regular.woff new file mode 100644 index 0000000000..f815f63f99 Binary files /dev/null and b/en/2.9/_static/css/fonts/Roboto-Slab-Regular.woff differ diff --git a/en/2.9/_static/css/fonts/Roboto-Slab-Regular.woff2 b/en/2.9/_static/css/fonts/Roboto-Slab-Regular.woff2 new file mode 100644 index 0000000000..f2c76e5bda Binary files /dev/null and b/en/2.9/_static/css/fonts/Roboto-Slab-Regular.woff2 differ diff --git a/en/2.9/_static/css/fonts/fontawesome-webfont.eot b/en/2.9/_static/css/fonts/fontawesome-webfont.eot new file mode 100644 index 0000000000..e9f60ca953 Binary files /dev/null and b/en/2.9/_static/css/fonts/fontawesome-webfont.eot differ diff --git a/en/2.9/_static/css/fonts/fontawesome-webfont.svg b/en/2.9/_static/css/fonts/fontawesome-webfont.svg new file mode 100644 index 0000000000..855c845e53 --- /dev/null +++ b/en/2.9/_static/css/fonts/fontawesome-webfont.svg @@ -0,0 +1,2671 @@ + + + + +Created by FontForge 20120731 at Mon Oct 24 17:37:40 2016 + By ,,, +Copyright Dave Gandy 2016. All rights reserved. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/en/2.9/_static/css/fonts/fontawesome-webfont.ttf b/en/2.9/_static/css/fonts/fontawesome-webfont.ttf new file mode 100644 index 0000000000..35acda2fa1 Binary files /dev/null and b/en/2.9/_static/css/fonts/fontawesome-webfont.ttf differ diff --git a/en/2.9/_static/css/fonts/fontawesome-webfont.woff b/en/2.9/_static/css/fonts/fontawesome-webfont.woff new file mode 100644 index 0000000000..400014a4b0 Binary files /dev/null and b/en/2.9/_static/css/fonts/fontawesome-webfont.woff differ diff --git a/en/2.9/_static/css/fonts/fontawesome-webfont.woff2 b/en/2.9/_static/css/fonts/fontawesome-webfont.woff2 new file mode 100644 index 0000000000..4d13fc6040 Binary files /dev/null and b/en/2.9/_static/css/fonts/fontawesome-webfont.woff2 differ diff --git a/en/2.9/_static/css/fonts/lato-bold-italic.woff b/en/2.9/_static/css/fonts/lato-bold-italic.woff new file mode 100644 index 0000000000..88ad05b9ff Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-bold-italic.woff differ diff --git a/en/2.9/_static/css/fonts/lato-bold-italic.woff2 b/en/2.9/_static/css/fonts/lato-bold-italic.woff2 new file mode 100644 index 0000000000..c4e3d804b5 Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-bold-italic.woff2 differ diff --git a/en/2.9/_static/css/fonts/lato-bold.woff b/en/2.9/_static/css/fonts/lato-bold.woff new file mode 100644 index 0000000000..c6dff51f06 Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-bold.woff differ diff --git a/en/2.9/_static/css/fonts/lato-bold.woff2 b/en/2.9/_static/css/fonts/lato-bold.woff2 new file mode 100644 index 0000000000..bb195043cf Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-bold.woff2 differ diff --git a/en/2.9/_static/css/fonts/lato-normal-italic.woff b/en/2.9/_static/css/fonts/lato-normal-italic.woff new file mode 100644 index 0000000000..76114bc033 Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-normal-italic.woff differ diff --git a/en/2.9/_static/css/fonts/lato-normal-italic.woff2 b/en/2.9/_static/css/fonts/lato-normal-italic.woff2 new file mode 100644 index 0000000000..3404f37e2e Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-normal-italic.woff2 differ diff --git a/en/2.9/_static/css/fonts/lato-normal.woff b/en/2.9/_static/css/fonts/lato-normal.woff new file mode 100644 index 0000000000..ae1307ff5f Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-normal.woff differ diff --git a/en/2.9/_static/css/fonts/lato-normal.woff2 b/en/2.9/_static/css/fonts/lato-normal.woff2 new file mode 100644 index 0000000000..3bf9843328 Binary files /dev/null and b/en/2.9/_static/css/fonts/lato-normal.woff2 differ diff --git a/en/2.9/_static/css/theme.css b/en/2.9/_static/css/theme.css new file mode 100644 index 0000000000..8cd4f101a9 --- /dev/null +++ b/en/2.9/_static/css/theme.css @@ -0,0 +1,4 @@ +html{box-sizing:border-box}*,:after,:before{box-sizing:inherit}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}[hidden],audio:not([controls]){display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:active,a:hover{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:700}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;text-decoration:none}ins,mark{color:#000}mark{background:#ff0;font-style:italic;font-weight:700}.rst-content code,.rst-content tt,code,kbd,pre,samp{font-family:monospace,serif;_font-family:courier new,monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:after,q:before{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sup{top:-.5em}sub{bottom:-.25em}dl,ol,ul{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure,form{margin:0}label{cursor:pointer}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type=button],input[type=reset],input[type=submit]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type=search]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}textarea{resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:.2em 0;background:#ccc;color:#000;padding:.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none!important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{body,html,section{background:none!important}*{box-shadow:none!important;text-shadow:none!important;filter:none!important;-ms-filter:none!important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="#"]:after,a[href^="javascript:"]:after{content:""}blockquote,pre{page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}img{max-width:100%!important}@page{margin:.5cm}.rst-content .toctree-wrapper>p.caption,h2,h3,p{orphans:3;widows:3}.rst-content .toctree-wrapper>p.caption,h2,h3{page-break-after:avoid}}.btn,.fa:before,.icon:before,.rst-content .admonition,.rst-content .admonition-title:before,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .code-block-caption .headerlink:before,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-alert,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before,.wy-nav-top a,.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a,input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week],select,textarea{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}/*! + * Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */@font-face{font-family:FontAwesome;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713);src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix&v=4.7.0) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#fontawesomeregular) format("svg");font-weight:400;font-style:normal}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14286em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14286em;width:2.14286em;top:.14286em;text-align:center}.fa-li.fa-lg{left:-1.85714em}.fa-border{padding:.2em .25em .15em;border:.08em solid #eee;border-radius:.1em}.fa-pull-left{float:left}.fa-pull-right{float:right}.fa-pull-left.icon,.fa.fa-pull-left,.rst-content .code-block-caption .fa-pull-left.headerlink,.rst-content .fa-pull-left.admonition-title,.rst-content code.download span.fa-pull-left:first-child,.rst-content dl dt .fa-pull-left.headerlink,.rst-content h1 .fa-pull-left.headerlink,.rst-content h2 .fa-pull-left.headerlink,.rst-content h3 .fa-pull-left.headerlink,.rst-content h4 .fa-pull-left.headerlink,.rst-content h5 .fa-pull-left.headerlink,.rst-content h6 .fa-pull-left.headerlink,.rst-content p.caption .fa-pull-left.headerlink,.rst-content table>caption .fa-pull-left.headerlink,.rst-content tt.download span.fa-pull-left:first-child,.wy-menu-vertical li.current>a span.fa-pull-left.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-left.toctree-expand,.wy-menu-vertical li span.fa-pull-left.toctree-expand{margin-right:.3em}.fa-pull-right.icon,.fa.fa-pull-right,.rst-content .code-block-caption .fa-pull-right.headerlink,.rst-content .fa-pull-right.admonition-title,.rst-content code.download span.fa-pull-right:first-child,.rst-content dl dt .fa-pull-right.headerlink,.rst-content h1 .fa-pull-right.headerlink,.rst-content h2 .fa-pull-right.headerlink,.rst-content h3 .fa-pull-right.headerlink,.rst-content h4 .fa-pull-right.headerlink,.rst-content h5 .fa-pull-right.headerlink,.rst-content h6 .fa-pull-right.headerlink,.rst-content p.caption .fa-pull-right.headerlink,.rst-content table>caption .fa-pull-right.headerlink,.rst-content tt.download span.fa-pull-right:first-child,.wy-menu-vertical li.current>a span.fa-pull-right.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-right.toctree-expand,.wy-menu-vertical li span.fa-pull-right.toctree-expand{margin-left:.3em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left,.pull-left.icon,.rst-content .code-block-caption .pull-left.headerlink,.rst-content .pull-left.admonition-title,.rst-content code.download span.pull-left:first-child,.rst-content dl dt .pull-left.headerlink,.rst-content h1 .pull-left.headerlink,.rst-content h2 .pull-left.headerlink,.rst-content h3 .pull-left.headerlink,.rst-content h4 .pull-left.headerlink,.rst-content h5 .pull-left.headerlink,.rst-content h6 .pull-left.headerlink,.rst-content p.caption .pull-left.headerlink,.rst-content table>caption .pull-left.headerlink,.rst-content tt.download span.pull-left:first-child,.wy-menu-vertical li.current>a span.pull-left.toctree-expand,.wy-menu-vertical li.on a span.pull-left.toctree-expand,.wy-menu-vertical li span.pull-left.toctree-expand{margin-right:.3em}.fa.pull-right,.pull-right.icon,.rst-content .code-block-caption .pull-right.headerlink,.rst-content .pull-right.admonition-title,.rst-content code.download span.pull-right:first-child,.rst-content dl dt .pull-right.headerlink,.rst-content h1 .pull-right.headerlink,.rst-content h2 .pull-right.headerlink,.rst-content h3 .pull-right.headerlink,.rst-content h4 .pull-right.headerlink,.rst-content h5 .pull-right.headerlink,.rst-content h6 .pull-right.headerlink,.rst-content p.caption .pull-right.headerlink,.rst-content table>caption .pull-right.headerlink,.rst-content tt.download span.pull-right:first-child,.wy-menu-vertical li.current>a span.pull-right.toctree-expand,.wy-menu-vertical li.on a span.pull-right.toctree-expand,.wy-menu-vertical li span.pull-right.toctree-expand{margin-left:.3em}.fa-spin{-webkit-animation:fa-spin 2s linear infinite;animation:fa-spin 2s linear infinite}.fa-pulse{-webkit-animation:fa-spin 1s steps(8) infinite;animation:fa-spin 1s steps(8) infinite}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";-webkit-transform:scaleX(-1);-ms-transform:scaleX(-1);transform:scaleX(-1)}.fa-flip-vertical{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";-webkit-transform:scaleY(-1);-ms-transform:scaleY(-1);transform:scaleY(-1)}:root .fa-flip-horizontal,:root .fa-flip-vertical,:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:""}.fa-music:before{content:""}.fa-search:before,.icon-search:before{content:""}.fa-envelope-o:before{content:""}.fa-heart:before{content:""}.fa-star:before{content:""}.fa-star-o:before{content:""}.fa-user:before{content:""}.fa-film:before{content:""}.fa-th-large:before{content:""}.fa-th:before{content:""}.fa-th-list:before{content:""}.fa-check:before{content:""}.fa-close:before,.fa-remove:before,.fa-times:before{content:""}.fa-search-plus:before{content:""}.fa-search-minus:before{content:""}.fa-power-off:before{content:""}.fa-signal:before{content:""}.fa-cog:before,.fa-gear:before{content:""}.fa-trash-o:before{content:""}.fa-home:before,.icon-home:before{content:""}.fa-file-o:before{content:""}.fa-clock-o:before{content:""}.fa-road:before{content:""}.fa-download:before,.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{content:""}.fa-arrow-circle-o-down:before{content:""}.fa-arrow-circle-o-up:before{content:""}.fa-inbox:before{content:""}.fa-play-circle-o:before{content:""}.fa-repeat:before,.fa-rotate-right:before{content:""}.fa-refresh:before{content:""}.fa-list-alt:before{content:""}.fa-lock:before{content:""}.fa-flag:before{content:""}.fa-headphones:before{content:""}.fa-volume-off:before{content:""}.fa-volume-down:before{content:""}.fa-volume-up:before{content:""}.fa-qrcode:before{content:""}.fa-barcode:before{content:""}.fa-tag:before{content:""}.fa-tags:before{content:""}.fa-book:before,.icon-book:before{content:""}.fa-bookmark:before{content:""}.fa-print:before{content:""}.fa-camera:before{content:""}.fa-font:before{content:""}.fa-bold:before{content:""}.fa-italic:before{content:""}.fa-text-height:before{content:""}.fa-text-width:before{content:""}.fa-align-left:before{content:""}.fa-align-center:before{content:""}.fa-align-right:before{content:""}.fa-align-justify:before{content:""}.fa-list:before{content:""}.fa-dedent:before,.fa-outdent:before{content:""}.fa-indent:before{content:""}.fa-video-camera:before{content:""}.fa-image:before,.fa-photo:before,.fa-picture-o:before{content:""}.fa-pencil:before{content:""}.fa-map-marker:before{content:""}.fa-adjust:before{content:""}.fa-tint:before{content:""}.fa-edit:before,.fa-pencil-square-o:before{content:""}.fa-share-square-o:before{content:""}.fa-check-square-o:before{content:""}.fa-arrows:before{content:""}.fa-step-backward:before{content:""}.fa-fast-backward:before{content:""}.fa-backward:before{content:""}.fa-play:before{content:""}.fa-pause:before{content:""}.fa-stop:before{content:""}.fa-forward:before{content:""}.fa-fast-forward:before{content:""}.fa-step-forward:before{content:""}.fa-eject:before{content:""}.fa-chevron-left:before{content:""}.fa-chevron-right:before{content:""}.fa-plus-circle:before{content:""}.fa-minus-circle:before{content:""}.fa-times-circle:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before{content:""}.fa-check-circle:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before{content:""}.fa-question-circle:before{content:""}.fa-info-circle:before{content:""}.fa-crosshairs:before{content:""}.fa-times-circle-o:before{content:""}.fa-check-circle-o:before{content:""}.fa-ban:before{content:""}.fa-arrow-left:before{content:""}.fa-arrow-right:before{content:""}.fa-arrow-up:before{content:""}.fa-arrow-down:before{content:""}.fa-mail-forward:before,.fa-share:before{content:""}.fa-expand:before{content:""}.fa-compress:before{content:""}.fa-plus:before{content:""}.fa-minus:before{content:""}.fa-asterisk:before{content:""}.fa-exclamation-circle:before,.rst-content .admonition-title:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before{content:""}.fa-gift:before{content:""}.fa-leaf:before{content:""}.fa-fire:before,.icon-fire:before{content:""}.fa-eye:before{content:""}.fa-eye-slash:before{content:""}.fa-exclamation-triangle:before,.fa-warning:before{content:""}.fa-plane:before{content:""}.fa-calendar:before{content:""}.fa-random:before{content:""}.fa-comment:before{content:""}.fa-magnet:before{content:""}.fa-chevron-up:before{content:""}.fa-chevron-down:before{content:""}.fa-retweet:before{content:""}.fa-shopping-cart:before{content:""}.fa-folder:before{content:""}.fa-folder-open:before{content:""}.fa-arrows-v:before{content:""}.fa-arrows-h:before{content:""}.fa-bar-chart-o:before,.fa-bar-chart:before{content:""}.fa-twitter-square:before{content:""}.fa-facebook-square:before{content:""}.fa-camera-retro:before{content:""}.fa-key:before{content:""}.fa-cogs:before,.fa-gears:before{content:""}.fa-comments:before{content:""}.fa-thumbs-o-up:before{content:""}.fa-thumbs-o-down:before{content:""}.fa-star-half:before{content:""}.fa-heart-o:before{content:""}.fa-sign-out:before{content:""}.fa-linkedin-square:before{content:""}.fa-thumb-tack:before{content:""}.fa-external-link:before{content:""}.fa-sign-in:before{content:""}.fa-trophy:before{content:""}.fa-github-square:before{content:""}.fa-upload:before{content:""}.fa-lemon-o:before{content:""}.fa-phone:before{content:""}.fa-square-o:before{content:""}.fa-bookmark-o:before{content:""}.fa-phone-square:before{content:""}.fa-twitter:before{content:""}.fa-facebook-f:before,.fa-facebook:before{content:""}.fa-github:before,.icon-github:before{content:""}.fa-unlock:before{content:""}.fa-credit-card:before{content:""}.fa-feed:before,.fa-rss:before{content:""}.fa-hdd-o:before{content:""}.fa-bullhorn:before{content:""}.fa-bell:before{content:""}.fa-certificate:before{content:""}.fa-hand-o-right:before{content:""}.fa-hand-o-left:before{content:""}.fa-hand-o-up:before{content:""}.fa-hand-o-down:before{content:""}.fa-arrow-circle-left:before,.icon-circle-arrow-left:before{content:""}.fa-arrow-circle-right:before,.icon-circle-arrow-right:before{content:""}.fa-arrow-circle-up:before{content:""}.fa-arrow-circle-down:before{content:""}.fa-globe:before{content:""}.fa-wrench:before{content:""}.fa-tasks:before{content:""}.fa-filter:before{content:""}.fa-briefcase:before{content:""}.fa-arrows-alt:before{content:""}.fa-group:before,.fa-users:before{content:""}.fa-chain:before,.fa-link:before,.icon-link:before{content:""}.fa-cloud:before{content:""}.fa-flask:before{content:""}.fa-cut:before,.fa-scissors:before{content:""}.fa-copy:before,.fa-files-o:before{content:""}.fa-paperclip:before{content:""}.fa-floppy-o:before,.fa-save:before{content:""}.fa-square:before{content:""}.fa-bars:before,.fa-navicon:before,.fa-reorder:before{content:""}.fa-list-ul:before{content:""}.fa-list-ol:before{content:""}.fa-strikethrough:before{content:""}.fa-underline:before{content:""}.fa-table:before{content:""}.fa-magic:before{content:""}.fa-truck:before{content:""}.fa-pinterest:before{content:""}.fa-pinterest-square:before{content:""}.fa-google-plus-square:before{content:""}.fa-google-plus:before{content:""}.fa-money:before{content:""}.fa-caret-down:before,.icon-caret-down:before,.wy-dropdown .caret:before{content:""}.fa-caret-up:before{content:""}.fa-caret-left:before{content:""}.fa-caret-right:before{content:""}.fa-columns:before{content:""}.fa-sort:before,.fa-unsorted:before{content:""}.fa-sort-desc:before,.fa-sort-down:before{content:""}.fa-sort-asc:before,.fa-sort-up:before{content:""}.fa-envelope:before{content:""}.fa-linkedin:before{content:""}.fa-rotate-left:before,.fa-undo:before{content:""}.fa-gavel:before,.fa-legal:before{content:""}.fa-dashboard:before,.fa-tachometer:before{content:""}.fa-comment-o:before{content:""}.fa-comments-o:before{content:""}.fa-bolt:before,.fa-flash:before{content:""}.fa-sitemap:before{content:""}.fa-umbrella:before{content:""}.fa-clipboard:before,.fa-paste:before{content:""}.fa-lightbulb-o:before{content:""}.fa-exchange:before{content:""}.fa-cloud-download:before{content:""}.fa-cloud-upload:before{content:""}.fa-user-md:before{content:""}.fa-stethoscope:before{content:""}.fa-suitcase:before{content:""}.fa-bell-o:before{content:""}.fa-coffee:before{content:""}.fa-cutlery:before{content:""}.fa-file-text-o:before{content:""}.fa-building-o:before{content:""}.fa-hospital-o:before{content:""}.fa-ambulance:before{content:""}.fa-medkit:before{content:""}.fa-fighter-jet:before{content:""}.fa-beer:before{content:""}.fa-h-square:before{content:""}.fa-plus-square:before{content:""}.fa-angle-double-left:before{content:""}.fa-angle-double-right:before{content:""}.fa-angle-double-up:before{content:""}.fa-angle-double-down:before{content:""}.fa-angle-left:before{content:""}.fa-angle-right:before{content:""}.fa-angle-up:before{content:""}.fa-angle-down:before{content:""}.fa-desktop:before{content:""}.fa-laptop:before{content:""}.fa-tablet:before{content:""}.fa-mobile-phone:before,.fa-mobile:before{content:""}.fa-circle-o:before{content:""}.fa-quote-left:before{content:""}.fa-quote-right:before{content:""}.fa-spinner:before{content:""}.fa-circle:before{content:""}.fa-mail-reply:before,.fa-reply:before{content:""}.fa-github-alt:before{content:""}.fa-folder-o:before{content:""}.fa-folder-open-o:before{content:""}.fa-smile-o:before{content:""}.fa-frown-o:before{content:""}.fa-meh-o:before{content:""}.fa-gamepad:before{content:""}.fa-keyboard-o:before{content:""}.fa-flag-o:before{content:""}.fa-flag-checkered:before{content:""}.fa-terminal:before{content:""}.fa-code:before{content:""}.fa-mail-reply-all:before,.fa-reply-all:before{content:""}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:""}.fa-location-arrow:before{content:""}.fa-crop:before{content:""}.fa-code-fork:before{content:""}.fa-chain-broken:before,.fa-unlink:before{content:""}.fa-question:before{content:""}.fa-info:before{content:""}.fa-exclamation:before{content:""}.fa-superscript:before{content:""}.fa-subscript:before{content:""}.fa-eraser:before{content:""}.fa-puzzle-piece:before{content:""}.fa-microphone:before{content:""}.fa-microphone-slash:before{content:""}.fa-shield:before{content:""}.fa-calendar-o:before{content:""}.fa-fire-extinguisher:before{content:""}.fa-rocket:before{content:""}.fa-maxcdn:before{content:""}.fa-chevron-circle-left:before{content:""}.fa-chevron-circle-right:before{content:""}.fa-chevron-circle-up:before{content:""}.fa-chevron-circle-down:before{content:""}.fa-html5:before{content:""}.fa-css3:before{content:""}.fa-anchor:before{content:""}.fa-unlock-alt:before{content:""}.fa-bullseye:before{content:""}.fa-ellipsis-h:before{content:""}.fa-ellipsis-v:before{content:""}.fa-rss-square:before{content:""}.fa-play-circle:before{content:""}.fa-ticket:before{content:""}.fa-minus-square:before{content:""}.fa-minus-square-o:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before{content:""}.fa-level-up:before{content:""}.fa-level-down:before{content:""}.fa-check-square:before{content:""}.fa-pencil-square:before{content:""}.fa-external-link-square:before{content:""}.fa-share-square:before{content:""}.fa-compass:before{content:""}.fa-caret-square-o-down:before,.fa-toggle-down:before{content:""}.fa-caret-square-o-up:before,.fa-toggle-up:before{content:""}.fa-caret-square-o-right:before,.fa-toggle-right:before{content:""}.fa-eur:before,.fa-euro:before{content:""}.fa-gbp:before{content:""}.fa-dollar:before,.fa-usd:before{content:""}.fa-inr:before,.fa-rupee:before{content:""}.fa-cny:before,.fa-jpy:before,.fa-rmb:before,.fa-yen:before{content:""}.fa-rouble:before,.fa-rub:before,.fa-ruble:before{content:""}.fa-krw:before,.fa-won:before{content:""}.fa-bitcoin:before,.fa-btc:before{content:""}.fa-file:before{content:""}.fa-file-text:before{content:""}.fa-sort-alpha-asc:before{content:""}.fa-sort-alpha-desc:before{content:""}.fa-sort-amount-asc:before{content:""}.fa-sort-amount-desc:before{content:""}.fa-sort-numeric-asc:before{content:""}.fa-sort-numeric-desc:before{content:""}.fa-thumbs-up:before{content:""}.fa-thumbs-down:before{content:""}.fa-youtube-square:before{content:""}.fa-youtube:before{content:""}.fa-xing:before{content:""}.fa-xing-square:before{content:""}.fa-youtube-play:before{content:""}.fa-dropbox:before{content:""}.fa-stack-overflow:before{content:""}.fa-instagram:before{content:""}.fa-flickr:before{content:""}.fa-adn:before{content:""}.fa-bitbucket:before,.icon-bitbucket:before{content:""}.fa-bitbucket-square:before{content:""}.fa-tumblr:before{content:""}.fa-tumblr-square:before{content:""}.fa-long-arrow-down:before{content:""}.fa-long-arrow-up:before{content:""}.fa-long-arrow-left:before{content:""}.fa-long-arrow-right:before{content:""}.fa-apple:before{content:""}.fa-windows:before{content:""}.fa-android:before{content:""}.fa-linux:before{content:""}.fa-dribbble:before{content:""}.fa-skype:before{content:""}.fa-foursquare:before{content:""}.fa-trello:before{content:""}.fa-female:before{content:""}.fa-male:before{content:""}.fa-gittip:before,.fa-gratipay:before{content:""}.fa-sun-o:before{content:""}.fa-moon-o:before{content:""}.fa-archive:before{content:""}.fa-bug:before{content:""}.fa-vk:before{content:""}.fa-weibo:before{content:""}.fa-renren:before{content:""}.fa-pagelines:before{content:""}.fa-stack-exchange:before{content:""}.fa-arrow-circle-o-right:before{content:""}.fa-arrow-circle-o-left:before{content:""}.fa-caret-square-o-left:before,.fa-toggle-left:before{content:""}.fa-dot-circle-o:before{content:""}.fa-wheelchair:before{content:""}.fa-vimeo-square:before{content:""}.fa-try:before,.fa-turkish-lira:before{content:""}.fa-plus-square-o:before,.wy-menu-vertical li span.toctree-expand:before{content:""}.fa-space-shuttle:before{content:""}.fa-slack:before{content:""}.fa-envelope-square:before{content:""}.fa-wordpress:before{content:""}.fa-openid:before{content:""}.fa-bank:before,.fa-institution:before,.fa-university:before{content:""}.fa-graduation-cap:before,.fa-mortar-board:before{content:""}.fa-yahoo:before{content:""}.fa-google:before{content:""}.fa-reddit:before{content:""}.fa-reddit-square:before{content:""}.fa-stumbleupon-circle:before{content:""}.fa-stumbleupon:before{content:""}.fa-delicious:before{content:""}.fa-digg:before{content:""}.fa-pied-piper-pp:before{content:""}.fa-pied-piper-alt:before{content:""}.fa-drupal:before{content:""}.fa-joomla:before{content:""}.fa-language:before{content:""}.fa-fax:before{content:""}.fa-building:before{content:""}.fa-child:before{content:""}.fa-paw:before{content:""}.fa-spoon:before{content:""}.fa-cube:before{content:""}.fa-cubes:before{content:""}.fa-behance:before{content:""}.fa-behance-square:before{content:""}.fa-steam:before{content:""}.fa-steam-square:before{content:""}.fa-recycle:before{content:""}.fa-automobile:before,.fa-car:before{content:""}.fa-cab:before,.fa-taxi:before{content:""}.fa-tree:before{content:""}.fa-spotify:before{content:""}.fa-deviantart:before{content:""}.fa-soundcloud:before{content:""}.fa-database:before{content:""}.fa-file-pdf-o:before{content:""}.fa-file-word-o:before{content:""}.fa-file-excel-o:before{content:""}.fa-file-powerpoint-o:before{content:""}.fa-file-image-o:before,.fa-file-photo-o:before,.fa-file-picture-o:before{content:""}.fa-file-archive-o:before,.fa-file-zip-o:before{content:""}.fa-file-audio-o:before,.fa-file-sound-o:before{content:""}.fa-file-movie-o:before,.fa-file-video-o:before{content:""}.fa-file-code-o:before{content:""}.fa-vine:before{content:""}.fa-codepen:before{content:""}.fa-jsfiddle:before{content:""}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-ring:before,.fa-life-saver:before,.fa-support:before{content:""}.fa-circle-o-notch:before{content:""}.fa-ra:before,.fa-rebel:before,.fa-resistance:before{content:""}.fa-empire:before,.fa-ge:before{content:""}.fa-git-square:before{content:""}.fa-git:before{content:""}.fa-hacker-news:before,.fa-y-combinator-square:before,.fa-yc-square:before{content:""}.fa-tencent-weibo:before{content:""}.fa-qq:before{content:""}.fa-wechat:before,.fa-weixin:before{content:""}.fa-paper-plane:before,.fa-send:before{content:""}.fa-paper-plane-o:before,.fa-send-o:before{content:""}.fa-history:before{content:""}.fa-circle-thin:before{content:""}.fa-header:before{content:""}.fa-paragraph:before{content:""}.fa-sliders:before{content:""}.fa-share-alt:before{content:""}.fa-share-alt-square:before{content:""}.fa-bomb:before{content:""}.fa-futbol-o:before,.fa-soccer-ball-o:before{content:""}.fa-tty:before{content:""}.fa-binoculars:before{content:""}.fa-plug:before{content:""}.fa-slideshare:before{content:""}.fa-twitch:before{content:""}.fa-yelp:before{content:""}.fa-newspaper-o:before{content:""}.fa-wifi:before{content:""}.fa-calculator:before{content:""}.fa-paypal:before{content:""}.fa-google-wallet:before{content:""}.fa-cc-visa:before{content:""}.fa-cc-mastercard:before{content:""}.fa-cc-discover:before{content:""}.fa-cc-amex:before{content:""}.fa-cc-paypal:before{content:""}.fa-cc-stripe:before{content:""}.fa-bell-slash:before{content:""}.fa-bell-slash-o:before{content:""}.fa-trash:before{content:""}.fa-copyright:before{content:""}.fa-at:before{content:""}.fa-eyedropper:before{content:""}.fa-paint-brush:before{content:""}.fa-birthday-cake:before{content:""}.fa-area-chart:before{content:""}.fa-pie-chart:before{content:""}.fa-line-chart:before{content:""}.fa-lastfm:before{content:""}.fa-lastfm-square:before{content:""}.fa-toggle-off:before{content:""}.fa-toggle-on:before{content:""}.fa-bicycle:before{content:""}.fa-bus:before{content:""}.fa-ioxhost:before{content:""}.fa-angellist:before{content:""}.fa-cc:before{content:""}.fa-ils:before,.fa-shekel:before,.fa-sheqel:before{content:""}.fa-meanpath:before{content:""}.fa-buysellads:before{content:""}.fa-connectdevelop:before{content:""}.fa-dashcube:before{content:""}.fa-forumbee:before{content:""}.fa-leanpub:before{content:""}.fa-sellsy:before{content:""}.fa-shirtsinbulk:before{content:""}.fa-simplybuilt:before{content:""}.fa-skyatlas:before{content:""}.fa-cart-plus:before{content:""}.fa-cart-arrow-down:before{content:""}.fa-diamond:before{content:""}.fa-ship:before{content:""}.fa-user-secret:before{content:""}.fa-motorcycle:before{content:""}.fa-street-view:before{content:""}.fa-heartbeat:before{content:""}.fa-venus:before{content:""}.fa-mars:before{content:""}.fa-mercury:before{content:""}.fa-intersex:before,.fa-transgender:before{content:""}.fa-transgender-alt:before{content:""}.fa-venus-double:before{content:""}.fa-mars-double:before{content:""}.fa-venus-mars:before{content:""}.fa-mars-stroke:before{content:""}.fa-mars-stroke-v:before{content:""}.fa-mars-stroke-h:before{content:""}.fa-neuter:before{content:""}.fa-genderless:before{content:""}.fa-facebook-official:before{content:""}.fa-pinterest-p:before{content:""}.fa-whatsapp:before{content:""}.fa-server:before{content:""}.fa-user-plus:before{content:""}.fa-user-times:before{content:""}.fa-bed:before,.fa-hotel:before{content:""}.fa-viacoin:before{content:""}.fa-train:before{content:""}.fa-subway:before{content:""}.fa-medium:before{content:""}.fa-y-combinator:before,.fa-yc:before{content:""}.fa-optin-monster:before{content:""}.fa-opencart:before{content:""}.fa-expeditedssl:before{content:""}.fa-battery-4:before,.fa-battery-full:before,.fa-battery:before{content:""}.fa-battery-3:before,.fa-battery-three-quarters:before{content:""}.fa-battery-2:before,.fa-battery-half:before{content:""}.fa-battery-1:before,.fa-battery-quarter:before{content:""}.fa-battery-0:before,.fa-battery-empty:before{content:""}.fa-mouse-pointer:before{content:""}.fa-i-cursor:before{content:""}.fa-object-group:before{content:""}.fa-object-ungroup:before{content:""}.fa-sticky-note:before{content:""}.fa-sticky-note-o:before{content:""}.fa-cc-jcb:before{content:""}.fa-cc-diners-club:before{content:""}.fa-clone:before{content:""}.fa-balance-scale:before{content:""}.fa-hourglass-o:before{content:""}.fa-hourglass-1:before,.fa-hourglass-start:before{content:""}.fa-hourglass-2:before,.fa-hourglass-half:before{content:""}.fa-hourglass-3:before,.fa-hourglass-end:before{content:""}.fa-hourglass:before{content:""}.fa-hand-grab-o:before,.fa-hand-rock-o:before{content:""}.fa-hand-paper-o:before,.fa-hand-stop-o:before{content:""}.fa-hand-scissors-o:before{content:""}.fa-hand-lizard-o:before{content:""}.fa-hand-spock-o:before{content:""}.fa-hand-pointer-o:before{content:""}.fa-hand-peace-o:before{content:""}.fa-trademark:before{content:""}.fa-registered:before{content:""}.fa-creative-commons:before{content:""}.fa-gg:before{content:""}.fa-gg-circle:before{content:""}.fa-tripadvisor:before{content:""}.fa-odnoklassniki:before{content:""}.fa-odnoklassniki-square:before{content:""}.fa-get-pocket:before{content:""}.fa-wikipedia-w:before{content:""}.fa-safari:before{content:""}.fa-chrome:before{content:""}.fa-firefox:before{content:""}.fa-opera:before{content:""}.fa-internet-explorer:before{content:""}.fa-television:before,.fa-tv:before{content:""}.fa-contao:before{content:""}.fa-500px:before{content:""}.fa-amazon:before{content:""}.fa-calendar-plus-o:before{content:""}.fa-calendar-minus-o:before{content:""}.fa-calendar-times-o:before{content:""}.fa-calendar-check-o:before{content:""}.fa-industry:before{content:""}.fa-map-pin:before{content:""}.fa-map-signs:before{content:""}.fa-map-o:before{content:""}.fa-map:before{content:""}.fa-commenting:before{content:""}.fa-commenting-o:before{content:""}.fa-houzz:before{content:""}.fa-vimeo:before{content:""}.fa-black-tie:before{content:""}.fa-fonticons:before{content:""}.fa-reddit-alien:before{content:""}.fa-edge:before{content:""}.fa-credit-card-alt:before{content:""}.fa-codiepie:before{content:""}.fa-modx:before{content:""}.fa-fort-awesome:before{content:""}.fa-usb:before{content:""}.fa-product-hunt:before{content:""}.fa-mixcloud:before{content:""}.fa-scribd:before{content:""}.fa-pause-circle:before{content:""}.fa-pause-circle-o:before{content:""}.fa-stop-circle:before{content:""}.fa-stop-circle-o:before{content:""}.fa-shopping-bag:before{content:""}.fa-shopping-basket:before{content:""}.fa-hashtag:before{content:""}.fa-bluetooth:before{content:""}.fa-bluetooth-b:before{content:""}.fa-percent:before{content:""}.fa-gitlab:before,.icon-gitlab:before{content:""}.fa-wpbeginner:before{content:""}.fa-wpforms:before{content:""}.fa-envira:before{content:""}.fa-universal-access:before{content:""}.fa-wheelchair-alt:before{content:""}.fa-question-circle-o:before{content:""}.fa-blind:before{content:""}.fa-audio-description:before{content:""}.fa-volume-control-phone:before{content:""}.fa-braille:before{content:""}.fa-assistive-listening-systems:before{content:""}.fa-american-sign-language-interpreting:before,.fa-asl-interpreting:before{content:""}.fa-deaf:before,.fa-deafness:before,.fa-hard-of-hearing:before{content:""}.fa-glide:before{content:""}.fa-glide-g:before{content:""}.fa-sign-language:before,.fa-signing:before{content:""}.fa-low-vision:before{content:""}.fa-viadeo:before{content:""}.fa-viadeo-square:before{content:""}.fa-snapchat:before{content:""}.fa-snapchat-ghost:before{content:""}.fa-snapchat-square:before{content:""}.fa-pied-piper:before{content:""}.fa-first-order:before{content:""}.fa-yoast:before{content:""}.fa-themeisle:before{content:""}.fa-google-plus-circle:before,.fa-google-plus-official:before{content:""}.fa-fa:before,.fa-font-awesome:before{content:""}.fa-handshake-o:before{content:""}.fa-envelope-open:before{content:""}.fa-envelope-open-o:before{content:""}.fa-linode:before{content:""}.fa-address-book:before{content:""}.fa-address-book-o:before{content:""}.fa-address-card:before,.fa-vcard:before{content:""}.fa-address-card-o:before,.fa-vcard-o:before{content:""}.fa-user-circle:before{content:""}.fa-user-circle-o:before{content:""}.fa-user-o:before{content:""}.fa-id-badge:before{content:""}.fa-drivers-license:before,.fa-id-card:before{content:""}.fa-drivers-license-o:before,.fa-id-card-o:before{content:""}.fa-quora:before{content:""}.fa-free-code-camp:before{content:""}.fa-telegram:before{content:""}.fa-thermometer-4:before,.fa-thermometer-full:before,.fa-thermometer:before{content:""}.fa-thermometer-3:before,.fa-thermometer-three-quarters:before{content:""}.fa-thermometer-2:before,.fa-thermometer-half:before{content:""}.fa-thermometer-1:before,.fa-thermometer-quarter:before{content:""}.fa-thermometer-0:before,.fa-thermometer-empty:before{content:""}.fa-shower:before{content:""}.fa-bath:before,.fa-bathtub:before,.fa-s15:before{content:""}.fa-podcast:before{content:""}.fa-window-maximize:before{content:""}.fa-window-minimize:before{content:""}.fa-window-restore:before{content:""}.fa-times-rectangle:before,.fa-window-close:before{content:""}.fa-times-rectangle-o:before,.fa-window-close-o:before{content:""}.fa-bandcamp:before{content:""}.fa-grav:before{content:""}.fa-etsy:before{content:""}.fa-imdb:before{content:""}.fa-ravelry:before{content:""}.fa-eercast:before{content:""}.fa-microchip:before{content:""}.fa-snowflake-o:before{content:""}.fa-superpowers:before{content:""}.fa-wpexplorer:before{content:""}.fa-meetup:before{content:""}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-dropdown .caret,.wy-inline-validate.wy-inline-validate-danger .wy-input-context,.wy-inline-validate.wy-inline-validate-info .wy-input-context,.wy-inline-validate.wy-inline-validate-success .wy-input-context,.wy-inline-validate.wy-inline-validate-warning .wy-input-context,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{font-family:inherit}.fa:before,.icon:before,.rst-content .admonition-title:before,.rst-content .code-block-caption .headerlink:before,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before{font-family:FontAwesome;display:inline-block;font-style:normal;font-weight:400;line-height:1;text-decoration:inherit}.rst-content .code-block-caption a .headerlink,.rst-content a .admonition-title,.rst-content code.download a span:first-child,.rst-content dl dt a .headerlink,.rst-content h1 a .headerlink,.rst-content h2 a .headerlink,.rst-content h3 a .headerlink,.rst-content h4 a .headerlink,.rst-content h5 a .headerlink,.rst-content h6 a .headerlink,.rst-content p.caption a .headerlink,.rst-content table>caption a .headerlink,.rst-content tt.download a span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li a span.toctree-expand,a .fa,a .icon,a .rst-content .admonition-title,a .rst-content .code-block-caption .headerlink,a .rst-content code.download span:first-child,a .rst-content dl dt .headerlink,a .rst-content h1 .headerlink,a .rst-content h2 .headerlink,a .rst-content h3 .headerlink,a .rst-content h4 .headerlink,a .rst-content h5 .headerlink,a .rst-content h6 .headerlink,a .rst-content p.caption .headerlink,a .rst-content table>caption .headerlink,a .rst-content tt.download span:first-child,a .wy-menu-vertical li span.toctree-expand{display:inline-block;text-decoration:inherit}.btn .fa,.btn .icon,.btn .rst-content .admonition-title,.btn .rst-content .code-block-caption .headerlink,.btn .rst-content code.download span:first-child,.btn .rst-content dl dt .headerlink,.btn .rst-content h1 .headerlink,.btn .rst-content h2 .headerlink,.btn .rst-content h3 .headerlink,.btn .rst-content h4 .headerlink,.btn .rst-content h5 .headerlink,.btn .rst-content h6 .headerlink,.btn .rst-content p.caption .headerlink,.btn .rst-content table>caption .headerlink,.btn .rst-content tt.download span:first-child,.btn .wy-menu-vertical li.current>a span.toctree-expand,.btn .wy-menu-vertical li.on a span.toctree-expand,.btn .wy-menu-vertical li span.toctree-expand,.nav .fa,.nav .icon,.nav .rst-content .admonition-title,.nav .rst-content .code-block-caption .headerlink,.nav .rst-content code.download span:first-child,.nav .rst-content dl dt .headerlink,.nav .rst-content h1 .headerlink,.nav .rst-content h2 .headerlink,.nav .rst-content h3 .headerlink,.nav .rst-content h4 .headerlink,.nav .rst-content h5 .headerlink,.nav .rst-content h6 .headerlink,.nav .rst-content p.caption .headerlink,.nav .rst-content table>caption .headerlink,.nav .rst-content tt.download span:first-child,.nav .wy-menu-vertical li.current>a span.toctree-expand,.nav .wy-menu-vertical li.on a span.toctree-expand,.nav .wy-menu-vertical li span.toctree-expand,.rst-content .btn .admonition-title,.rst-content .code-block-caption .btn .headerlink,.rst-content .code-block-caption .nav .headerlink,.rst-content .nav .admonition-title,.rst-content code.download .btn span:first-child,.rst-content code.download .nav span:first-child,.rst-content dl dt .btn .headerlink,.rst-content dl dt .nav .headerlink,.rst-content h1 .btn .headerlink,.rst-content h1 .nav .headerlink,.rst-content h2 .btn .headerlink,.rst-content h2 .nav .headerlink,.rst-content h3 .btn .headerlink,.rst-content h3 .nav .headerlink,.rst-content h4 .btn .headerlink,.rst-content h4 .nav .headerlink,.rst-content h5 .btn .headerlink,.rst-content h5 .nav .headerlink,.rst-content h6 .btn .headerlink,.rst-content h6 .nav .headerlink,.rst-content p.caption .btn .headerlink,.rst-content p.caption .nav .headerlink,.rst-content table>caption .btn .headerlink,.rst-content table>caption .nav .headerlink,.rst-content tt.download .btn span:first-child,.rst-content tt.download .nav span:first-child,.wy-menu-vertical li .btn span.toctree-expand,.wy-menu-vertical li.current>a .btn span.toctree-expand,.wy-menu-vertical li.current>a .nav span.toctree-expand,.wy-menu-vertical li .nav span.toctree-expand,.wy-menu-vertical li.on a .btn span.toctree-expand,.wy-menu-vertical li.on a .nav span.toctree-expand{display:inline}.btn .fa-large.icon,.btn .fa.fa-large,.btn .rst-content .code-block-caption .fa-large.headerlink,.btn .rst-content .fa-large.admonition-title,.btn .rst-content code.download span.fa-large:first-child,.btn .rst-content dl dt .fa-large.headerlink,.btn .rst-content h1 .fa-large.headerlink,.btn .rst-content h2 .fa-large.headerlink,.btn .rst-content h3 .fa-large.headerlink,.btn .rst-content h4 .fa-large.headerlink,.btn .rst-content h5 .fa-large.headerlink,.btn .rst-content h6 .fa-large.headerlink,.btn .rst-content p.caption .fa-large.headerlink,.btn .rst-content table>caption .fa-large.headerlink,.btn .rst-content tt.download span.fa-large:first-child,.btn .wy-menu-vertical li span.fa-large.toctree-expand,.nav .fa-large.icon,.nav .fa.fa-large,.nav .rst-content .code-block-caption .fa-large.headerlink,.nav .rst-content .fa-large.admonition-title,.nav .rst-content code.download span.fa-large:first-child,.nav .rst-content dl dt .fa-large.headerlink,.nav .rst-content h1 .fa-large.headerlink,.nav .rst-content h2 .fa-large.headerlink,.nav .rst-content h3 .fa-large.headerlink,.nav .rst-content h4 .fa-large.headerlink,.nav .rst-content h5 .fa-large.headerlink,.nav .rst-content h6 .fa-large.headerlink,.nav .rst-content p.caption .fa-large.headerlink,.nav .rst-content table>caption .fa-large.headerlink,.nav .rst-content tt.download span.fa-large:first-child,.nav .wy-menu-vertical li span.fa-large.toctree-expand,.rst-content .btn .fa-large.admonition-title,.rst-content .code-block-caption .btn .fa-large.headerlink,.rst-content .code-block-caption .nav .fa-large.headerlink,.rst-content .nav .fa-large.admonition-title,.rst-content code.download .btn span.fa-large:first-child,.rst-content code.download .nav span.fa-large:first-child,.rst-content dl dt .btn .fa-large.headerlink,.rst-content dl dt .nav .fa-large.headerlink,.rst-content h1 .btn .fa-large.headerlink,.rst-content h1 .nav .fa-large.headerlink,.rst-content h2 .btn .fa-large.headerlink,.rst-content h2 .nav .fa-large.headerlink,.rst-content h3 .btn .fa-large.headerlink,.rst-content h3 .nav .fa-large.headerlink,.rst-content h4 .btn .fa-large.headerlink,.rst-content h4 .nav .fa-large.headerlink,.rst-content h5 .btn .fa-large.headerlink,.rst-content h5 .nav .fa-large.headerlink,.rst-content h6 .btn .fa-large.headerlink,.rst-content h6 .nav .fa-large.headerlink,.rst-content p.caption .btn .fa-large.headerlink,.rst-content p.caption .nav .fa-large.headerlink,.rst-content table>caption .btn .fa-large.headerlink,.rst-content table>caption .nav .fa-large.headerlink,.rst-content tt.download .btn span.fa-large:first-child,.rst-content tt.download .nav span.fa-large:first-child,.wy-menu-vertical li .btn span.fa-large.toctree-expand,.wy-menu-vertical li .nav span.fa-large.toctree-expand{line-height:.9em}.btn .fa-spin.icon,.btn .fa.fa-spin,.btn .rst-content .code-block-caption .fa-spin.headerlink,.btn .rst-content .fa-spin.admonition-title,.btn .rst-content code.download span.fa-spin:first-child,.btn .rst-content dl dt .fa-spin.headerlink,.btn .rst-content h1 .fa-spin.headerlink,.btn .rst-content h2 .fa-spin.headerlink,.btn .rst-content h3 .fa-spin.headerlink,.btn .rst-content h4 .fa-spin.headerlink,.btn .rst-content h5 .fa-spin.headerlink,.btn .rst-content h6 .fa-spin.headerlink,.btn .rst-content p.caption .fa-spin.headerlink,.btn .rst-content table>caption .fa-spin.headerlink,.btn .rst-content tt.download span.fa-spin:first-child,.btn .wy-menu-vertical li span.fa-spin.toctree-expand,.nav .fa-spin.icon,.nav .fa.fa-spin,.nav .rst-content .code-block-caption .fa-spin.headerlink,.nav .rst-content .fa-spin.admonition-title,.nav .rst-content code.download span.fa-spin:first-child,.nav .rst-content dl dt .fa-spin.headerlink,.nav .rst-content h1 .fa-spin.headerlink,.nav .rst-content h2 .fa-spin.headerlink,.nav .rst-content h3 .fa-spin.headerlink,.nav .rst-content h4 .fa-spin.headerlink,.nav .rst-content h5 .fa-spin.headerlink,.nav .rst-content h6 .fa-spin.headerlink,.nav .rst-content p.caption .fa-spin.headerlink,.nav .rst-content table>caption .fa-spin.headerlink,.nav .rst-content tt.download span.fa-spin:first-child,.nav .wy-menu-vertical li span.fa-spin.toctree-expand,.rst-content .btn .fa-spin.admonition-title,.rst-content .code-block-caption .btn .fa-spin.headerlink,.rst-content .code-block-caption .nav .fa-spin.headerlink,.rst-content .nav .fa-spin.admonition-title,.rst-content code.download .btn span.fa-spin:first-child,.rst-content code.download .nav span.fa-spin:first-child,.rst-content dl dt .btn .fa-spin.headerlink,.rst-content dl dt .nav .fa-spin.headerlink,.rst-content h1 .btn .fa-spin.headerlink,.rst-content h1 .nav .fa-spin.headerlink,.rst-content h2 .btn .fa-spin.headerlink,.rst-content h2 .nav .fa-spin.headerlink,.rst-content h3 .btn .fa-spin.headerlink,.rst-content h3 .nav .fa-spin.headerlink,.rst-content h4 .btn .fa-spin.headerlink,.rst-content h4 .nav .fa-spin.headerlink,.rst-content h5 .btn .fa-spin.headerlink,.rst-content h5 .nav .fa-spin.headerlink,.rst-content h6 .btn .fa-spin.headerlink,.rst-content h6 .nav .fa-spin.headerlink,.rst-content p.caption .btn .fa-spin.headerlink,.rst-content p.caption .nav .fa-spin.headerlink,.rst-content table>caption .btn .fa-spin.headerlink,.rst-content table>caption .nav .fa-spin.headerlink,.rst-content tt.download .btn span.fa-spin:first-child,.rst-content tt.download .nav span.fa-spin:first-child,.wy-menu-vertical li .btn span.fa-spin.toctree-expand,.wy-menu-vertical li .nav span.fa-spin.toctree-expand{display:inline-block}.btn.fa:before,.btn.icon:before,.rst-content .btn.admonition-title:before,.rst-content .code-block-caption .btn.headerlink:before,.rst-content code.download span.btn:first-child:before,.rst-content dl dt .btn.headerlink:before,.rst-content h1 .btn.headerlink:before,.rst-content h2 .btn.headerlink:before,.rst-content h3 .btn.headerlink:before,.rst-content h4 .btn.headerlink:before,.rst-content h5 .btn.headerlink:before,.rst-content h6 .btn.headerlink:before,.rst-content p.caption .btn.headerlink:before,.rst-content table>caption .btn.headerlink:before,.rst-content tt.download span.btn:first-child:before,.wy-menu-vertical li span.btn.toctree-expand:before{opacity:.5;-webkit-transition:opacity .05s ease-in;-moz-transition:opacity .05s ease-in;transition:opacity .05s ease-in}.btn.fa:hover:before,.btn.icon:hover:before,.rst-content .btn.admonition-title:hover:before,.rst-content .code-block-caption .btn.headerlink:hover:before,.rst-content code.download span.btn:first-child:hover:before,.rst-content dl dt .btn.headerlink:hover:before,.rst-content h1 .btn.headerlink:hover:before,.rst-content h2 .btn.headerlink:hover:before,.rst-content h3 .btn.headerlink:hover:before,.rst-content h4 .btn.headerlink:hover:before,.rst-content h5 .btn.headerlink:hover:before,.rst-content h6 .btn.headerlink:hover:before,.rst-content p.caption .btn.headerlink:hover:before,.rst-content table>caption .btn.headerlink:hover:before,.rst-content tt.download span.btn:first-child:hover:before,.wy-menu-vertical li span.btn.toctree-expand:hover:before{opacity:1}.btn-mini .fa:before,.btn-mini .icon:before,.btn-mini .rst-content .admonition-title:before,.btn-mini .rst-content .code-block-caption .headerlink:before,.btn-mini .rst-content code.download span:first-child:before,.btn-mini .rst-content dl dt .headerlink:before,.btn-mini .rst-content h1 .headerlink:before,.btn-mini .rst-content h2 .headerlink:before,.btn-mini .rst-content h3 .headerlink:before,.btn-mini .rst-content h4 .headerlink:before,.btn-mini .rst-content h5 .headerlink:before,.btn-mini .rst-content h6 .headerlink:before,.btn-mini .rst-content p.caption .headerlink:before,.btn-mini .rst-content table>caption .headerlink:before,.btn-mini .rst-content tt.download span:first-child:before,.btn-mini .wy-menu-vertical li span.toctree-expand:before,.rst-content .btn-mini .admonition-title:before,.rst-content .code-block-caption .btn-mini .headerlink:before,.rst-content code.download .btn-mini span:first-child:before,.rst-content dl dt .btn-mini .headerlink:before,.rst-content h1 .btn-mini .headerlink:before,.rst-content h2 .btn-mini .headerlink:before,.rst-content h3 .btn-mini .headerlink:before,.rst-content h4 .btn-mini .headerlink:before,.rst-content h5 .btn-mini .headerlink:before,.rst-content h6 .btn-mini .headerlink:before,.rst-content p.caption .btn-mini .headerlink:before,.rst-content table>caption .btn-mini .headerlink:before,.rst-content tt.download .btn-mini span:first-child:before,.wy-menu-vertical li .btn-mini span.toctree-expand:before{font-size:14px;vertical-align:-15%}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.wy-alert{padding:12px;line-height:24px;margin-bottom:24px;background:#e7f2fa}.rst-content .admonition-title,.wy-alert-title{font-weight:700;display:block;color:#fff;background:#6ab0de;padding:6px 12px;margin:-12px -12px 12px}.rst-content .danger,.rst-content .error,.rst-content .wy-alert-danger.admonition,.rst-content .wy-alert-danger.admonition-todo,.rst-content .wy-alert-danger.attention,.rst-content .wy-alert-danger.caution,.rst-content .wy-alert-danger.hint,.rst-content .wy-alert-danger.important,.rst-content .wy-alert-danger.note,.rst-content .wy-alert-danger.seealso,.rst-content .wy-alert-danger.tip,.rst-content .wy-alert-danger.warning,.wy-alert.wy-alert-danger{background:#fdf3f2}.rst-content .danger .admonition-title,.rst-content .danger .wy-alert-title,.rst-content .error .admonition-title,.rst-content .error .wy-alert-title,.rst-content .wy-alert-danger.admonition-todo .admonition-title,.rst-content .wy-alert-danger.admonition-todo .wy-alert-title,.rst-content .wy-alert-danger.admonition .admonition-title,.rst-content .wy-alert-danger.admonition .wy-alert-title,.rst-content .wy-alert-danger.attention .admonition-title,.rst-content .wy-alert-danger.attention .wy-alert-title,.rst-content .wy-alert-danger.caution .admonition-title,.rst-content .wy-alert-danger.caution .wy-alert-title,.rst-content .wy-alert-danger.hint .admonition-title,.rst-content .wy-alert-danger.hint .wy-alert-title,.rst-content .wy-alert-danger.important .admonition-title,.rst-content .wy-alert-danger.important .wy-alert-title,.rst-content .wy-alert-danger.note .admonition-title,.rst-content .wy-alert-danger.note .wy-alert-title,.rst-content .wy-alert-danger.seealso .admonition-title,.rst-content .wy-alert-danger.seealso .wy-alert-title,.rst-content .wy-alert-danger.tip .admonition-title,.rst-content .wy-alert-danger.tip .wy-alert-title,.rst-content .wy-alert-danger.warning .admonition-title,.rst-content .wy-alert-danger.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-danger .admonition-title,.wy-alert.wy-alert-danger .rst-content .admonition-title,.wy-alert.wy-alert-danger .wy-alert-title{background:#f29f97}.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .warning,.rst-content .wy-alert-warning.admonition,.rst-content .wy-alert-warning.danger,.rst-content .wy-alert-warning.error,.rst-content .wy-alert-warning.hint,.rst-content .wy-alert-warning.important,.rst-content .wy-alert-warning.note,.rst-content .wy-alert-warning.seealso,.rst-content .wy-alert-warning.tip,.wy-alert.wy-alert-warning{background:#ffedcc}.rst-content .admonition-todo .admonition-title,.rst-content .admonition-todo .wy-alert-title,.rst-content .attention .admonition-title,.rst-content .attention .wy-alert-title,.rst-content .caution .admonition-title,.rst-content .caution .wy-alert-title,.rst-content .warning .admonition-title,.rst-content .warning .wy-alert-title,.rst-content .wy-alert-warning.admonition .admonition-title,.rst-content .wy-alert-warning.admonition .wy-alert-title,.rst-content .wy-alert-warning.danger .admonition-title,.rst-content .wy-alert-warning.danger .wy-alert-title,.rst-content .wy-alert-warning.error .admonition-title,.rst-content .wy-alert-warning.error .wy-alert-title,.rst-content .wy-alert-warning.hint .admonition-title,.rst-content .wy-alert-warning.hint .wy-alert-title,.rst-content .wy-alert-warning.important .admonition-title,.rst-content .wy-alert-warning.important .wy-alert-title,.rst-content .wy-alert-warning.note .admonition-title,.rst-content .wy-alert-warning.note .wy-alert-title,.rst-content .wy-alert-warning.seealso .admonition-title,.rst-content .wy-alert-warning.seealso .wy-alert-title,.rst-content .wy-alert-warning.tip .admonition-title,.rst-content .wy-alert-warning.tip .wy-alert-title,.rst-content .wy-alert.wy-alert-warning .admonition-title,.wy-alert.wy-alert-warning .rst-content .admonition-title,.wy-alert.wy-alert-warning .wy-alert-title{background:#f0b37e}.rst-content .note,.rst-content .seealso,.rst-content .wy-alert-info.admonition,.rst-content .wy-alert-info.admonition-todo,.rst-content .wy-alert-info.attention,.rst-content .wy-alert-info.caution,.rst-content .wy-alert-info.danger,.rst-content .wy-alert-info.error,.rst-content .wy-alert-info.hint,.rst-content .wy-alert-info.important,.rst-content .wy-alert-info.tip,.rst-content .wy-alert-info.warning,.wy-alert.wy-alert-info{background:#e7f2fa}.rst-content .note .admonition-title,.rst-content .note .wy-alert-title,.rst-content .seealso .admonition-title,.rst-content .seealso .wy-alert-title,.rst-content .wy-alert-info.admonition-todo .admonition-title,.rst-content .wy-alert-info.admonition-todo .wy-alert-title,.rst-content .wy-alert-info.admonition .admonition-title,.rst-content .wy-alert-info.admonition .wy-alert-title,.rst-content .wy-alert-info.attention .admonition-title,.rst-content .wy-alert-info.attention .wy-alert-title,.rst-content .wy-alert-info.caution .admonition-title,.rst-content .wy-alert-info.caution .wy-alert-title,.rst-content .wy-alert-info.danger .admonition-title,.rst-content .wy-alert-info.danger .wy-alert-title,.rst-content .wy-alert-info.error .admonition-title,.rst-content .wy-alert-info.error .wy-alert-title,.rst-content .wy-alert-info.hint .admonition-title,.rst-content .wy-alert-info.hint .wy-alert-title,.rst-content .wy-alert-info.important .admonition-title,.rst-content .wy-alert-info.important .wy-alert-title,.rst-content .wy-alert-info.tip .admonition-title,.rst-content .wy-alert-info.tip .wy-alert-title,.rst-content .wy-alert-info.warning .admonition-title,.rst-content .wy-alert-info.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-info .admonition-title,.wy-alert.wy-alert-info .rst-content .admonition-title,.wy-alert.wy-alert-info .wy-alert-title{background:#6ab0de}.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .wy-alert-success.admonition,.rst-content .wy-alert-success.admonition-todo,.rst-content .wy-alert-success.attention,.rst-content .wy-alert-success.caution,.rst-content .wy-alert-success.danger,.rst-content .wy-alert-success.error,.rst-content .wy-alert-success.note,.rst-content .wy-alert-success.seealso,.rst-content .wy-alert-success.warning,.wy-alert.wy-alert-success{background:#dbfaf4}.rst-content .hint .admonition-title,.rst-content .hint .wy-alert-title,.rst-content .important .admonition-title,.rst-content .important .wy-alert-title,.rst-content .tip .admonition-title,.rst-content .tip .wy-alert-title,.rst-content .wy-alert-success.admonition-todo .admonition-title,.rst-content .wy-alert-success.admonition-todo .wy-alert-title,.rst-content .wy-alert-success.admonition .admonition-title,.rst-content .wy-alert-success.admonition .wy-alert-title,.rst-content .wy-alert-success.attention .admonition-title,.rst-content .wy-alert-success.attention .wy-alert-title,.rst-content .wy-alert-success.caution .admonition-title,.rst-content .wy-alert-success.caution .wy-alert-title,.rst-content .wy-alert-success.danger .admonition-title,.rst-content .wy-alert-success.danger .wy-alert-title,.rst-content .wy-alert-success.error .admonition-title,.rst-content .wy-alert-success.error .wy-alert-title,.rst-content .wy-alert-success.note .admonition-title,.rst-content .wy-alert-success.note .wy-alert-title,.rst-content .wy-alert-success.seealso .admonition-title,.rst-content .wy-alert-success.seealso .wy-alert-title,.rst-content .wy-alert-success.warning .admonition-title,.rst-content .wy-alert-success.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-success .admonition-title,.wy-alert.wy-alert-success .rst-content .admonition-title,.wy-alert.wy-alert-success .wy-alert-title{background:#1abc9c}.rst-content .wy-alert-neutral.admonition,.rst-content .wy-alert-neutral.admonition-todo,.rst-content .wy-alert-neutral.attention,.rst-content .wy-alert-neutral.caution,.rst-content .wy-alert-neutral.danger,.rst-content .wy-alert-neutral.error,.rst-content .wy-alert-neutral.hint,.rst-content .wy-alert-neutral.important,.rst-content .wy-alert-neutral.note,.rst-content .wy-alert-neutral.seealso,.rst-content .wy-alert-neutral.tip,.rst-content .wy-alert-neutral.warning,.wy-alert.wy-alert-neutral{background:#f3f6f6}.rst-content .wy-alert-neutral.admonition-todo .admonition-title,.rst-content .wy-alert-neutral.admonition-todo .wy-alert-title,.rst-content .wy-alert-neutral.admonition .admonition-title,.rst-content .wy-alert-neutral.admonition .wy-alert-title,.rst-content .wy-alert-neutral.attention .admonition-title,.rst-content .wy-alert-neutral.attention .wy-alert-title,.rst-content .wy-alert-neutral.caution .admonition-title,.rst-content .wy-alert-neutral.caution .wy-alert-title,.rst-content .wy-alert-neutral.danger .admonition-title,.rst-content .wy-alert-neutral.danger .wy-alert-title,.rst-content .wy-alert-neutral.error .admonition-title,.rst-content .wy-alert-neutral.error .wy-alert-title,.rst-content .wy-alert-neutral.hint .admonition-title,.rst-content .wy-alert-neutral.hint .wy-alert-title,.rst-content .wy-alert-neutral.important .admonition-title,.rst-content .wy-alert-neutral.important .wy-alert-title,.rst-content .wy-alert-neutral.note .admonition-title,.rst-content .wy-alert-neutral.note .wy-alert-title,.rst-content .wy-alert-neutral.seealso .admonition-title,.rst-content .wy-alert-neutral.seealso .wy-alert-title,.rst-content .wy-alert-neutral.tip .admonition-title,.rst-content .wy-alert-neutral.tip .wy-alert-title,.rst-content .wy-alert-neutral.warning .admonition-title,.rst-content .wy-alert-neutral.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-neutral .admonition-title,.wy-alert.wy-alert-neutral .rst-content .admonition-title,.wy-alert.wy-alert-neutral .wy-alert-title{color:#404040;background:#e1e4e5}.rst-content .wy-alert-neutral.admonition-todo a,.rst-content .wy-alert-neutral.admonition a,.rst-content .wy-alert-neutral.attention a,.rst-content .wy-alert-neutral.caution a,.rst-content .wy-alert-neutral.danger a,.rst-content .wy-alert-neutral.error a,.rst-content .wy-alert-neutral.hint a,.rst-content .wy-alert-neutral.important a,.rst-content .wy-alert-neutral.note a,.rst-content .wy-alert-neutral.seealso a,.rst-content .wy-alert-neutral.tip a,.rst-content .wy-alert-neutral.warning a,.wy-alert.wy-alert-neutral a{color:#2980b9}.rst-content .admonition-todo p:last-child,.rst-content .admonition p:last-child,.rst-content .attention p:last-child,.rst-content .caution p:last-child,.rst-content .danger p:last-child,.rst-content .error p:last-child,.rst-content .hint p:last-child,.rst-content .important p:last-child,.rst-content .note p:last-child,.rst-content .seealso p:last-child,.rst-content .tip p:last-child,.rst-content .warning p:last-child,.wy-alert p:last-child{margin-bottom:0}.wy-tray-container{position:fixed;bottom:0;left:0;z-index:600}.wy-tray-container li{display:block;width:300px;background:transparent;color:#fff;text-align:center;box-shadow:0 5px 5px 0 rgba(0,0,0,.1);padding:0 24px;min-width:20%;opacity:0;height:0;line-height:56px;overflow:hidden;-webkit-transition:all .3s ease-in;-moz-transition:all .3s ease-in;transition:all .3s ease-in}.wy-tray-container li.wy-tray-item-success{background:#27ae60}.wy-tray-container li.wy-tray-item-info{background:#2980b9}.wy-tray-container li.wy-tray-item-warning{background:#e67e22}.wy-tray-container li.wy-tray-item-danger{background:#e74c3c}.wy-tray-container li.on{opacity:1;height:56px}@media screen and (max-width:768px){.wy-tray-container{bottom:auto;top:0;width:100%}.wy-tray-container li{width:100%}}button{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle;cursor:pointer;line-height:normal;-webkit-appearance:button;*overflow:visible}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}button[disabled]{cursor:default}.btn{display:inline-block;border-radius:2px;line-height:normal;white-space:nowrap;text-align:center;cursor:pointer;font-size:100%;padding:6px 12px 8px;color:#fff;border:1px solid rgba(0,0,0,.1);background-color:#27ae60;text-decoration:none;font-weight:400;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 2px -1px hsla(0,0%,100%,.5),inset 0 -2px 0 0 rgba(0,0,0,.1);outline-none:false;vertical-align:middle;*display:inline;zoom:1;-webkit-user-drag:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transition:all .1s linear;-moz-transition:all .1s linear;transition:all .1s linear}.btn-hover{background:#2e8ece;color:#fff}.btn:hover{background:#2cc36b;color:#fff}.btn:focus{background:#2cc36b;outline:0}.btn:active{box-shadow:inset 0 -1px 0 0 rgba(0,0,0,.05),inset 0 2px 0 0 rgba(0,0,0,.1);padding:8px 12px 6px}.btn:visited{color:#fff}.btn-disabled,.btn-disabled:active,.btn-disabled:focus,.btn-disabled:hover,.btn:disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:.4;cursor:not-allowed;box-shadow:none}.btn::-moz-focus-inner{padding:0;border:0}.btn-small{font-size:80%}.btn-info{background-color:#2980b9!important}.btn-info:hover{background-color:#2e8ece!important}.btn-neutral{background-color:#f3f6f6!important;color:#404040!important}.btn-neutral:hover{background-color:#e5ebeb!important;color:#404040}.btn-neutral:visited{color:#404040!important}.btn-success{background-color:#27ae60!important}.btn-success:hover{background-color:#295!important}.btn-danger{background-color:#e74c3c!important}.btn-danger:hover{background-color:#ea6153!important}.btn-warning{background-color:#e67e22!important}.btn-warning:hover{background-color:#e98b39!important}.btn-invert{background-color:#222}.btn-invert:hover{background-color:#2f2f2f!important}.btn-link{background-color:transparent!important;color:#2980b9;box-shadow:none;border-color:transparent!important}.btn-link:active,.btn-link:hover{background-color:transparent!important;color:#409ad5!important;box-shadow:none}.btn-link:visited{color:#9b59b6}.wy-btn-group .btn,.wy-control .btn{vertical-align:middle}.wy-btn-group{margin-bottom:24px;*zoom:1}.wy-btn-group:after,.wy-btn-group:before{display:table;content:""}.wy-btn-group:after{clear:both}.wy-dropdown{position:relative;display:inline-block}.wy-dropdown-active .wy-dropdown-menu{display:block}.wy-dropdown-menu{position:absolute;left:0;display:none;float:left;top:100%;min-width:100%;background:#fcfcfc;z-index:100;border:1px solid #cfd7dd;box-shadow:0 2px 2px 0 rgba(0,0,0,.1);padding:12px}.wy-dropdown-menu>dd>a{display:block;clear:both;color:#404040;white-space:nowrap;font-size:90%;padding:0 12px;cursor:pointer}.wy-dropdown-menu>dd>a:hover{background:#2980b9;color:#fff}.wy-dropdown-menu>dd.divider{border-top:1px solid #cfd7dd;margin:6px 0}.wy-dropdown-menu>dd.search{padding-bottom:12px}.wy-dropdown-menu>dd.search input[type=search]{width:100%}.wy-dropdown-menu>dd.call-to-action{background:#e3e3e3;text-transform:uppercase;font-weight:500;font-size:80%}.wy-dropdown-menu>dd.call-to-action:hover{background:#e3e3e3}.wy-dropdown-menu>dd.call-to-action .btn{color:#fff}.wy-dropdown.wy-dropdown-up .wy-dropdown-menu{bottom:100%;top:auto;left:auto;right:0}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu{background:#fcfcfc;margin-top:2px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a{padding:6px 12px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a:hover{background:#2980b9;color:#fff}.wy-dropdown.wy-dropdown-left .wy-dropdown-menu{right:0;left:auto;text-align:right}.wy-dropdown-arrow:before{content:" ";border-bottom:5px solid #f5f5f5;border-left:5px solid transparent;border-right:5px solid transparent;position:absolute;display:block;top:-4px;left:50%;margin-left:-3px}.wy-dropdown-arrow.wy-dropdown-arrow-left:before{left:11px}.wy-form-stacked select{display:block}.wy-form-aligned .wy-help-inline,.wy-form-aligned input,.wy-form-aligned label,.wy-form-aligned select,.wy-form-aligned textarea{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-form-aligned .wy-control-group>label{display:inline-block;vertical-align:middle;width:10em;margin:6px 12px 0 0;float:left}.wy-form-aligned .wy-control{float:left}.wy-form-aligned .wy-control label{display:block}.wy-form-aligned .wy-control select{margin-top:6px}fieldset{margin:0}fieldset,legend{border:0;padding:0}legend{width:100%;white-space:normal;margin-bottom:24px;font-size:150%;*margin-left:-7px}label,legend{display:block}label{margin:0 0 .3125em;color:#333;font-size:90%}input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}.wy-control-group{margin-bottom:24px;max-width:1200px;margin-left:auto;margin-right:auto;*zoom:1}.wy-control-group:after,.wy-control-group:before{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group.wy-control-group-required>label:after{content:" *";color:#e74c3c}.wy-control-group .wy-form-full,.wy-control-group .wy-form-halves,.wy-control-group .wy-form-thirds{padding-bottom:12px}.wy-control-group .wy-form-full input[type=color],.wy-control-group .wy-form-full input[type=date],.wy-control-group .wy-form-full input[type=datetime-local],.wy-control-group .wy-form-full input[type=datetime],.wy-control-group .wy-form-full input[type=email],.wy-control-group .wy-form-full input[type=month],.wy-control-group .wy-form-full input[type=number],.wy-control-group .wy-form-full input[type=password],.wy-control-group .wy-form-full input[type=search],.wy-control-group .wy-form-full input[type=tel],.wy-control-group .wy-form-full input[type=text],.wy-control-group .wy-form-full input[type=time],.wy-control-group .wy-form-full input[type=url],.wy-control-group .wy-form-full input[type=week],.wy-control-group .wy-form-full select,.wy-control-group .wy-form-halves input[type=color],.wy-control-group .wy-form-halves input[type=date],.wy-control-group .wy-form-halves input[type=datetime-local],.wy-control-group .wy-form-halves input[type=datetime],.wy-control-group .wy-form-halves input[type=email],.wy-control-group .wy-form-halves input[type=month],.wy-control-group .wy-form-halves input[type=number],.wy-control-group .wy-form-halves input[type=password],.wy-control-group .wy-form-halves input[type=search],.wy-control-group .wy-form-halves input[type=tel],.wy-control-group .wy-form-halves input[type=text],.wy-control-group .wy-form-halves input[type=time],.wy-control-group .wy-form-halves input[type=url],.wy-control-group .wy-form-halves input[type=week],.wy-control-group .wy-form-halves select,.wy-control-group .wy-form-thirds input[type=color],.wy-control-group .wy-form-thirds input[type=date],.wy-control-group .wy-form-thirds input[type=datetime-local],.wy-control-group .wy-form-thirds input[type=datetime],.wy-control-group .wy-form-thirds input[type=email],.wy-control-group .wy-form-thirds input[type=month],.wy-control-group .wy-form-thirds input[type=number],.wy-control-group .wy-form-thirds input[type=password],.wy-control-group .wy-form-thirds input[type=search],.wy-control-group .wy-form-thirds input[type=tel],.wy-control-group .wy-form-thirds input[type=text],.wy-control-group .wy-form-thirds input[type=time],.wy-control-group .wy-form-thirds input[type=url],.wy-control-group .wy-form-thirds input[type=week],.wy-control-group .wy-form-thirds select{width:100%}.wy-control-group .wy-form-full{float:left;display:block;width:100%;margin-right:0}.wy-control-group .wy-form-full:last-child{margin-right:0}.wy-control-group .wy-form-halves{float:left;display:block;margin-right:2.35765%;width:48.82117%}.wy-control-group .wy-form-halves:last-child,.wy-control-group .wy-form-halves:nth-of-type(2n){margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(odd){clear:left}.wy-control-group .wy-form-thirds{float:left;display:block;margin-right:2.35765%;width:31.76157%}.wy-control-group .wy-form-thirds:last-child,.wy-control-group .wy-form-thirds:nth-of-type(3n){margin-right:0}.wy-control-group .wy-form-thirds:nth-of-type(3n+1){clear:left}.wy-control-group.wy-control-group-no-input .wy-control,.wy-control-no-input{margin:6px 0 0;font-size:90%}.wy-control-no-input{display:inline-block}.wy-control-group.fluid-input input[type=color],.wy-control-group.fluid-input input[type=date],.wy-control-group.fluid-input input[type=datetime-local],.wy-control-group.fluid-input input[type=datetime],.wy-control-group.fluid-input input[type=email],.wy-control-group.fluid-input input[type=month],.wy-control-group.fluid-input input[type=number],.wy-control-group.fluid-input input[type=password],.wy-control-group.fluid-input input[type=search],.wy-control-group.fluid-input input[type=tel],.wy-control-group.fluid-input input[type=text],.wy-control-group.fluid-input input[type=time],.wy-control-group.fluid-input input[type=url],.wy-control-group.fluid-input input[type=week]{width:100%}.wy-form-message-inline{padding-left:.3em;color:#666;font-size:90%}.wy-form-message{display:block;color:#999;font-size:70%;margin-top:.3125em;font-style:italic}.wy-form-message p{font-size:inherit;font-style:italic;margin-bottom:6px}.wy-form-message p:last-child{margin-bottom:0}input{line-height:normal}input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;*overflow:visible}input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week]{-webkit-appearance:none;padding:6px;display:inline-block;border:1px solid #ccc;font-size:80%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 3px #ddd;border-radius:0;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}input[type=datetime-local]{padding:.34375em .625em}input[disabled]{cursor:default}input[type=checkbox],input[type=radio]{padding:0;margin-right:.3125em;*height:13px;*width:13px}input[type=checkbox],input[type=radio],input[type=search]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}input[type=search]::-webkit-search-cancel-button,input[type=search]::-webkit-search-decoration{-webkit-appearance:none}input[type=color]:focus,input[type=date]:focus,input[type=datetime-local]:focus,input[type=datetime]:focus,input[type=email]:focus,input[type=month]:focus,input[type=number]:focus,input[type=password]:focus,input[type=search]:focus,input[type=tel]:focus,input[type=text]:focus,input[type=time]:focus,input[type=url]:focus,input[type=week]:focus{outline:0;outline:thin dotted\9;border-color:#333}input.no-focus:focus{border-color:#ccc!important}input[type=checkbox]:focus,input[type=file]:focus,input[type=radio]:focus{outline:thin dotted #333;outline:1px auto #129fea}input[type=color][disabled],input[type=date][disabled],input[type=datetime-local][disabled],input[type=datetime][disabled],input[type=email][disabled],input[type=month][disabled],input[type=number][disabled],input[type=password][disabled],input[type=search][disabled],input[type=tel][disabled],input[type=text][disabled],input[type=time][disabled],input[type=url][disabled],input[type=week][disabled]{cursor:not-allowed;background-color:#fafafa}input:focus:invalid,select:focus:invalid,textarea:focus:invalid{color:#e74c3c;border:1px solid #e74c3c}input:focus:invalid:focus,select:focus:invalid:focus,textarea:focus:invalid:focus{border-color:#e74c3c}input[type=checkbox]:focus:invalid:focus,input[type=file]:focus:invalid:focus,input[type=radio]:focus:invalid:focus{outline-color:#e74c3c}input.wy-input-large{padding:12px;font-size:100%}textarea{overflow:auto;vertical-align:top;width:100%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif}select,textarea{padding:.5em .625em;display:inline-block;border:1px solid #ccc;font-size:80%;box-shadow:inset 0 1px 3px #ddd;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}select{border:1px solid #ccc;background-color:#fff}select[multiple]{height:auto}select:focus,textarea:focus{outline:0}input[readonly],select[disabled],select[readonly],textarea[disabled],textarea[readonly]{cursor:not-allowed;background-color:#fafafa}input[type=checkbox][disabled],input[type=radio][disabled]{cursor:not-allowed}.wy-checkbox,.wy-radio{margin:6px 0;color:#404040;display:block}.wy-checkbox input,.wy-radio input{vertical-align:baseline}.wy-form-message-inline{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-input-prefix,.wy-input-suffix{white-space:nowrap;padding:6px}.wy-input-prefix .wy-input-context,.wy-input-suffix .wy-input-context{line-height:27px;padding:0 8px;display:inline-block;font-size:80%;background-color:#f3f6f6;border:1px solid #ccc;color:#999}.wy-input-suffix .wy-input-context{border-left:0}.wy-input-prefix .wy-input-context{border-right:0}.wy-switch{position:relative;display:block;height:24px;margin-top:12px;cursor:pointer}.wy-switch:before{left:0;top:0;width:36px;height:12px;background:#ccc}.wy-switch:after,.wy-switch:before{position:absolute;content:"";display:block;border-radius:4px;-webkit-transition:all .2s ease-in-out;-moz-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.wy-switch:after{width:18px;height:18px;background:#999;left:-3px;top:-3px}.wy-switch span{position:absolute;left:48px;display:block;font-size:12px;color:#ccc;line-height:1}.wy-switch.active:before{background:#1e8449}.wy-switch.active:after{left:24px;background:#27ae60}.wy-switch.disabled{cursor:not-allowed;opacity:.8}.wy-control-group.wy-control-group-error .wy-form-message,.wy-control-group.wy-control-group-error>label{color:#e74c3c}.wy-control-group.wy-control-group-error input[type=color],.wy-control-group.wy-control-group-error input[type=date],.wy-control-group.wy-control-group-error input[type=datetime-local],.wy-control-group.wy-control-group-error input[type=datetime],.wy-control-group.wy-control-group-error input[type=email],.wy-control-group.wy-control-group-error input[type=month],.wy-control-group.wy-control-group-error input[type=number],.wy-control-group.wy-control-group-error input[type=password],.wy-control-group.wy-control-group-error input[type=search],.wy-control-group.wy-control-group-error input[type=tel],.wy-control-group.wy-control-group-error input[type=text],.wy-control-group.wy-control-group-error input[type=time],.wy-control-group.wy-control-group-error input[type=url],.wy-control-group.wy-control-group-error input[type=week],.wy-control-group.wy-control-group-error textarea{border:1px solid #e74c3c}.wy-inline-validate{white-space:nowrap}.wy-inline-validate .wy-input-context{padding:.5em .625em;display:inline-block;font-size:80%}.wy-inline-validate.wy-inline-validate-success .wy-input-context{color:#27ae60}.wy-inline-validate.wy-inline-validate-danger .wy-input-context{color:#e74c3c}.wy-inline-validate.wy-inline-validate-warning .wy-input-context{color:#e67e22}.wy-inline-validate.wy-inline-validate-info .wy-input-context{color:#2980b9}.rotate-90{-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.rotate-180{-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.rotate-270{-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.mirror{-webkit-transform:scaleX(-1);-moz-transform:scaleX(-1);-ms-transform:scaleX(-1);-o-transform:scaleX(-1);transform:scaleX(-1)}.mirror.rotate-90{-webkit-transform:scaleX(-1) rotate(90deg);-moz-transform:scaleX(-1) rotate(90deg);-ms-transform:scaleX(-1) rotate(90deg);-o-transform:scaleX(-1) rotate(90deg);transform:scaleX(-1) rotate(90deg)}.mirror.rotate-180{-webkit-transform:scaleX(-1) rotate(180deg);-moz-transform:scaleX(-1) rotate(180deg);-ms-transform:scaleX(-1) rotate(180deg);-o-transform:scaleX(-1) rotate(180deg);transform:scaleX(-1) rotate(180deg)}.mirror.rotate-270{-webkit-transform:scaleX(-1) rotate(270deg);-moz-transform:scaleX(-1) rotate(270deg);-ms-transform:scaleX(-1) rotate(270deg);-o-transform:scaleX(-1) rotate(270deg);transform:scaleX(-1) rotate(270deg)}@media only screen and (max-width:480px){.wy-form button[type=submit]{margin:.7em 0 0}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=text],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week],.wy-form label{margin-bottom:.3em;display:block}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week]{margin-bottom:0}.wy-form-aligned .wy-control-group label{margin-bottom:.3em;text-align:left;display:block;width:100%}.wy-form-aligned .wy-control{margin:1.5em 0 0}.wy-form-message,.wy-form-message-inline,.wy-form .wy-help-inline{display:block;font-size:80%;padding:6px 0}}@media screen and (max-width:768px){.tablet-hide{display:none}}@media screen and (max-width:480px){.mobile-hide{display:none}}.float-left{float:left}.float-right{float:right}.full-width{width:100%}.rst-content table.docutils,.rst-content table.field-list,.wy-table{border-collapse:collapse;border-spacing:0;empty-cells:show;margin-bottom:24px}.rst-content table.docutils caption,.rst-content table.field-list caption,.wy-table caption{color:#000;font:italic 85%/1 arial,sans-serif;padding:1em 0;text-align:center}.rst-content table.docutils td,.rst-content table.docutils th,.rst-content table.field-list td,.rst-content table.field-list th,.wy-table td,.wy-table th{font-size:90%;margin:0;overflow:visible;padding:8px 16px}.rst-content table.docutils td:first-child,.rst-content table.docutils th:first-child,.rst-content table.field-list td:first-child,.rst-content table.field-list th:first-child,.wy-table td:first-child,.wy-table th:first-child{border-left-width:0}.rst-content table.docutils thead,.rst-content table.field-list thead,.wy-table thead{color:#000;text-align:left;vertical-align:bottom;white-space:nowrap}.rst-content table.docutils thead th,.rst-content table.field-list thead th,.wy-table thead th{font-weight:700;border-bottom:2px solid #e1e4e5}.rst-content table.docutils td,.rst-content table.field-list td,.wy-table td{background-color:transparent;vertical-align:middle}.rst-content table.docutils td p,.rst-content table.field-list td p,.wy-table td p{line-height:18px}.rst-content table.docutils td p:last-child,.rst-content table.field-list td p:last-child,.wy-table td p:last-child{margin-bottom:0}.rst-content table.docutils .wy-table-cell-min,.rst-content table.field-list .wy-table-cell-min,.wy-table .wy-table-cell-min{width:1%;padding-right:0}.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-content table.field-list .wy-table-cell-min input[type=checkbox],.wy-table .wy-table-cell-min input[type=checkbox]{margin:0}.wy-table-secondary{color:grey;font-size:90%}.wy-table-tertiary{color:grey;font-size:80%}.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td,.wy-table-backed,.wy-table-odd td,.wy-table-striped tr:nth-child(2n-1) td{background-color:#f3f6f6}.rst-content table.docutils,.wy-table-bordered-all{border:1px solid #e1e4e5}.rst-content table.docutils td,.wy-table-bordered-all td{border-bottom:1px solid #e1e4e5;border-left:1px solid #e1e4e5}.rst-content table.docutils tbody>tr:last-child td,.wy-table-bordered-all tbody>tr:last-child td{border-bottom-width:0}.wy-table-bordered{border:1px solid #e1e4e5}.wy-table-bordered-rows td{border-bottom:1px solid #e1e4e5}.wy-table-bordered-rows tbody>tr:last-child td{border-bottom-width:0}.wy-table-horizontal td,.wy-table-horizontal th{border-width:0 0 1px;border-bottom:1px solid #e1e4e5}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-responsive{margin-bottom:24px;max-width:100%;overflow:auto}.wy-table-responsive table{margin-bottom:0!important}.wy-table-responsive table td,.wy-table-responsive table th{white-space:nowrap}a{color:#2980b9;text-decoration:none;cursor:pointer}a:hover{color:#3091d1}a:visited{color:#9b59b6}html{height:100%}body,html{overflow-x:hidden}body{font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;font-weight:400;color:#404040;min-height:100%;background:#edf0f2}.wy-text-left{text-align:left}.wy-text-center{text-align:center}.wy-text-right{text-align:right}.wy-text-large{font-size:120%}.wy-text-normal{font-size:100%}.wy-text-small,small{font-size:80%}.wy-text-strike{text-decoration:line-through}.wy-text-warning{color:#e67e22!important}a.wy-text-warning:hover{color:#eb9950!important}.wy-text-info{color:#2980b9!important}a.wy-text-info:hover{color:#409ad5!important}.wy-text-success{color:#27ae60!important}a.wy-text-success:hover{color:#36d278!important}.wy-text-danger{color:#e74c3c!important}a.wy-text-danger:hover{color:#ed7669!important}.wy-text-neutral{color:#404040!important}a.wy-text-neutral:hover{color:#595959!important}.rst-content .toctree-wrapper>p.caption,h1,h2,h3,h4,h5,h6,legend{margin-top:0;font-weight:700;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif}p{line-height:24px;font-size:16px;margin:0 0 24px}h1{font-size:175%}.rst-content .toctree-wrapper>p.caption,h2{font-size:150%}h3{font-size:125%}h4{font-size:115%}h5{font-size:110%}h6{font-size:100%}hr{display:block;height:1px;border:0;border-top:1px solid #e1e4e5;margin:24px 0;padding:0}.rst-content code,.rst-content tt,code{white-space:nowrap;max-width:100%;background:#fff;border:1px solid #e1e4e5;font-size:75%;padding:0 5px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#e74c3c;overflow-x:auto}.rst-content tt.code-large,code.code-large{font-size:90%}.rst-content .section ul,.rst-content .toctree-wrapper ul,.wy-plain-list-disc,article ul{list-style:disc;line-height:24px;margin-bottom:24px}.rst-content .section ul li,.rst-content .toctree-wrapper ul li,.wy-plain-list-disc li,article ul li{list-style:disc;margin-left:24px}.rst-content .section ul li p:last-child,.rst-content .section ul li ul,.rst-content .toctree-wrapper ul li p:last-child,.rst-content .toctree-wrapper ul li ul,.wy-plain-list-disc li p:last-child,.wy-plain-list-disc li ul,article ul li p:last-child,article ul li ul{margin-bottom:0}.rst-content .section ul li li,.rst-content .toctree-wrapper ul li li,.wy-plain-list-disc li li,article ul li li{list-style:circle}.rst-content .section ul li li li,.rst-content .toctree-wrapper ul li li li,.wy-plain-list-disc li li li,article ul li li li{list-style:square}.rst-content .section ul li ol li,.rst-content .toctree-wrapper ul li ol li,.wy-plain-list-disc li ol li,article ul li ol li{list-style:decimal}.rst-content .section ol,.rst-content ol.arabic,.wy-plain-list-decimal,article ol{list-style:decimal;line-height:24px;margin-bottom:24px}.rst-content .section ol li,.rst-content ol.arabic li,.wy-plain-list-decimal li,article ol li{list-style:decimal;margin-left:24px}.rst-content .section ol li p:last-child,.rst-content .section ol li ul,.rst-content ol.arabic li p:last-child,.rst-content ol.arabic li ul,.wy-plain-list-decimal li p:last-child,.wy-plain-list-decimal li ul,article ol li p:last-child,article ol li ul{margin-bottom:0}.rst-content .section ol li ul li,.rst-content ol.arabic li ul li,.wy-plain-list-decimal li ul li,article ol li ul li{list-style:disc}.wy-breadcrumbs{*zoom:1}.wy-breadcrumbs:after,.wy-breadcrumbs:before{display:table;content:""}.wy-breadcrumbs:after{clear:both}.wy-breadcrumbs li{display:inline-block}.wy-breadcrumbs li.wy-breadcrumbs-aside{float:right}.wy-breadcrumbs li a{display:inline-block;padding:5px}.wy-breadcrumbs li a:first-child{padding-left:0}.rst-content .wy-breadcrumbs li tt,.wy-breadcrumbs li .rst-content tt,.wy-breadcrumbs li code{padding:5px;border:none;background:none}.rst-content .wy-breadcrumbs li tt.literal,.wy-breadcrumbs li .rst-content tt.literal,.wy-breadcrumbs li code.literal{color:#404040}.wy-breadcrumbs-extra{margin-bottom:0;color:#b3b3b3;font-size:80%;display:inline-block}@media screen and (max-width:480px){.wy-breadcrumbs-extra,.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}@media print{.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}html{font-size:16px}.wy-affix{position:fixed;top:1.618em}.wy-menu a:hover{text-decoration:none}.wy-menu-horiz{*zoom:1}.wy-menu-horiz:after,.wy-menu-horiz:before{display:table;content:""}.wy-menu-horiz:after{clear:both}.wy-menu-horiz li,.wy-menu-horiz ul{display:inline-block}.wy-menu-horiz li:hover{background:hsla(0,0%,100%,.1)}.wy-menu-horiz li.divide-left{border-left:1px solid #404040}.wy-menu-horiz li.divide-right{border-right:1px solid #404040}.wy-menu-horiz a{height:32px;display:inline-block;line-height:32px;padding:0 16px}.wy-menu-vertical{width:300px}.wy-menu-vertical header,.wy-menu-vertical p.caption{color:#55a5d9;height:32px;line-height:32px;padding:0 1.618em;margin:12px 0 0;display:block;font-weight:700;text-transform:uppercase;font-size:85%;white-space:nowrap}.wy-menu-vertical ul{margin-bottom:0}.wy-menu-vertical li.divide-top{border-top:1px solid #404040}.wy-menu-vertical li.divide-bottom{border-bottom:1px solid #404040}.wy-menu-vertical li.current{background:#e3e3e3}.wy-menu-vertical li.current a{color:grey;border-right:1px solid #c9c9c9;padding:.4045em 2.427em}.wy-menu-vertical li.current a:hover{background:#d6d6d6}.rst-content .wy-menu-vertical li tt,.wy-menu-vertical li .rst-content tt,.wy-menu-vertical li code{border:none;background:inherit;color:inherit;padding-left:0;padding-right:0}.wy-menu-vertical li span.toctree-expand{display:block;float:left;margin-left:-1.2em;font-size:.8em;line-height:1.6em;color:#4d4d4d}.wy-menu-vertical li.current>a,.wy-menu-vertical li.on a{color:#404040;font-weight:700;position:relative;background:#fcfcfc;border:none;padding:.4045em 1.618em}.wy-menu-vertical li.current>a:hover,.wy-menu-vertical li.on a:hover{background:#fcfcfc}.wy-menu-vertical li.current>a:hover span.toctree-expand,.wy-menu-vertical li.on a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand{display:block;font-size:.8em;line-height:1.6em;color:#333}.wy-menu-vertical li.toctree-l1.current>a{border-bottom:1px solid #c9c9c9;border-top:1px solid #c9c9c9}.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .toctree-l11>ul{display:none}.wy-menu-vertical .toctree-l1.current .current.toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .current.toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .current.toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .current.toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .current.toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .current.toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .current.toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .current.toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .current.toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .current.toctree-l11>ul{display:block}.wy-menu-vertical li.toctree-l3,.wy-menu-vertical li.toctree-l4{font-size:.9em}.wy-menu-vertical li.toctree-l2 a,.wy-menu-vertical li.toctree-l3 a,.wy-menu-vertical li.toctree-l4 a,.wy-menu-vertical li.toctree-l5 a,.wy-menu-vertical li.toctree-l6 a,.wy-menu-vertical li.toctree-l7 a,.wy-menu-vertical li.toctree-l8 a,.wy-menu-vertical li.toctree-l9 a,.wy-menu-vertical li.toctree-l10 a{color:#404040}.wy-menu-vertical li.toctree-l2 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l3 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l4 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l5 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l6 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l7 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l8 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l9 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l10 a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{display:block}.wy-menu-vertical li.toctree-l2.current>a{padding:.4045em 2.427em}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current>a{padding:.4045em 4.045em}.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current>a{padding:.4045em 5.663em}.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current>a{padding:.4045em 7.281em}.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current>a{padding:.4045em 8.899em}.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current>a{padding:.4045em 10.517em}.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current>a{padding:.4045em 12.135em}.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current>a{padding:.4045em 13.753em}.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current>a{padding:.4045em 15.371em}.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{padding:.4045em 16.989em}.wy-menu-vertical li.toctree-l2.current>a,.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{background:#c9c9c9}.wy-menu-vertical li.toctree-l2 span.toctree-expand{color:#a3a3a3}.wy-menu-vertical li.toctree-l3.current>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{background:#bdbdbd}.wy-menu-vertical li.toctree-l3 span.toctree-expand{color:#969696}.wy-menu-vertical li.current ul{display:block}.wy-menu-vertical li ul{margin-bottom:0;display:none}.wy-menu-vertical li ul li a{margin-bottom:0;color:#d9d9d9;font-weight:400}.wy-menu-vertical a{line-height:18px;padding:.4045em 1.618em;display:block;position:relative;font-size:90%;color:#d9d9d9}.wy-menu-vertical a:hover{background-color:#4e4a4a;cursor:pointer}.wy-menu-vertical a:hover span.toctree-expand{color:#d9d9d9}.wy-menu-vertical a:active{background-color:#2980b9;cursor:pointer;color:#fff}.wy-menu-vertical a:active span.toctree-expand{color:#fff}.wy-side-nav-search{display:block;width:300px;padding:.809em;margin-bottom:.809em;z-index:200;background-color:#2980b9;text-align:center;color:#fcfcfc}.wy-side-nav-search input[type=text]{width:100%;border-radius:50px;padding:6px 12px;border-color:#2472a4}.wy-side-nav-search img{display:block;margin:auto auto .809em;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a{color:#fcfcfc;font-size:100%;font-weight:700;display:inline-block;padding:4px 6px;margin-bottom:.809em}.wy-side-nav-search .wy-dropdown>a:hover,.wy-side-nav-search>a:hover{background:hsla(0,0%,100%,.1)}.wy-side-nav-search .wy-dropdown>a img.logo,.wy-side-nav-search>a img.logo{display:block;margin:0 auto;height:auto;width:auto;border-radius:0;max-width:100%;background:transparent}.wy-side-nav-search .wy-dropdown>a.icon img.logo,.wy-side-nav-search>a.icon img.logo{margin-top:.85em}.wy-side-nav-search>div.version{margin-top:-.4045em;margin-bottom:.809em;font-weight:400;color:hsla(0,0%,100%,.3)}.wy-nav .wy-menu-vertical header{color:#2980b9}.wy-nav .wy-menu-vertical a{color:#b3b3b3}.wy-nav .wy-menu-vertical a:hover{background-color:#2980b9;color:#fff}[data-menu-wrap]{-webkit-transition:all .2s ease-in;-moz-transition:all .2s ease-in;transition:all .2s ease-in;position:absolute;opacity:1;width:100%;opacity:0}[data-menu-wrap].move-center{left:0;right:auto;opacity:1}[data-menu-wrap].move-left{right:auto;left:-100%;opacity:0}[data-menu-wrap].move-right{right:-100%;left:auto;opacity:0}.wy-body-for-nav{background:#fcfcfc}.wy-grid-for-nav{position:absolute;width:100%;height:100%}.wy-nav-side{position:fixed;top:0;bottom:0;left:0;padding-bottom:2em;width:300px;overflow-x:hidden;overflow-y:hidden;min-height:100%;color:#9b9b9b;background:#343131;z-index:200}.wy-side-scroll{width:320px;position:relative;overflow-x:hidden;overflow-y:scroll;height:100%}.wy-nav-top{display:none;background:#2980b9;color:#fff;padding:.4045em .809em;position:relative;line-height:50px;text-align:center;font-size:100%;*zoom:1}.wy-nav-top:after,.wy-nav-top:before{display:table;content:""}.wy-nav-top:after{clear:both}.wy-nav-top a{color:#fff;font-weight:700}.wy-nav-top img{margin-right:12px;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-nav-top i{font-size:30px;float:left;cursor:pointer;padding-top:inherit}.wy-nav-content-wrap{margin-left:300px;background:#fcfcfc;min-height:100%}.wy-nav-content{padding:1.618em 3.236em;height:100%;max-width:800px;margin:auto}.wy-body-mask{position:fixed;width:100%;height:100%;background:rgba(0,0,0,.2);display:none;z-index:499}.wy-body-mask.on{display:block}footer{color:grey}footer p{margin-bottom:12px}.rst-content footer span.commit tt,footer span.commit .rst-content tt,footer span.commit code{padding:0;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:1em;background:none;border:none;color:grey}.rst-footer-buttons{*zoom:1}.rst-footer-buttons:after,.rst-footer-buttons:before{width:100%;display:table;content:""}.rst-footer-buttons:after{clear:both}.rst-breadcrumbs-buttons{margin-top:12px;*zoom:1}.rst-breadcrumbs-buttons:after,.rst-breadcrumbs-buttons:before{display:table;content:""}.rst-breadcrumbs-buttons:after{clear:both}#search-results .search li{margin-bottom:24px;border-bottom:1px solid #e1e4e5;padding-bottom:24px}#search-results .search li:first-child{border-top:1px solid #e1e4e5;padding-top:24px}#search-results .search li a{font-size:120%;margin-bottom:12px;display:inline-block}#search-results .context{color:grey;font-size:90%}.genindextable li>ul{margin-left:24px}@media screen and (max-width:768px){.wy-body-for-nav{background:#fcfcfc}.wy-nav-top{display:block}.wy-nav-side{left:-300px}.wy-nav-side.shift{width:85%;left:0}.wy-menu.wy-menu-vertical,.wy-side-nav-search,.wy-side-scroll{width:auto}.wy-nav-content-wrap{margin-left:0}.wy-nav-content-wrap .wy-nav-content{padding:1.618em}.wy-nav-content-wrap.shift{position:fixed;min-width:100%;left:85%;top:0;height:100%;overflow:hidden}}@media screen and (min-width:1100px){.wy-nav-content-wrap{background:rgba(0,0,0,.05)}.wy-nav-content{margin:0;background:#fcfcfc}}@media print{.rst-versions,.wy-nav-side,footer{display:none}.wy-nav-content-wrap{margin-left:0}}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60;*zoom:1}.rst-versions .rst-current-version:after,.rst-versions .rst-current-version:before{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-content .code-block-caption .rst-versions .rst-current-version .headerlink,.rst-content .rst-versions .rst-current-version .admonition-title,.rst-content code.download .rst-versions .rst-current-version span:first-child,.rst-content dl dt .rst-versions .rst-current-version .headerlink,.rst-content h1 .rst-versions .rst-current-version .headerlink,.rst-content h2 .rst-versions .rst-current-version .headerlink,.rst-content h3 .rst-versions .rst-current-version .headerlink,.rst-content h4 .rst-versions .rst-current-version .headerlink,.rst-content h5 .rst-versions .rst-current-version .headerlink,.rst-content h6 .rst-versions .rst-current-version .headerlink,.rst-content p.caption .rst-versions .rst-current-version .headerlink,.rst-content table>caption .rst-versions .rst-current-version .headerlink,.rst-content tt.download .rst-versions .rst-current-version span:first-child,.rst-versions .rst-current-version .fa,.rst-versions .rst-current-version .icon,.rst-versions .rst-current-version .rst-content .admonition-title,.rst-versions .rst-current-version .rst-content .code-block-caption .headerlink,.rst-versions .rst-current-version .rst-content code.download span:first-child,.rst-versions .rst-current-version .rst-content dl dt .headerlink,.rst-versions .rst-current-version .rst-content h1 .headerlink,.rst-versions .rst-current-version .rst-content h2 .headerlink,.rst-versions .rst-current-version .rst-content h3 .headerlink,.rst-versions .rst-current-version .rst-content h4 .headerlink,.rst-versions .rst-current-version .rst-content h5 .headerlink,.rst-versions .rst-current-version .rst-content h6 .headerlink,.rst-versions .rst-current-version .rst-content p.caption .headerlink,.rst-versions .rst-current-version .rst-content table>caption .headerlink,.rst-versions .rst-current-version .rst-content tt.download span:first-child,.rst-versions .rst-current-version .wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li .rst-versions .rst-current-version span.toctree-expand{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}}.rst-content img{max-width:100%;height:auto}.rst-content div.figure{margin-bottom:24px}.rst-content div.figure p.caption{font-style:italic}.rst-content div.figure p:last-child.caption{margin-bottom:0}.rst-content div.figure.align-center{text-align:center}.rst-content .section>a>img,.rst-content .section>img{margin-bottom:24px}.rst-content abbr[title]{text-decoration:none}.rst-content.style-external-links a.reference.external:after{font-family:FontAwesome;content:"\f08e";color:#b3b3b3;vertical-align:super;font-size:60%;margin:0 .2em}.rst-content blockquote{margin-left:24px;line-height:24px;margin-bottom:24px}.rst-content pre.literal-block{white-space:pre;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;display:block;overflow:auto}.rst-content div[class^=highlight],.rst-content pre.literal-block{border:1px solid #e1e4e5;overflow-x:auto;margin:1px 0 24px}.rst-content div[class^=highlight] div[class^=highlight],.rst-content pre.literal-block div[class^=highlight]{padding:0;border:none;margin:0}.rst-content div[class^=highlight] td.code{width:100%}.rst-content .linenodiv pre{border-right:1px solid #e6e9ea;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;user-select:none;pointer-events:none}.rst-content div[class^=highlight] pre{white-space:pre;margin:0;padding:12px;display:block;overflow:auto}.rst-content div[class^=highlight] pre .hll{display:block;margin:0 -12px;padding:0 12px}.rst-content .linenodiv pre,.rst-content div[class^=highlight] pre,.rst-content pre.literal-block{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:12px;line-height:1.4}.rst-content div.highlight .gp{user-select:none;pointer-events:none}.rst-content .code-block-caption{font-style:italic;font-size:85%;line-height:1;padding:1em 0;text-align:center}@media print{.rst-content .codeblock,.rst-content div[class^=highlight],.rst-content div[class^=highlight] pre{white-space:pre-wrap}}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning{clear:both}.rst-content .admonition-todo .last,.rst-content .admonition-todo>:last-child,.rst-content .admonition .last,.rst-content .admonition>:last-child,.rst-content .attention .last,.rst-content .attention>:last-child,.rst-content .caution .last,.rst-content .caution>:last-child,.rst-content .danger .last,.rst-content .danger>:last-child,.rst-content .error .last,.rst-content .error>:last-child,.rst-content .hint .last,.rst-content .hint>:last-child,.rst-content .important .last,.rst-content .important>:last-child,.rst-content .note .last,.rst-content .note>:last-child,.rst-content .seealso .last,.rst-content .seealso>:last-child,.rst-content .tip .last,.rst-content .tip>:last-child,.rst-content .warning .last,.rst-content .warning>:last-child{margin-bottom:0}.rst-content .admonition-title:before{margin-right:4px}.rst-content .admonition table{border-color:rgba(0,0,0,.1)}.rst-content .admonition table td,.rst-content .admonition table th{background:transparent!important;border-color:rgba(0,0,0,.1)!important}.rst-content .section ol.loweralpha,.rst-content .section ol.loweralpha>li{list-style:lower-alpha}.rst-content .section ol.upperalpha,.rst-content .section ol.upperalpha>li{list-style:upper-alpha}.rst-content .section ol li>*,.rst-content .section ul li>*{margin-top:12px;margin-bottom:12px}.rst-content .section ol li>:first-child,.rst-content .section ul li>:first-child{margin-top:0}.rst-content .section ol li>p,.rst-content .section ol li>p:last-child,.rst-content .section ul li>p,.rst-content .section ul li>p:last-child{margin-bottom:12px}.rst-content .section ol li>p:only-child,.rst-content .section ol li>p:only-child:last-child,.rst-content .section ul li>p:only-child,.rst-content .section ul li>p:only-child:last-child{margin-bottom:0}.rst-content .section ol li>ol,.rst-content .section ol li>ul,.rst-content .section ul li>ol,.rst-content .section ul li>ul{margin-bottom:12px}.rst-content .section ol.simple li>*,.rst-content .section ol.simple li ol,.rst-content .section ol.simple li ul,.rst-content .section ul.simple li>*,.rst-content .section ul.simple li ol,.rst-content .section ul.simple li ul{margin-top:0;margin-bottom:0}.rst-content .line-block{margin-left:0;margin-bottom:24px;line-height:24px}.rst-content .line-block .line-block{margin-left:24px;margin-bottom:0}.rst-content .topic-title{font-weight:700;margin-bottom:12px}.rst-content .toc-backref{color:#404040}.rst-content .align-right{float:right;margin:0 0 24px 24px}.rst-content .align-left{float:left;margin:0 24px 24px 0}.rst-content .align-center{margin:auto}.rst-content .align-center:not(table){display:block}.rst-content .code-block-caption .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink{visibility:hidden;font-size:14px}.rst-content .code-block-caption .headerlink:after,.rst-content .toctree-wrapper>p.caption .headerlink:after,.rst-content dl dt .headerlink:after,.rst-content h1 .headerlink:after,.rst-content h2 .headerlink:after,.rst-content h3 .headerlink:after,.rst-content h4 .headerlink:after,.rst-content h5 .headerlink:after,.rst-content h6 .headerlink:after,.rst-content p.caption .headerlink:after,.rst-content table>caption .headerlink:after{content:"\f0c1";font-family:FontAwesome}.rst-content .code-block-caption:hover .headerlink:after,.rst-content .toctree-wrapper>p.caption:hover .headerlink:after,.rst-content dl dt:hover .headerlink:after,.rst-content h1:hover .headerlink:after,.rst-content h2:hover .headerlink:after,.rst-content h3:hover .headerlink:after,.rst-content h4:hover .headerlink:after,.rst-content h5:hover .headerlink:after,.rst-content h6:hover .headerlink:after,.rst-content p.caption:hover .headerlink:after,.rst-content table>caption:hover .headerlink:after{visibility:visible}.rst-content table>caption .headerlink:after{font-size:12px}.rst-content .centered{text-align:center}.rst-content .sidebar{float:right;width:40%;display:block;margin:0 0 24px 24px;padding:24px;background:#f3f6f6;border:1px solid #e1e4e5}.rst-content .sidebar dl,.rst-content .sidebar p,.rst-content .sidebar ul{font-size:90%}.rst-content .sidebar .last,.rst-content .sidebar>:last-child{margin-bottom:0}.rst-content .sidebar .sidebar-title{display:block;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif;font-weight:700;background:#e1e4e5;padding:6px 12px;margin:-24px -24px 24px;font-size:100%}.rst-content .highlighted{background:#f1c40f;box-shadow:0 0 0 2px #f1c40f;display:inline;font-weight:700}.rst-content .citation-reference,.rst-content .footnote-reference{vertical-align:baseline;position:relative;top:-.4em;line-height:0;font-size:90%}.rst-content .hlist{width:100%}html.writer-html4 .rst-content table.docutils.citation,html.writer-html4 .rst-content table.docutils.footnote{background:none;border:none}html.writer-html4 .rst-content table.docutils.citation td,html.writer-html4 .rst-content table.docutils.citation tr,html.writer-html4 .rst-content table.docutils.footnote td,html.writer-html4 .rst-content table.docutils.footnote tr{border:none;background-color:transparent!important;white-space:normal}html.writer-html4 .rst-content table.docutils.citation td.label,html.writer-html4 .rst-content table.docutils.footnote td.label{padding-left:0;padding-right:0;vertical-align:top}html.writer-html5 .rst-content dl dt span.classifier:before{content:" : "}html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{display:grid;grid-template-columns:max-content auto}html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{padding-left:1rem}html.writer-html5 .rst-content dl.field-list>dt:after,html.writer-html5 .rst-content dl.footnote>dt:after{content:":"}html.writer-html5 .rst-content dl.field-list>dd,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dd,html.writer-html5 .rst-content dl.footnote>dt{margin-bottom:0}html.writer-html5 .rst-content dl.footnote{font-size:.9rem}html.writer-html5 .rst-content dl.footnote>dt{margin:0 .5rem .5rem 0;line-height:1.2rem;word-break:break-all;font-weight:400}html.writer-html5 .rst-content dl.footnote>dt>span.brackets{margin-right:.5rem}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:before{content:"["}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:after{content:"]"}html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref{font-style:italic}html.writer-html5 .rst-content dl.footnote>dd{margin:0 0 .5rem;line-height:1.2rem}html.writer-html5 .rst-content dl.footnote>dd p,html.writer-html5 .rst-content dl.option-list kbd{font-size:.9rem}.rst-content table.docutils.footnote,html.writer-html4 .rst-content table.docutils.citation,html.writer-html5 .rst-content dl.footnote{color:grey}.rst-content table.docutils.footnote code,.rst-content table.docutils.footnote tt,html.writer-html4 .rst-content table.docutils.citation code,html.writer-html4 .rst-content table.docutils.citation tt,html.writer-html5 .rst-content dl.footnote code,html.writer-html5 .rst-content dl.footnote tt{color:#555}.rst-content .wy-table-responsive.citation,.rst-content .wy-table-responsive.footnote{margin-bottom:0}.rst-content .wy-table-responsive.citation+:not(.citation),.rst-content .wy-table-responsive.footnote+:not(.footnote){margin-top:24px}.rst-content .wy-table-responsive.citation:last-child,.rst-content .wy-table-responsive.footnote:last-child{margin-bottom:24px}.rst-content table.docutils th{border-color:#e1e4e5}html.writer-html5 .rst-content table.docutils th{border:1px solid #e1e4e5}html.writer-html5 .rst-content table.docutils td>p,html.writer-html5 .rst-content table.docutils th>p{line-height:1rem;margin-bottom:0;font-size:.9rem}.rst-content table.docutils td .last,.rst-content table.docutils td .last>:last-child{margin-bottom:0}.rst-content table.field-list,.rst-content table.field-list td{border:none}.rst-content table.field-list td p{font-size:inherit;line-height:inherit}.rst-content table.field-list td>strong{display:inline-block}.rst-content table.field-list .field-name{padding-right:10px;text-align:left;white-space:nowrap}.rst-content table.field-list .field-body{text-align:left}.rst-content code,.rst-content tt{color:#000;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;padding:2px 5px}.rst-content code big,.rst-content code em,.rst-content tt big,.rst-content tt em{font-size:100%!important;line-height:normal}.rst-content code.literal,.rst-content tt.literal{color:#e74c3c}.rst-content code.xref,.rst-content tt.xref,a .rst-content code,a .rst-content tt{font-weight:700;color:#404040}.rst-content kbd,.rst-content pre,.rst-content samp{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace}.rst-content a code,.rst-content a tt{color:#2980b9}.rst-content dl{margin-bottom:24px}.rst-content dl dt{font-weight:700;margin-bottom:12px}.rst-content dl ol,.rst-content dl p,.rst-content dl table,.rst-content dl ul{margin-bottom:12px}.rst-content dl dd{margin:0 0 12px 24px;line-height:24px}html.writer-html4 .rst-content dl:not(.docutils),html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple){margin-bottom:24px}html.writer-html4 .rst-content dl:not(.docutils)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt{display:table;margin:6px 0;font-size:90%;line-height:normal;background:#e7f2fa;color:#2980b9;border-top:3px solid #6ab0de;padding:6px;position:relative}html.writer-html4 .rst-content dl:not(.docutils)>dt:before,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:before{color:#6ab0de}html.writer-html4 .rst-content dl:not(.docutils)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt{margin-bottom:6px;border:none;border-left:3px solid #ccc;background:#f0f0f0;color:#555}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils)>dt:first-child,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:first-child{margin-top:0}html.writer-html4 .rst-content dl:not(.docutils) code,html.writer-html4 .rst-content dl:not(.docutils) tt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) code.descclassname,html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descclassname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{background-color:transparent;border:none;padding:0;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .optional,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .optional{display:inline-block;padding:0 4px;color:#000;font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .property,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .property{display:inline-block;padding-right:8px}.rst-content .viewcode-back,.rst-content .viewcode-link{display:inline-block;color:#27ae60;font-size:80%;padding-left:24px}.rst-content .viewcode-back{display:block;float:right}.rst-content p.rubric{margin-bottom:12px;font-weight:700}.rst-content code.download,.rst-content tt.download{background:inherit;padding:inherit;font-weight:400;font-family:inherit;font-size:inherit;color:inherit;border:inherit;white-space:inherit}.rst-content code.download span:first-child,.rst-content tt.download span:first-child{-webkit-font-smoothing:subpixel-antialiased}.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{margin-right:4px}.rst-content .guilabel{border:1px solid #7fbbe3;background:#e7f2fa;font-size:80%;font-weight:700;border-radius:4px;padding:2.4px 6px;margin:auto 2px}.rst-content .versionmodified{font-style:italic}@media screen and (max-width:480px){.rst-content .sidebar{width:100%}}span[id*=MathJax-Span]{color:#404040}.math{text-align:center}@font-face{font-family:Lato;src:url(fonts/lato-normal.woff2?bd03a2cc277bbbc338d464e679fe9942) format("woff2"),url(fonts/lato-normal.woff?27bd77b9162d388cb8d4c4217c7c5e2a) format("woff");font-weight:400;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold.woff2?cccb897485813c7c256901dbca54ecf2) format("woff2"),url(fonts/lato-bold.woff?d878b6c29b10beca227e9eef4246111b) format("woff");font-weight:700;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold-italic.woff2?0b6bb6725576b072c5d0b02ecdd1900d) format("woff2"),url(fonts/lato-bold-italic.woff?9c7e4e9eb485b4a121c760e61bc3707c) format("woff");font-weight:700;font-style:italic;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-normal-italic.woff2?4eb103b4d12be57cb1d040ed5e162e9d) format("woff2"),url(fonts/lato-normal-italic.woff?f28f2d6482446544ef1ea1ccc6dd5892) format("woff");font-weight:400;font-style:italic;font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:400;src:url(fonts/Roboto-Slab-Regular.woff2?7abf5b8d04d26a2cafea937019bca958) format("woff2"),url(fonts/Roboto-Slab-Regular.woff?c1be9284088d487c5e3ff0a10a92e58c) format("woff");font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:700;src:url(fonts/Roboto-Slab-Bold.woff2?9984f4a9bda09be08e83f2506954adbe) format("woff2"),url(fonts/Roboto-Slab-Bold.woff?bed5564a116b05148e3b3bea6fb1162a) format("woff");font-display:block} \ No newline at end of file diff --git a/en/2.9/_static/doctools.js b/en/2.9/_static/doctools.js new file mode 100644 index 0000000000..527b876ca6 --- /dev/null +++ b/en/2.9/_static/doctools.js @@ -0,0 +1,156 @@ +/* + * doctools.js + * ~~~~~~~~~~~ + * + * Base JavaScript utilities for all Sphinx HTML documentation. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ +"use strict"; + +const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([ + "TEXTAREA", + "INPUT", + "SELECT", + "BUTTON", +]); + +const _ready = (callback) => { + if (document.readyState !== "loading") { + callback(); + } else { + document.addEventListener("DOMContentLoaded", callback); + } +}; + +/** + * Small JavaScript module for the documentation. + */ +const Documentation = { + init: () => { + Documentation.initDomainIndexTable(); + Documentation.initOnKeyListeners(); + }, + + /** + * i18n support + */ + TRANSLATIONS: {}, + PLURAL_EXPR: (n) => (n === 1 ? 0 : 1), + LOCALE: "unknown", + + // gettext and ngettext don't access this so that the functions + // can safely bound to a different name (_ = Documentation.gettext) + gettext: (string) => { + const translated = Documentation.TRANSLATIONS[string]; + switch (typeof translated) { + case "undefined": + return string; // no translation + case "string": + return translated; // translation exists + default: + return translated[0]; // (singular, plural) translation tuple exists + } + }, + + ngettext: (singular, plural, n) => { + const translated = Documentation.TRANSLATIONS[singular]; + if (typeof translated !== "undefined") + return translated[Documentation.PLURAL_EXPR(n)]; + return n === 1 ? singular : plural; + }, + + addTranslations: (catalog) => { + Object.assign(Documentation.TRANSLATIONS, catalog.messages); + Documentation.PLURAL_EXPR = new Function( + "n", + `return (${catalog.plural_expr})` + ); + Documentation.LOCALE = catalog.locale; + }, + + /** + * helper function to focus on search bar + */ + focusSearchBar: () => { + document.querySelectorAll("input[name=q]")[0]?.focus(); + }, + + /** + * Initialise the domain index toggle buttons + */ + initDomainIndexTable: () => { + const toggler = (el) => { + const idNumber = el.id.substr(7); + const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`); + if (el.src.substr(-9) === "minus.png") { + el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`; + toggledRows.forEach((el) => (el.style.display = "none")); + } else { + el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`; + toggledRows.forEach((el) => (el.style.display = "")); + } + }; + + const togglerElements = document.querySelectorAll("img.toggler"); + togglerElements.forEach((el) => + el.addEventListener("click", (event) => toggler(event.currentTarget)) + ); + togglerElements.forEach((el) => (el.style.display = "")); + if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler); + }, + + initOnKeyListeners: () => { + // only install a listener if it is really needed + if ( + !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS && + !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS + ) + return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.altKey || event.ctrlKey || event.metaKey) return; + + if (!event.shiftKey) { + switch (event.key) { + case "ArrowLeft": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const prevLink = document.querySelector('link[rel="prev"]'); + if (prevLink && prevLink.href) { + window.location.href = prevLink.href; + event.preventDefault(); + } + break; + case "ArrowRight": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const nextLink = document.querySelector('link[rel="next"]'); + if (nextLink && nextLink.href) { + window.location.href = nextLink.href; + event.preventDefault(); + } + break; + } + } + + // some keyboard layouts may need Shift to get / + switch (event.key) { + case "/": + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break; + Documentation.focusSearchBar(); + event.preventDefault(); + } + }); + }, +}; + +// quick alias for translations +const _ = Documentation.gettext; + +_ready(Documentation.init); diff --git a/en/2.9/_static/documentation_options.js b/en/2.9/_static/documentation_options.js new file mode 100644 index 0000000000..a3b39416b6 --- /dev/null +++ b/en/2.9/_static/documentation_options.js @@ -0,0 +1,14 @@ +var DOCUMENTATION_OPTIONS = { + URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), + VERSION: 'Netris v2.9', + LANGUAGE: 'en', + COLLAPSE_INDEX: false, + BUILDER: 'html', + FILE_SUFFIX: '.html', + LINK_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt', + NAVIGATION_WITH_KEYS: false, + SHOW_SEARCH_SUMMARY: true, + ENABLE_SEARCH_SHORTCUTS: true, +}; \ No newline at end of file diff --git a/en/2.9/_static/file.png b/en/2.9/_static/file.png new file mode 100644 index 0000000000..a858a410e4 Binary files /dev/null and b/en/2.9/_static/file.png differ diff --git a/en/2.9/_static/jquery-3.6.0.js b/en/2.9/_static/jquery-3.6.0.js new file mode 100644 index 0000000000..fc6c299b73 --- /dev/null +++ b/en/2.9/_static/jquery-3.6.0.js @@ -0,0 +1,10881 @@ +/*! + * jQuery JavaScript Library v3.6.0 + * https://jquery.com/ + * + * Includes Sizzle.js + * https://sizzlejs.com/ + * + * Copyright OpenJS Foundation and other contributors + * Released under the MIT license + * https://jquery.org/license + * + * Date: 2021-03-02T17:08Z + */ +( function( global, factory ) { + + "use strict"; + + if ( typeof module === "object" && typeof module.exports === "object" ) { + + // For CommonJS and CommonJS-like environments where a proper `window` + // is present, execute the factory and get jQuery. + // For environments that do not have a `window` with a `document` + // (such as Node.js), expose a factory as module.exports. + // This accentuates the need for the creation of a real `window`. + // e.g. var jQuery = require("jquery")(window); + // See ticket #14549 for more info. + module.exports = global.document ? + factory( global, true ) : + function( w ) { + if ( !w.document ) { + throw new Error( "jQuery requires a window with a document" ); + } + return factory( w ); + }; + } else { + factory( global ); + } + +// Pass this if window is not defined yet +} )( typeof window !== "undefined" ? window : this, function( window, noGlobal ) { + +// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 - 9.1 +// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict mode +// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict mode should be common +// enough that all such attempts are guarded in a try block. +"use strict"; + +var arr = []; + +var getProto = Object.getPrototypeOf; + +var slice = arr.slice; + +var flat = arr.flat ? function( array ) { + return arr.flat.call( array ); +} : function( array ) { + return arr.concat.apply( [], array ); +}; + + +var push = arr.push; + +var indexOf = arr.indexOf; + +var class2type = {}; + +var toString = class2type.toString; + +var hasOwn = class2type.hasOwnProperty; + +var fnToString = hasOwn.toString; + +var ObjectFunctionString = fnToString.call( Object ); + +var support = {}; + +var isFunction = function isFunction( obj ) { + + // Support: Chrome <=57, Firefox <=52 + // In some browsers, typeof returns "function" for HTML elements + // (i.e., `typeof document.createElement( "object" ) === "function"`). + // We don't want to classify *any* DOM node as a function. + // Support: QtWeb <=3.8.5, WebKit <=534.34, wkhtmltopdf tool <=0.12.5 + // Plus for old WebKit, typeof returns "function" for HTML collections + // (e.g., `typeof document.getElementsByTagName("div") === "function"`). (gh-4756) + return typeof obj === "function" && typeof obj.nodeType !== "number" && + typeof obj.item !== "function"; + }; + + +var isWindow = function isWindow( obj ) { + return obj != null && obj === obj.window; + }; + + +var document = window.document; + + + + var preservedScriptAttributes = { + type: true, + src: true, + nonce: true, + noModule: true + }; + + function DOMEval( code, node, doc ) { + doc = doc || document; + + var i, val, + script = doc.createElement( "script" ); + + script.text = code; + if ( node ) { + for ( i in preservedScriptAttributes ) { + + // Support: Firefox 64+, Edge 18+ + // Some browsers don't support the "nonce" property on scripts. + // On the other hand, just using `getAttribute` is not enough as + // the `nonce` attribute is reset to an empty string whenever it + // becomes browsing-context connected. + // See https://github.com/whatwg/html/issues/2369 + // See https://html.spec.whatwg.org/#nonce-attributes + // The `node.getAttribute` check was added for the sake of + // `jQuery.globalEval` so that it can fake a nonce-containing node + // via an object. + val = node[ i ] || node.getAttribute && node.getAttribute( i ); + if ( val ) { + script.setAttribute( i, val ); + } + } + } + doc.head.appendChild( script ).parentNode.removeChild( script ); + } + + +function toType( obj ) { + if ( obj == null ) { + return obj + ""; + } + + // Support: Android <=2.3 only (functionish RegExp) + return typeof obj === "object" || typeof obj === "function" ? + class2type[ toString.call( obj ) ] || "object" : + typeof obj; +} +/* global Symbol */ +// Defining this global in .eslintrc.json would create a danger of using the global +// unguarded in another place, it seems safer to define global only for this module + + + +var + version = "3.6.0", + + // Define a local copy of jQuery + jQuery = function( selector, context ) { + + // The jQuery object is actually just the init constructor 'enhanced' + // Need init if jQuery is called (just allow error to be thrown if not included) + return new jQuery.fn.init( selector, context ); + }; + +jQuery.fn = jQuery.prototype = { + + // The current version of jQuery being used + jquery: version, + + constructor: jQuery, + + // The default length of a jQuery object is 0 + length: 0, + + toArray: function() { + return slice.call( this ); + }, + + // Get the Nth element in the matched element set OR + // Get the whole matched element set as a clean array + get: function( num ) { + + // Return all the elements in a clean array + if ( num == null ) { + return slice.call( this ); + } + + // Return just the one element from the set + return num < 0 ? this[ num + this.length ] : this[ num ]; + }, + + // Take an array of elements and push it onto the stack + // (returning the new matched element set) + pushStack: function( elems ) { + + // Build a new jQuery matched element set + var ret = jQuery.merge( this.constructor(), elems ); + + // Add the old object onto the stack (as a reference) + ret.prevObject = this; + + // Return the newly-formed element set + return ret; + }, + + // Execute a callback for every element in the matched set. + each: function( callback ) { + return jQuery.each( this, callback ); + }, + + map: function( callback ) { + return this.pushStack( jQuery.map( this, function( elem, i ) { + return callback.call( elem, i, elem ); + } ) ); + }, + + slice: function() { + return this.pushStack( slice.apply( this, arguments ) ); + }, + + first: function() { + return this.eq( 0 ); + }, + + last: function() { + return this.eq( -1 ); + }, + + even: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return ( i + 1 ) % 2; + } ) ); + }, + + odd: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return i % 2; + } ) ); + }, + + eq: function( i ) { + var len = this.length, + j = +i + ( i < 0 ? len : 0 ); + return this.pushStack( j >= 0 && j < len ? [ this[ j ] ] : [] ); + }, + + end: function() { + return this.prevObject || this.constructor(); + }, + + // For internal use only. + // Behaves like an Array's method, not like a jQuery method. + push: push, + sort: arr.sort, + splice: arr.splice +}; + +jQuery.extend = jQuery.fn.extend = function() { + var options, name, src, copy, copyIsArray, clone, + target = arguments[ 0 ] || {}, + i = 1, + length = arguments.length, + deep = false; + + // Handle a deep copy situation + if ( typeof target === "boolean" ) { + deep = target; + + // Skip the boolean and the target + target = arguments[ i ] || {}; + i++; + } + + // Handle case when target is a string or something (possible in deep copy) + if ( typeof target !== "object" && !isFunction( target ) ) { + target = {}; + } + + // Extend jQuery itself if only one argument is passed + if ( i === length ) { + target = this; + i--; + } + + for ( ; i < length; i++ ) { + + // Only deal with non-null/undefined values + if ( ( options = arguments[ i ] ) != null ) { + + // Extend the base object + for ( name in options ) { + copy = options[ name ]; + + // Prevent Object.prototype pollution + // Prevent never-ending loop + if ( name === "__proto__" || target === copy ) { + continue; + } + + // Recurse if we're merging plain objects or arrays + if ( deep && copy && ( jQuery.isPlainObject( copy ) || + ( copyIsArray = Array.isArray( copy ) ) ) ) { + src = target[ name ]; + + // Ensure proper type for the source value + if ( copyIsArray && !Array.isArray( src ) ) { + clone = []; + } else if ( !copyIsArray && !jQuery.isPlainObject( src ) ) { + clone = {}; + } else { + clone = src; + } + copyIsArray = false; + + // Never move original objects, clone them + target[ name ] = jQuery.extend( deep, clone, copy ); + + // Don't bring in undefined values + } else if ( copy !== undefined ) { + target[ name ] = copy; + } + } + } + } + + // Return the modified object + return target; +}; + +jQuery.extend( { + + // Unique for each copy of jQuery on the page + expando: "jQuery" + ( version + Math.random() ).replace( /\D/g, "" ), + + // Assume jQuery is ready without the ready module + isReady: true, + + error: function( msg ) { + throw new Error( msg ); + }, + + noop: function() {}, + + isPlainObject: function( obj ) { + var proto, Ctor; + + // Detect obvious negatives + // Use toString instead of jQuery.type to catch host objects + if ( !obj || toString.call( obj ) !== "[object Object]" ) { + return false; + } + + proto = getProto( obj ); + + // Objects with no prototype (e.g., `Object.create( null )`) are plain + if ( !proto ) { + return true; + } + + // Objects with prototype are plain iff they were constructed by a global Object function + Ctor = hasOwn.call( proto, "constructor" ) && proto.constructor; + return typeof Ctor === "function" && fnToString.call( Ctor ) === ObjectFunctionString; + }, + + isEmptyObject: function( obj ) { + var name; + + for ( name in obj ) { + return false; + } + return true; + }, + + // Evaluates a script in a provided context; falls back to the global one + // if not specified. + globalEval: function( code, options, doc ) { + DOMEval( code, { nonce: options && options.nonce }, doc ); + }, + + each: function( obj, callback ) { + var length, i = 0; + + if ( isArrayLike( obj ) ) { + length = obj.length; + for ( ; i < length; i++ ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } else { + for ( i in obj ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } + + return obj; + }, + + // results is for internal usage only + makeArray: function( arr, results ) { + var ret = results || []; + + if ( arr != null ) { + if ( isArrayLike( Object( arr ) ) ) { + jQuery.merge( ret, + typeof arr === "string" ? + [ arr ] : arr + ); + } else { + push.call( ret, arr ); + } + } + + return ret; + }, + + inArray: function( elem, arr, i ) { + return arr == null ? -1 : indexOf.call( arr, elem, i ); + }, + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + merge: function( first, second ) { + var len = +second.length, + j = 0, + i = first.length; + + for ( ; j < len; j++ ) { + first[ i++ ] = second[ j ]; + } + + first.length = i; + + return first; + }, + + grep: function( elems, callback, invert ) { + var callbackInverse, + matches = [], + i = 0, + length = elems.length, + callbackExpect = !invert; + + // Go through the array, only saving the items + // that pass the validator function + for ( ; i < length; i++ ) { + callbackInverse = !callback( elems[ i ], i ); + if ( callbackInverse !== callbackExpect ) { + matches.push( elems[ i ] ); + } + } + + return matches; + }, + + // arg is for internal usage only + map: function( elems, callback, arg ) { + var length, value, + i = 0, + ret = []; + + // Go through the array, translating each of the items to their new values + if ( isArrayLike( elems ) ) { + length = elems.length; + for ( ; i < length; i++ ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + + // Go through every key on the object, + } else { + for ( i in elems ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + } + + // Flatten any nested arrays + return flat( ret ); + }, + + // A global GUID counter for objects + guid: 1, + + // jQuery.support is not used in Core but other projects attach their + // properties to it so it needs to exist. + support: support +} ); + +if ( typeof Symbol === "function" ) { + jQuery.fn[ Symbol.iterator ] = arr[ Symbol.iterator ]; +} + +// Populate the class2type map +jQuery.each( "Boolean Number String Function Array Date RegExp Object Error Symbol".split( " " ), + function( _i, name ) { + class2type[ "[object " + name + "]" ] = name.toLowerCase(); + } ); + +function isArrayLike( obj ) { + + // Support: real iOS 8.2 only (not reproducible in simulator) + // `in` check used to prevent JIT error (gh-2145) + // hasOwn isn't used here due to false negatives + // regarding Nodelist length in IE + var length = !!obj && "length" in obj && obj.length, + type = toType( obj ); + + if ( isFunction( obj ) || isWindow( obj ) ) { + return false; + } + + return type === "array" || length === 0 || + typeof length === "number" && length > 0 && ( length - 1 ) in obj; +} +var Sizzle = +/*! + * Sizzle CSS Selector Engine v2.3.6 + * https://sizzlejs.com/ + * + * Copyright JS Foundation and other contributors + * Released under the MIT license + * https://js.foundation/ + * + * Date: 2021-02-16 + */ +( function( window ) { +var i, + support, + Expr, + getText, + isXML, + tokenize, + compile, + select, + outermostContext, + sortInput, + hasDuplicate, + + // Local document vars + setDocument, + document, + docElem, + documentIsHTML, + rbuggyQSA, + rbuggyMatches, + matches, + contains, + + // Instance-specific data + expando = "sizzle" + 1 * new Date(), + preferredDoc = window.document, + dirruns = 0, + done = 0, + classCache = createCache(), + tokenCache = createCache(), + compilerCache = createCache(), + nonnativeSelectorCache = createCache(), + sortOrder = function( a, b ) { + if ( a === b ) { + hasDuplicate = true; + } + return 0; + }, + + // Instance methods + hasOwn = ( {} ).hasOwnProperty, + arr = [], + pop = arr.pop, + pushNative = arr.push, + push = arr.push, + slice = arr.slice, + + // Use a stripped-down indexOf as it's faster than native + // https://jsperf.com/thor-indexof-vs-for/5 + indexOf = function( list, elem ) { + var i = 0, + len = list.length; + for ( ; i < len; i++ ) { + if ( list[ i ] === elem ) { + return i; + } + } + return -1; + }, + + booleans = "checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|" + + "ismap|loop|multiple|open|readonly|required|scoped", + + // Regular expressions + + // http://www.w3.org/TR/css3-selectors/#whitespace + whitespace = "[\\x20\\t\\r\\n\\f]", + + // https://www.w3.org/TR/css-syntax-3/#ident-token-diagram + identifier = "(?:\\\\[\\da-fA-F]{1,6}" + whitespace + + "?|\\\\[^\\r\\n\\f]|[\\w-]|[^\0-\\x7f])+", + + // Attribute selectors: http://www.w3.org/TR/selectors/#attribute-selectors + attributes = "\\[" + whitespace + "*(" + identifier + ")(?:" + whitespace + + + // Operator (capture 2) + "*([*^$|!~]?=)" + whitespace + + + // "Attribute values must be CSS identifiers [capture 5] + // or strings [capture 3 or capture 4]" + "*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|(" + identifier + "))|)" + + whitespace + "*\\]", + + pseudos = ":(" + identifier + ")(?:\\((" + + + // To reduce the number of selectors needing tokenize in the preFilter, prefer arguments: + // 1. quoted (capture 3; capture 4 or capture 5) + "('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|" + + + // 2. simple (capture 6) + "((?:\\\\.|[^\\\\()[\\]]|" + attributes + ")*)|" + + + // 3. anything else (capture 2) + ".*" + + ")\\)|)", + + // Leading and non-escaped trailing whitespace, capturing some non-whitespace characters preceding the latter + rwhitespace = new RegExp( whitespace + "+", "g" ), + rtrim = new RegExp( "^" + whitespace + "+|((?:^|[^\\\\])(?:\\\\.)*)" + + whitespace + "+$", "g" ), + + rcomma = new RegExp( "^" + whitespace + "*," + whitespace + "*" ), + rcombinators = new RegExp( "^" + whitespace + "*([>+~]|" + whitespace + ")" + whitespace + + "*" ), + rdescend = new RegExp( whitespace + "|>" ), + + rpseudo = new RegExp( pseudos ), + ridentifier = new RegExp( "^" + identifier + "$" ), + + matchExpr = { + "ID": new RegExp( "^#(" + identifier + ")" ), + "CLASS": new RegExp( "^\\.(" + identifier + ")" ), + "TAG": new RegExp( "^(" + identifier + "|[*])" ), + "ATTR": new RegExp( "^" + attributes ), + "PSEUDO": new RegExp( "^" + pseudos ), + "CHILD": new RegExp( "^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\(" + + whitespace + "*(even|odd|(([+-]|)(\\d*)n|)" + whitespace + "*(?:([+-]|)" + + whitespace + "*(\\d+)|))" + whitespace + "*\\)|)", "i" ), + "bool": new RegExp( "^(?:" + booleans + ")$", "i" ), + + // For use in libraries implementing .is() + // We use this for POS matching in `select` + "needsContext": new RegExp( "^" + whitespace + + "*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\(" + whitespace + + "*((?:-\\d)?\\d*)" + whitespace + "*\\)|)(?=[^-]|$)", "i" ) + }, + + rhtml = /HTML$/i, + rinputs = /^(?:input|select|textarea|button)$/i, + rheader = /^h\d$/i, + + rnative = /^[^{]+\{\s*\[native \w/, + + // Easily-parseable/retrievable ID or TAG or CLASS selectors + rquickExpr = /^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/, + + rsibling = /[+~]/, + + // CSS escapes + // http://www.w3.org/TR/CSS21/syndata.html#escaped-characters + runescape = new RegExp( "\\\\[\\da-fA-F]{1,6}" + whitespace + "?|\\\\([^\\r\\n\\f])", "g" ), + funescape = function( escape, nonHex ) { + var high = "0x" + escape.slice( 1 ) - 0x10000; + + return nonHex ? + + // Strip the backslash prefix from a non-hex escape sequence + nonHex : + + // Replace a hexadecimal escape sequence with the encoded Unicode code point + // Support: IE <=11+ + // For values outside the Basic Multilingual Plane (BMP), manually construct a + // surrogate pair + high < 0 ? + String.fromCharCode( high + 0x10000 ) : + String.fromCharCode( high >> 10 | 0xD800, high & 0x3FF | 0xDC00 ); + }, + + // CSS string/identifier serialization + // https://drafts.csswg.org/cssom/#common-serializing-idioms + rcssescape = /([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g, + fcssescape = function( ch, asCodePoint ) { + if ( asCodePoint ) { + + // U+0000 NULL becomes U+FFFD REPLACEMENT CHARACTER + if ( ch === "\0" ) { + return "\uFFFD"; + } + + // Control characters and (dependent upon position) numbers get escaped as code points + return ch.slice( 0, -1 ) + "\\" + + ch.charCodeAt( ch.length - 1 ).toString( 16 ) + " "; + } + + // Other potentially-special ASCII characters get backslash-escaped + return "\\" + ch; + }, + + // Used for iframes + // See setDocument() + // Removing the function wrapper causes a "Permission Denied" + // error in IE + unloadHandler = function() { + setDocument(); + }, + + inDisabledFieldset = addCombinator( + function( elem ) { + return elem.disabled === true && elem.nodeName.toLowerCase() === "fieldset"; + }, + { dir: "parentNode", next: "legend" } + ); + +// Optimize for push.apply( _, NodeList ) +try { + push.apply( + ( arr = slice.call( preferredDoc.childNodes ) ), + preferredDoc.childNodes + ); + + // Support: Android<4.0 + // Detect silently failing push.apply + // eslint-disable-next-line no-unused-expressions + arr[ preferredDoc.childNodes.length ].nodeType; +} catch ( e ) { + push = { apply: arr.length ? + + // Leverage slice if possible + function( target, els ) { + pushNative.apply( target, slice.call( els ) ); + } : + + // Support: IE<9 + // Otherwise append directly + function( target, els ) { + var j = target.length, + i = 0; + + // Can't trust NodeList.length + while ( ( target[ j++ ] = els[ i++ ] ) ) {} + target.length = j - 1; + } + }; +} + +function Sizzle( selector, context, results, seed ) { + var m, i, elem, nid, match, groups, newSelector, + newContext = context && context.ownerDocument, + + // nodeType defaults to 9, since context defaults to document + nodeType = context ? context.nodeType : 9; + + results = results || []; + + // Return early from calls with invalid selector or context + if ( typeof selector !== "string" || !selector || + nodeType !== 1 && nodeType !== 9 && nodeType !== 11 ) { + + return results; + } + + // Try to shortcut find operations (as opposed to filters) in HTML documents + if ( !seed ) { + setDocument( context ); + context = context || document; + + if ( documentIsHTML ) { + + // If the selector is sufficiently simple, try using a "get*By*" DOM method + // (excepting DocumentFragment context, where the methods don't exist) + if ( nodeType !== 11 && ( match = rquickExpr.exec( selector ) ) ) { + + // ID selector + if ( ( m = match[ 1 ] ) ) { + + // Document context + if ( nodeType === 9 ) { + if ( ( elem = context.getElementById( m ) ) ) { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( elem.id === m ) { + results.push( elem ); + return results; + } + } else { + return results; + } + + // Element context + } else { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( newContext && ( elem = newContext.getElementById( m ) ) && + contains( context, elem ) && + elem.id === m ) { + + results.push( elem ); + return results; + } + } + + // Type selector + } else if ( match[ 2 ] ) { + push.apply( results, context.getElementsByTagName( selector ) ); + return results; + + // Class selector + } else if ( ( m = match[ 3 ] ) && support.getElementsByClassName && + context.getElementsByClassName ) { + + push.apply( results, context.getElementsByClassName( m ) ); + return results; + } + } + + // Take advantage of querySelectorAll + if ( support.qsa && + !nonnativeSelectorCache[ selector + " " ] && + ( !rbuggyQSA || !rbuggyQSA.test( selector ) ) && + + // Support: IE 8 only + // Exclude object elements + ( nodeType !== 1 || context.nodeName.toLowerCase() !== "object" ) ) { + + newSelector = selector; + newContext = context; + + // qSA considers elements outside a scoping root when evaluating child or + // descendant combinators, which is not what we want. + // In such cases, we work around the behavior by prefixing every selector in the + // list with an ID selector referencing the scope context. + // The technique has to be used as well when a leading combinator is used + // as such selectors are not recognized by querySelectorAll. + // Thanks to Andrew Dupont for this technique. + if ( nodeType === 1 && + ( rdescend.test( selector ) || rcombinators.test( selector ) ) ) { + + // Expand context for sibling selectors + newContext = rsibling.test( selector ) && testContext( context.parentNode ) || + context; + + // We can use :scope instead of the ID hack if the browser + // supports it & if we're not changing the context. + if ( newContext !== context || !support.scope ) { + + // Capture the context ID, setting it first if necessary + if ( ( nid = context.getAttribute( "id" ) ) ) { + nid = nid.replace( rcssescape, fcssescape ); + } else { + context.setAttribute( "id", ( nid = expando ) ); + } + } + + // Prefix every selector in the list + groups = tokenize( selector ); + i = groups.length; + while ( i-- ) { + groups[ i ] = ( nid ? "#" + nid : ":scope" ) + " " + + toSelector( groups[ i ] ); + } + newSelector = groups.join( "," ); + } + + try { + push.apply( results, + newContext.querySelectorAll( newSelector ) + ); + return results; + } catch ( qsaError ) { + nonnativeSelectorCache( selector, true ); + } finally { + if ( nid === expando ) { + context.removeAttribute( "id" ); + } + } + } + } + } + + // All others + return select( selector.replace( rtrim, "$1" ), context, results, seed ); +} + +/** + * Create key-value caches of limited size + * @returns {function(string, object)} Returns the Object data after storing it on itself with + * property name the (space-suffixed) string and (if the cache is larger than Expr.cacheLength) + * deleting the oldest entry + */ +function createCache() { + var keys = []; + + function cache( key, value ) { + + // Use (key + " ") to avoid collision with native prototype properties (see Issue #157) + if ( keys.push( key + " " ) > Expr.cacheLength ) { + + // Only keep the most recent entries + delete cache[ keys.shift() ]; + } + return ( cache[ key + " " ] = value ); + } + return cache; +} + +/** + * Mark a function for special use by Sizzle + * @param {Function} fn The function to mark + */ +function markFunction( fn ) { + fn[ expando ] = true; + return fn; +} + +/** + * Support testing using an element + * @param {Function} fn Passed the created element and returns a boolean result + */ +function assert( fn ) { + var el = document.createElement( "fieldset" ); + + try { + return !!fn( el ); + } catch ( e ) { + return false; + } finally { + + // Remove from its parent by default + if ( el.parentNode ) { + el.parentNode.removeChild( el ); + } + + // release memory in IE + el = null; + } +} + +/** + * Adds the same handler for all of the specified attrs + * @param {String} attrs Pipe-separated list of attributes + * @param {Function} handler The method that will be applied + */ +function addHandle( attrs, handler ) { + var arr = attrs.split( "|" ), + i = arr.length; + + while ( i-- ) { + Expr.attrHandle[ arr[ i ] ] = handler; + } +} + +/** + * Checks document order of two siblings + * @param {Element} a + * @param {Element} b + * @returns {Number} Returns less than 0 if a precedes b, greater than 0 if a follows b + */ +function siblingCheck( a, b ) { + var cur = b && a, + diff = cur && a.nodeType === 1 && b.nodeType === 1 && + a.sourceIndex - b.sourceIndex; + + // Use IE sourceIndex if available on both nodes + if ( diff ) { + return diff; + } + + // Check if b follows a + if ( cur ) { + while ( ( cur = cur.nextSibling ) ) { + if ( cur === b ) { + return -1; + } + } + } + + return a ? 1 : -1; +} + +/** + * Returns a function to use in pseudos for input types + * @param {String} type + */ +function createInputPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for buttons + * @param {String} type + */ +function createButtonPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return ( name === "input" || name === "button" ) && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for :enabled/:disabled + * @param {Boolean} disabled true for :disabled; false for :enabled + */ +function createDisabledPseudo( disabled ) { + + // Known :disabled false positives: fieldset[disabled] > legend:nth-of-type(n+2) :can-disable + return function( elem ) { + + // Only certain elements can match :enabled or :disabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-enabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-disabled + if ( "form" in elem ) { + + // Check for inherited disabledness on relevant non-disabled elements: + // * listed form-associated elements in a disabled fieldset + // https://html.spec.whatwg.org/multipage/forms.html#category-listed + // https://html.spec.whatwg.org/multipage/forms.html#concept-fe-disabled + // * option elements in a disabled optgroup + // https://html.spec.whatwg.org/multipage/forms.html#concept-option-disabled + // All such elements have a "form" property. + if ( elem.parentNode && elem.disabled === false ) { + + // Option elements defer to a parent optgroup if present + if ( "label" in elem ) { + if ( "label" in elem.parentNode ) { + return elem.parentNode.disabled === disabled; + } else { + return elem.disabled === disabled; + } + } + + // Support: IE 6 - 11 + // Use the isDisabled shortcut property to check for disabled fieldset ancestors + return elem.isDisabled === disabled || + + // Where there is no isDisabled, check manually + /* jshint -W018 */ + elem.isDisabled !== !disabled && + inDisabledFieldset( elem ) === disabled; + } + + return elem.disabled === disabled; + + // Try to winnow out elements that can't be disabled before trusting the disabled property. + // Some victims get caught in our net (label, legend, menu, track), but it shouldn't + // even exist on them, let alone have a boolean value. + } else if ( "label" in elem ) { + return elem.disabled === disabled; + } + + // Remaining elements are neither :enabled nor :disabled + return false; + }; +} + +/** + * Returns a function to use in pseudos for positionals + * @param {Function} fn + */ +function createPositionalPseudo( fn ) { + return markFunction( function( argument ) { + argument = +argument; + return markFunction( function( seed, matches ) { + var j, + matchIndexes = fn( [], seed.length, argument ), + i = matchIndexes.length; + + // Match elements found at the specified indexes + while ( i-- ) { + if ( seed[ ( j = matchIndexes[ i ] ) ] ) { + seed[ j ] = !( matches[ j ] = seed[ j ] ); + } + } + } ); + } ); +} + +/** + * Checks a node for validity as a Sizzle context + * @param {Element|Object=} context + * @returns {Element|Object|Boolean} The input node if acceptable, otherwise a falsy value + */ +function testContext( context ) { + return context && typeof context.getElementsByTagName !== "undefined" && context; +} + +// Expose support vars for convenience +support = Sizzle.support = {}; + +/** + * Detects XML nodes + * @param {Element|Object} elem An element or a document + * @returns {Boolean} True iff elem is a non-HTML XML node + */ +isXML = Sizzle.isXML = function( elem ) { + var namespace = elem && elem.namespaceURI, + docElem = elem && ( elem.ownerDocument || elem ).documentElement; + + // Support: IE <=8 + // Assume HTML when documentElement doesn't yet exist, such as inside loading iframes + // https://bugs.jquery.com/ticket/4833 + return !rhtml.test( namespace || docElem && docElem.nodeName || "HTML" ); +}; + +/** + * Sets document-related variables once based on the current document + * @param {Element|Object} [doc] An element or document object to use to set the document + * @returns {Object} Returns the current document + */ +setDocument = Sizzle.setDocument = function( node ) { + var hasCompare, subWindow, + doc = node ? node.ownerDocument || node : preferredDoc; + + // Return early if doc is invalid or already selected + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( doc == document || doc.nodeType !== 9 || !doc.documentElement ) { + return document; + } + + // Update global variables + document = doc; + docElem = document.documentElement; + documentIsHTML = !isXML( document ); + + // Support: IE 9 - 11+, Edge 12 - 18+ + // Accessing iframe documents after unload throws "permission denied" errors (jQuery #13936) + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( preferredDoc != document && + ( subWindow = document.defaultView ) && subWindow.top !== subWindow ) { + + // Support: IE 11, Edge + if ( subWindow.addEventListener ) { + subWindow.addEventListener( "unload", unloadHandler, false ); + + // Support: IE 9 - 10 only + } else if ( subWindow.attachEvent ) { + subWindow.attachEvent( "onunload", unloadHandler ); + } + } + + // Support: IE 8 - 11+, Edge 12 - 18+, Chrome <=16 - 25 only, Firefox <=3.6 - 31 only, + // Safari 4 - 5 only, Opera <=11.6 - 12.x only + // IE/Edge & older browsers don't support the :scope pseudo-class. + // Support: Safari 6.0 only + // Safari 6.0 supports :scope but it's an alias of :root there. + support.scope = assert( function( el ) { + docElem.appendChild( el ).appendChild( document.createElement( "div" ) ); + return typeof el.querySelectorAll !== "undefined" && + !el.querySelectorAll( ":scope fieldset div" ).length; + } ); + + /* Attributes + ---------------------------------------------------------------------- */ + + // Support: IE<8 + // Verify that getAttribute really returns attributes and not properties + // (excepting IE8 booleans) + support.attributes = assert( function( el ) { + el.className = "i"; + return !el.getAttribute( "className" ); + } ); + + /* getElement(s)By* + ---------------------------------------------------------------------- */ + + // Check if getElementsByTagName("*") returns only elements + support.getElementsByTagName = assert( function( el ) { + el.appendChild( document.createComment( "" ) ); + return !el.getElementsByTagName( "*" ).length; + } ); + + // Support: IE<9 + support.getElementsByClassName = rnative.test( document.getElementsByClassName ); + + // Support: IE<10 + // Check if getElementById returns elements by name + // The broken getElementById methods don't pick up programmatically-set names, + // so use a roundabout getElementsByName test + support.getById = assert( function( el ) { + docElem.appendChild( el ).id = expando; + return !document.getElementsByName || !document.getElementsByName( expando ).length; + } ); + + // ID filter and find + if ( support.getById ) { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + return elem.getAttribute( "id" ) === attrId; + }; + }; + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var elem = context.getElementById( id ); + return elem ? [ elem ] : []; + } + }; + } else { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + var node = typeof elem.getAttributeNode !== "undefined" && + elem.getAttributeNode( "id" ); + return node && node.value === attrId; + }; + }; + + // Support: IE 6 - 7 only + // getElementById is not reliable as a find shortcut + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var node, i, elems, + elem = context.getElementById( id ); + + if ( elem ) { + + // Verify the id attribute + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + + // Fall back on getElementsByName + elems = context.getElementsByName( id ); + i = 0; + while ( ( elem = elems[ i++ ] ) ) { + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + } + } + + return []; + } + }; + } + + // Tag + Expr.find[ "TAG" ] = support.getElementsByTagName ? + function( tag, context ) { + if ( typeof context.getElementsByTagName !== "undefined" ) { + return context.getElementsByTagName( tag ); + + // DocumentFragment nodes don't have gEBTN + } else if ( support.qsa ) { + return context.querySelectorAll( tag ); + } + } : + + function( tag, context ) { + var elem, + tmp = [], + i = 0, + + // By happy coincidence, a (broken) gEBTN appears on DocumentFragment nodes too + results = context.getElementsByTagName( tag ); + + // Filter out possible comments + if ( tag === "*" ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem.nodeType === 1 ) { + tmp.push( elem ); + } + } + + return tmp; + } + return results; + }; + + // Class + Expr.find[ "CLASS" ] = support.getElementsByClassName && function( className, context ) { + if ( typeof context.getElementsByClassName !== "undefined" && documentIsHTML ) { + return context.getElementsByClassName( className ); + } + }; + + /* QSA/matchesSelector + ---------------------------------------------------------------------- */ + + // QSA and matchesSelector support + + // matchesSelector(:active) reports false when true (IE9/Opera 11.5) + rbuggyMatches = []; + + // qSa(:focus) reports false when true (Chrome 21) + // We allow this because of a bug in IE8/9 that throws an error + // whenever `document.activeElement` is accessed on an iframe + // So, we allow :focus to pass through QSA all the time to avoid the IE error + // See https://bugs.jquery.com/ticket/13378 + rbuggyQSA = []; + + if ( ( support.qsa = rnative.test( document.querySelectorAll ) ) ) { + + // Build QSA regex + // Regex strategy adopted from Diego Perini + assert( function( el ) { + + var input; + + // Select is set to empty string on purpose + // This is to test IE's treatment of not explicitly + // setting a boolean content attribute, + // since its presence should be enough + // https://bugs.jquery.com/ticket/12359 + docElem.appendChild( el ).innerHTML = "" + + ""; + + // Support: IE8, Opera 11-12.16 + // Nothing should be selected when empty strings follow ^= or $= or *= + // The test attribute must be unknown in Opera but "safe" for WinRT + // https://msdn.microsoft.com/en-us/library/ie/hh465388.aspx#attribute_section + if ( el.querySelectorAll( "[msallowcapture^='']" ).length ) { + rbuggyQSA.push( "[*^$]=" + whitespace + "*(?:''|\"\")" ); + } + + // Support: IE8 + // Boolean attributes and "value" are not treated correctly + if ( !el.querySelectorAll( "[selected]" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*(?:value|" + booleans + ")" ); + } + + // Support: Chrome<29, Android<4.4, Safari<7.0+, iOS<7.0+, PhantomJS<1.9.8+ + if ( !el.querySelectorAll( "[id~=" + expando + "-]" ).length ) { + rbuggyQSA.push( "~=" ); + } + + // Support: IE 11+, Edge 15 - 18+ + // IE 11/Edge don't find elements on a `[name='']` query in some cases. + // Adding a temporary attribute to the document before the selection works + // around the issue. + // Interestingly, IE 10 & older don't seem to have the issue. + input = document.createElement( "input" ); + input.setAttribute( "name", "" ); + el.appendChild( input ); + if ( !el.querySelectorAll( "[name='']" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*name" + whitespace + "*=" + + whitespace + "*(?:''|\"\")" ); + } + + // Webkit/Opera - :checked should return selected option elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + // IE8 throws error here and will not see later tests + if ( !el.querySelectorAll( ":checked" ).length ) { + rbuggyQSA.push( ":checked" ); + } + + // Support: Safari 8+, iOS 8+ + // https://bugs.webkit.org/show_bug.cgi?id=136851 + // In-page `selector#id sibling-combinator selector` fails + if ( !el.querySelectorAll( "a#" + expando + "+*" ).length ) { + rbuggyQSA.push( ".#.+[+~]" ); + } + + // Support: Firefox <=3.6 - 5 only + // Old Firefox doesn't throw on a badly-escaped identifier. + el.querySelectorAll( "\\\f" ); + rbuggyQSA.push( "[\\r\\n\\f]" ); + } ); + + assert( function( el ) { + el.innerHTML = "" + + ""; + + // Support: Windows 8 Native Apps + // The type and name attributes are restricted during .innerHTML assignment + var input = document.createElement( "input" ); + input.setAttribute( "type", "hidden" ); + el.appendChild( input ).setAttribute( "name", "D" ); + + // Support: IE8 + // Enforce case-sensitivity of name attribute + if ( el.querySelectorAll( "[name=d]" ).length ) { + rbuggyQSA.push( "name" + whitespace + "*[*^$|!~]?=" ); + } + + // FF 3.5 - :enabled/:disabled and hidden elements (hidden elements are still enabled) + // IE8 throws error here and will not see later tests + if ( el.querySelectorAll( ":enabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: IE9-11+ + // IE's :disabled selector does not pick up the children of disabled fieldsets + docElem.appendChild( el ).disabled = true; + if ( el.querySelectorAll( ":disabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: Opera 10 - 11 only + // Opera 10-11 does not throw on post-comma invalid pseudos + el.querySelectorAll( "*,:x" ); + rbuggyQSA.push( ",.*:" ); + } ); + } + + if ( ( support.matchesSelector = rnative.test( ( matches = docElem.matches || + docElem.webkitMatchesSelector || + docElem.mozMatchesSelector || + docElem.oMatchesSelector || + docElem.msMatchesSelector ) ) ) ) { + + assert( function( el ) { + + // Check to see if it's possible to do matchesSelector + // on a disconnected node (IE 9) + support.disconnectedMatch = matches.call( el, "*" ); + + // This should fail with an exception + // Gecko does not error, returns false instead + matches.call( el, "[s!='']:x" ); + rbuggyMatches.push( "!=", pseudos ); + } ); + } + + rbuggyQSA = rbuggyQSA.length && new RegExp( rbuggyQSA.join( "|" ) ); + rbuggyMatches = rbuggyMatches.length && new RegExp( rbuggyMatches.join( "|" ) ); + + /* Contains + ---------------------------------------------------------------------- */ + hasCompare = rnative.test( docElem.compareDocumentPosition ); + + // Element contains another + // Purposefully self-exclusive + // As in, an element does not contain itself + contains = hasCompare || rnative.test( docElem.contains ) ? + function( a, b ) { + var adown = a.nodeType === 9 ? a.documentElement : a, + bup = b && b.parentNode; + return a === bup || !!( bup && bup.nodeType === 1 && ( + adown.contains ? + adown.contains( bup ) : + a.compareDocumentPosition && a.compareDocumentPosition( bup ) & 16 + ) ); + } : + function( a, b ) { + if ( b ) { + while ( ( b = b.parentNode ) ) { + if ( b === a ) { + return true; + } + } + } + return false; + }; + + /* Sorting + ---------------------------------------------------------------------- */ + + // Document order sorting + sortOrder = hasCompare ? + function( a, b ) { + + // Flag for duplicate removal + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + // Sort on method existence if only one input has compareDocumentPosition + var compare = !a.compareDocumentPosition - !b.compareDocumentPosition; + if ( compare ) { + return compare; + } + + // Calculate position if both inputs belong to the same document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + compare = ( a.ownerDocument || a ) == ( b.ownerDocument || b ) ? + a.compareDocumentPosition( b ) : + + // Otherwise we know they are disconnected + 1; + + // Disconnected nodes + if ( compare & 1 || + ( !support.sortDetached && b.compareDocumentPosition( a ) === compare ) ) { + + // Choose the first element that is related to our preferred document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( a == document || a.ownerDocument == preferredDoc && + contains( preferredDoc, a ) ) { + return -1; + } + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( b == document || b.ownerDocument == preferredDoc && + contains( preferredDoc, b ) ) { + return 1; + } + + // Maintain original order + return sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + } + + return compare & 4 ? -1 : 1; + } : + function( a, b ) { + + // Exit early if the nodes are identical + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + var cur, + i = 0, + aup = a.parentNode, + bup = b.parentNode, + ap = [ a ], + bp = [ b ]; + + // Parentless nodes are either documents or disconnected + if ( !aup || !bup ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + return a == document ? -1 : + b == document ? 1 : + /* eslint-enable eqeqeq */ + aup ? -1 : + bup ? 1 : + sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + + // If the nodes are siblings, we can do a quick check + } else if ( aup === bup ) { + return siblingCheck( a, b ); + } + + // Otherwise we need full lists of their ancestors for comparison + cur = a; + while ( ( cur = cur.parentNode ) ) { + ap.unshift( cur ); + } + cur = b; + while ( ( cur = cur.parentNode ) ) { + bp.unshift( cur ); + } + + // Walk down the tree looking for a discrepancy + while ( ap[ i ] === bp[ i ] ) { + i++; + } + + return i ? + + // Do a sibling check if the nodes have a common ancestor + siblingCheck( ap[ i ], bp[ i ] ) : + + // Otherwise nodes in our document sort first + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + ap[ i ] == preferredDoc ? -1 : + bp[ i ] == preferredDoc ? 1 : + /* eslint-enable eqeqeq */ + 0; + }; + + return document; +}; + +Sizzle.matches = function( expr, elements ) { + return Sizzle( expr, null, null, elements ); +}; + +Sizzle.matchesSelector = function( elem, expr ) { + setDocument( elem ); + + if ( support.matchesSelector && documentIsHTML && + !nonnativeSelectorCache[ expr + " " ] && + ( !rbuggyMatches || !rbuggyMatches.test( expr ) ) && + ( !rbuggyQSA || !rbuggyQSA.test( expr ) ) ) { + + try { + var ret = matches.call( elem, expr ); + + // IE 9's matchesSelector returns false on disconnected nodes + if ( ret || support.disconnectedMatch || + + // As well, disconnected nodes are said to be in a document + // fragment in IE 9 + elem.document && elem.document.nodeType !== 11 ) { + return ret; + } + } catch ( e ) { + nonnativeSelectorCache( expr, true ); + } + } + + return Sizzle( expr, document, null, [ elem ] ).length > 0; +}; + +Sizzle.contains = function( context, elem ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( context.ownerDocument || context ) != document ) { + setDocument( context ); + } + return contains( context, elem ); +}; + +Sizzle.attr = function( elem, name ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( elem.ownerDocument || elem ) != document ) { + setDocument( elem ); + } + + var fn = Expr.attrHandle[ name.toLowerCase() ], + + // Don't get fooled by Object.prototype properties (jQuery #13807) + val = fn && hasOwn.call( Expr.attrHandle, name.toLowerCase() ) ? + fn( elem, name, !documentIsHTML ) : + undefined; + + return val !== undefined ? + val : + support.attributes || !documentIsHTML ? + elem.getAttribute( name ) : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; +}; + +Sizzle.escape = function( sel ) { + return ( sel + "" ).replace( rcssescape, fcssescape ); +}; + +Sizzle.error = function( msg ) { + throw new Error( "Syntax error, unrecognized expression: " + msg ); +}; + +/** + * Document sorting and removing duplicates + * @param {ArrayLike} results + */ +Sizzle.uniqueSort = function( results ) { + var elem, + duplicates = [], + j = 0, + i = 0; + + // Unless we *know* we can detect duplicates, assume their presence + hasDuplicate = !support.detectDuplicates; + sortInput = !support.sortStable && results.slice( 0 ); + results.sort( sortOrder ); + + if ( hasDuplicate ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem === results[ i ] ) { + j = duplicates.push( i ); + } + } + while ( j-- ) { + results.splice( duplicates[ j ], 1 ); + } + } + + // Clear input after sorting to release objects + // See https://github.com/jquery/sizzle/pull/225 + sortInput = null; + + return results; +}; + +/** + * Utility function for retrieving the text value of an array of DOM nodes + * @param {Array|Element} elem + */ +getText = Sizzle.getText = function( elem ) { + var node, + ret = "", + i = 0, + nodeType = elem.nodeType; + + if ( !nodeType ) { + + // If no nodeType, this is expected to be an array + while ( ( node = elem[ i++ ] ) ) { + + // Do not traverse comment nodes + ret += getText( node ); + } + } else if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) { + + // Use textContent for elements + // innerText usage removed for consistency of new lines (jQuery #11153) + if ( typeof elem.textContent === "string" ) { + return elem.textContent; + } else { + + // Traverse its children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + ret += getText( elem ); + } + } + } else if ( nodeType === 3 || nodeType === 4 ) { + return elem.nodeValue; + } + + // Do not include comment or processing instruction nodes + + return ret; +}; + +Expr = Sizzle.selectors = { + + // Can be adjusted by the user + cacheLength: 50, + + createPseudo: markFunction, + + match: matchExpr, + + attrHandle: {}, + + find: {}, + + relative: { + ">": { dir: "parentNode", first: true }, + " ": { dir: "parentNode" }, + "+": { dir: "previousSibling", first: true }, + "~": { dir: "previousSibling" } + }, + + preFilter: { + "ATTR": function( match ) { + match[ 1 ] = match[ 1 ].replace( runescape, funescape ); + + // Move the given value to match[3] whether quoted or unquoted + match[ 3 ] = ( match[ 3 ] || match[ 4 ] || + match[ 5 ] || "" ).replace( runescape, funescape ); + + if ( match[ 2 ] === "~=" ) { + match[ 3 ] = " " + match[ 3 ] + " "; + } + + return match.slice( 0, 4 ); + }, + + "CHILD": function( match ) { + + /* matches from matchExpr["CHILD"] + 1 type (only|nth|...) + 2 what (child|of-type) + 3 argument (even|odd|\d*|\d*n([+-]\d+)?|...) + 4 xn-component of xn+y argument ([+-]?\d*n|) + 5 sign of xn-component + 6 x of xn-component + 7 sign of y-component + 8 y of y-component + */ + match[ 1 ] = match[ 1 ].toLowerCase(); + + if ( match[ 1 ].slice( 0, 3 ) === "nth" ) { + + // nth-* requires argument + if ( !match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + // numeric x and y parameters for Expr.filter.CHILD + // remember that false/true cast respectively to 0/1 + match[ 4 ] = +( match[ 4 ] ? + match[ 5 ] + ( match[ 6 ] || 1 ) : + 2 * ( match[ 3 ] === "even" || match[ 3 ] === "odd" ) ); + match[ 5 ] = +( ( match[ 7 ] + match[ 8 ] ) || match[ 3 ] === "odd" ); + + // other types prohibit arguments + } else if ( match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + return match; + }, + + "PSEUDO": function( match ) { + var excess, + unquoted = !match[ 6 ] && match[ 2 ]; + + if ( matchExpr[ "CHILD" ].test( match[ 0 ] ) ) { + return null; + } + + // Accept quoted arguments as-is + if ( match[ 3 ] ) { + match[ 2 ] = match[ 4 ] || match[ 5 ] || ""; + + // Strip excess characters from unquoted arguments + } else if ( unquoted && rpseudo.test( unquoted ) && + + // Get excess from tokenize (recursively) + ( excess = tokenize( unquoted, true ) ) && + + // advance to the next closing parenthesis + ( excess = unquoted.indexOf( ")", unquoted.length - excess ) - unquoted.length ) ) { + + // excess is a negative index + match[ 0 ] = match[ 0 ].slice( 0, excess ); + match[ 2 ] = unquoted.slice( 0, excess ); + } + + // Return only captures needed by the pseudo filter method (type and argument) + return match.slice( 0, 3 ); + } + }, + + filter: { + + "TAG": function( nodeNameSelector ) { + var nodeName = nodeNameSelector.replace( runescape, funescape ).toLowerCase(); + return nodeNameSelector === "*" ? + function() { + return true; + } : + function( elem ) { + return elem.nodeName && elem.nodeName.toLowerCase() === nodeName; + }; + }, + + "CLASS": function( className ) { + var pattern = classCache[ className + " " ]; + + return pattern || + ( pattern = new RegExp( "(^|" + whitespace + + ")" + className + "(" + whitespace + "|$)" ) ) && classCache( + className, function( elem ) { + return pattern.test( + typeof elem.className === "string" && elem.className || + typeof elem.getAttribute !== "undefined" && + elem.getAttribute( "class" ) || + "" + ); + } ); + }, + + "ATTR": function( name, operator, check ) { + return function( elem ) { + var result = Sizzle.attr( elem, name ); + + if ( result == null ) { + return operator === "!="; + } + if ( !operator ) { + return true; + } + + result += ""; + + /* eslint-disable max-len */ + + return operator === "=" ? result === check : + operator === "!=" ? result !== check : + operator === "^=" ? check && result.indexOf( check ) === 0 : + operator === "*=" ? check && result.indexOf( check ) > -1 : + operator === "$=" ? check && result.slice( -check.length ) === check : + operator === "~=" ? ( " " + result.replace( rwhitespace, " " ) + " " ).indexOf( check ) > -1 : + operator === "|=" ? result === check || result.slice( 0, check.length + 1 ) === check + "-" : + false; + /* eslint-enable max-len */ + + }; + }, + + "CHILD": function( type, what, _argument, first, last ) { + var simple = type.slice( 0, 3 ) !== "nth", + forward = type.slice( -4 ) !== "last", + ofType = what === "of-type"; + + return first === 1 && last === 0 ? + + // Shortcut for :nth-*(n) + function( elem ) { + return !!elem.parentNode; + } : + + function( elem, _context, xml ) { + var cache, uniqueCache, outerCache, node, nodeIndex, start, + dir = simple !== forward ? "nextSibling" : "previousSibling", + parent = elem.parentNode, + name = ofType && elem.nodeName.toLowerCase(), + useCache = !xml && !ofType, + diff = false; + + if ( parent ) { + + // :(first|last|only)-(child|of-type) + if ( simple ) { + while ( dir ) { + node = elem; + while ( ( node = node[ dir ] ) ) { + if ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) { + + return false; + } + } + + // Reverse direction for :only-* (if we haven't yet done so) + start = dir = type === "only" && !start && "nextSibling"; + } + return true; + } + + start = [ forward ? parent.firstChild : parent.lastChild ]; + + // non-xml :nth-child(...) stores cache data on `parent` + if ( forward && useCache ) { + + // Seek `elem` from a previously-cached index + + // ...in a gzip-friendly way + node = parent; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex && cache[ 2 ]; + node = nodeIndex && parent.childNodes[ nodeIndex ]; + + while ( ( node = ++nodeIndex && node && node[ dir ] || + + // Fallback to seeking `elem` from the start + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + // When found, cache indexes on `parent` and break + if ( node.nodeType === 1 && ++diff && node === elem ) { + uniqueCache[ type ] = [ dirruns, nodeIndex, diff ]; + break; + } + } + + } else { + + // Use previously-cached element index if available + if ( useCache ) { + + // ...in a gzip-friendly way + node = elem; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex; + } + + // xml :nth-child(...) + // or :nth-last-child(...) or :nth(-last)?-of-type(...) + if ( diff === false ) { + + // Use the same loop as above to seek `elem` from the start + while ( ( node = ++nodeIndex && node && node[ dir ] || + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + if ( ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) && + ++diff ) { + + // Cache the index of each encountered element + if ( useCache ) { + outerCache = node[ expando ] || + ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + uniqueCache[ type ] = [ dirruns, diff ]; + } + + if ( node === elem ) { + break; + } + } + } + } + } + + // Incorporate the offset, then check against cycle size + diff -= last; + return diff === first || ( diff % first === 0 && diff / first >= 0 ); + } + }; + }, + + "PSEUDO": function( pseudo, argument ) { + + // pseudo-class names are case-insensitive + // http://www.w3.org/TR/selectors/#pseudo-classes + // Prioritize by case sensitivity in case custom pseudos are added with uppercase letters + // Remember that setFilters inherits from pseudos + var args, + fn = Expr.pseudos[ pseudo ] || Expr.setFilters[ pseudo.toLowerCase() ] || + Sizzle.error( "unsupported pseudo: " + pseudo ); + + // The user may use createPseudo to indicate that + // arguments are needed to create the filter function + // just as Sizzle does + if ( fn[ expando ] ) { + return fn( argument ); + } + + // But maintain support for old signatures + if ( fn.length > 1 ) { + args = [ pseudo, pseudo, "", argument ]; + return Expr.setFilters.hasOwnProperty( pseudo.toLowerCase() ) ? + markFunction( function( seed, matches ) { + var idx, + matched = fn( seed, argument ), + i = matched.length; + while ( i-- ) { + idx = indexOf( seed, matched[ i ] ); + seed[ idx ] = !( matches[ idx ] = matched[ i ] ); + } + } ) : + function( elem ) { + return fn( elem, 0, args ); + }; + } + + return fn; + } + }, + + pseudos: { + + // Potentially complex pseudos + "not": markFunction( function( selector ) { + + // Trim the selector passed to compile + // to avoid treating leading and trailing + // spaces as combinators + var input = [], + results = [], + matcher = compile( selector.replace( rtrim, "$1" ) ); + + return matcher[ expando ] ? + markFunction( function( seed, matches, _context, xml ) { + var elem, + unmatched = matcher( seed, null, xml, [] ), + i = seed.length; + + // Match elements unmatched by `matcher` + while ( i-- ) { + if ( ( elem = unmatched[ i ] ) ) { + seed[ i ] = !( matches[ i ] = elem ); + } + } + } ) : + function( elem, _context, xml ) { + input[ 0 ] = elem; + matcher( input, null, xml, results ); + + // Don't keep the element (issue #299) + input[ 0 ] = null; + return !results.pop(); + }; + } ), + + "has": markFunction( function( selector ) { + return function( elem ) { + return Sizzle( selector, elem ).length > 0; + }; + } ), + + "contains": markFunction( function( text ) { + text = text.replace( runescape, funescape ); + return function( elem ) { + return ( elem.textContent || getText( elem ) ).indexOf( text ) > -1; + }; + } ), + + // "Whether an element is represented by a :lang() selector + // is based solely on the element's language value + // being equal to the identifier C, + // or beginning with the identifier C immediately followed by "-". + // The matching of C against the element's language value is performed case-insensitively. + // The identifier C does not have to be a valid language name." + // http://www.w3.org/TR/selectors/#lang-pseudo + "lang": markFunction( function( lang ) { + + // lang value must be a valid identifier + if ( !ridentifier.test( lang || "" ) ) { + Sizzle.error( "unsupported lang: " + lang ); + } + lang = lang.replace( runescape, funescape ).toLowerCase(); + return function( elem ) { + var elemLang; + do { + if ( ( elemLang = documentIsHTML ? + elem.lang : + elem.getAttribute( "xml:lang" ) || elem.getAttribute( "lang" ) ) ) { + + elemLang = elemLang.toLowerCase(); + return elemLang === lang || elemLang.indexOf( lang + "-" ) === 0; + } + } while ( ( elem = elem.parentNode ) && elem.nodeType === 1 ); + return false; + }; + } ), + + // Miscellaneous + "target": function( elem ) { + var hash = window.location && window.location.hash; + return hash && hash.slice( 1 ) === elem.id; + }, + + "root": function( elem ) { + return elem === docElem; + }, + + "focus": function( elem ) { + return elem === document.activeElement && + ( !document.hasFocus || document.hasFocus() ) && + !!( elem.type || elem.href || ~elem.tabIndex ); + }, + + // Boolean properties + "enabled": createDisabledPseudo( false ), + "disabled": createDisabledPseudo( true ), + + "checked": function( elem ) { + + // In CSS3, :checked should return both checked and selected elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + var nodeName = elem.nodeName.toLowerCase(); + return ( nodeName === "input" && !!elem.checked ) || + ( nodeName === "option" && !!elem.selected ); + }, + + "selected": function( elem ) { + + // Accessing this property makes selected-by-default + // options in Safari work properly + if ( elem.parentNode ) { + // eslint-disable-next-line no-unused-expressions + elem.parentNode.selectedIndex; + } + + return elem.selected === true; + }, + + // Contents + "empty": function( elem ) { + + // http://www.w3.org/TR/selectors/#empty-pseudo + // :empty is negated by element (1) or content nodes (text: 3; cdata: 4; entity ref: 5), + // but not by others (comment: 8; processing instruction: 7; etc.) + // nodeType < 6 works because attributes (2) do not appear as children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + if ( elem.nodeType < 6 ) { + return false; + } + } + return true; + }, + + "parent": function( elem ) { + return !Expr.pseudos[ "empty" ]( elem ); + }, + + // Element/input types + "header": function( elem ) { + return rheader.test( elem.nodeName ); + }, + + "input": function( elem ) { + return rinputs.test( elem.nodeName ); + }, + + "button": function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === "button" || name === "button"; + }, + + "text": function( elem ) { + var attr; + return elem.nodeName.toLowerCase() === "input" && + elem.type === "text" && + + // Support: IE<8 + // New HTML5 attribute values (e.g., "search") appear with elem.type === "text" + ( ( attr = elem.getAttribute( "type" ) ) == null || + attr.toLowerCase() === "text" ); + }, + + // Position-in-collection + "first": createPositionalPseudo( function() { + return [ 0 ]; + } ), + + "last": createPositionalPseudo( function( _matchIndexes, length ) { + return [ length - 1 ]; + } ), + + "eq": createPositionalPseudo( function( _matchIndexes, length, argument ) { + return [ argument < 0 ? argument + length : argument ]; + } ), + + "even": createPositionalPseudo( function( matchIndexes, length ) { + var i = 0; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "odd": createPositionalPseudo( function( matchIndexes, length ) { + var i = 1; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "lt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? + argument + length : + argument > length ? + length : + argument; + for ( ; --i >= 0; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "gt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? argument + length : argument; + for ( ; ++i < length; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ) + } +}; + +Expr.pseudos[ "nth" ] = Expr.pseudos[ "eq" ]; + +// Add button/input type pseudos +for ( i in { radio: true, checkbox: true, file: true, password: true, image: true } ) { + Expr.pseudos[ i ] = createInputPseudo( i ); +} +for ( i in { submit: true, reset: true } ) { + Expr.pseudos[ i ] = createButtonPseudo( i ); +} + +// Easy API for creating new setFilters +function setFilters() {} +setFilters.prototype = Expr.filters = Expr.pseudos; +Expr.setFilters = new setFilters(); + +tokenize = Sizzle.tokenize = function( selector, parseOnly ) { + var matched, match, tokens, type, + soFar, groups, preFilters, + cached = tokenCache[ selector + " " ]; + + if ( cached ) { + return parseOnly ? 0 : cached.slice( 0 ); + } + + soFar = selector; + groups = []; + preFilters = Expr.preFilter; + + while ( soFar ) { + + // Comma and first run + if ( !matched || ( match = rcomma.exec( soFar ) ) ) { + if ( match ) { + + // Don't consume trailing commas as valid + soFar = soFar.slice( match[ 0 ].length ) || soFar; + } + groups.push( ( tokens = [] ) ); + } + + matched = false; + + // Combinators + if ( ( match = rcombinators.exec( soFar ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + + // Cast descendant combinators to space + type: match[ 0 ].replace( rtrim, " " ) + } ); + soFar = soFar.slice( matched.length ); + } + + // Filters + for ( type in Expr.filter ) { + if ( ( match = matchExpr[ type ].exec( soFar ) ) && ( !preFilters[ type ] || + ( match = preFilters[ type ]( match ) ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + type: type, + matches: match + } ); + soFar = soFar.slice( matched.length ); + } + } + + if ( !matched ) { + break; + } + } + + // Return the length of the invalid excess + // if we're just parsing + // Otherwise, throw an error or return tokens + return parseOnly ? + soFar.length : + soFar ? + Sizzle.error( selector ) : + + // Cache the tokens + tokenCache( selector, groups ).slice( 0 ); +}; + +function toSelector( tokens ) { + var i = 0, + len = tokens.length, + selector = ""; + for ( ; i < len; i++ ) { + selector += tokens[ i ].value; + } + return selector; +} + +function addCombinator( matcher, combinator, base ) { + var dir = combinator.dir, + skip = combinator.next, + key = skip || dir, + checkNonElements = base && key === "parentNode", + doneName = done++; + + return combinator.first ? + + // Check against closest ancestor/preceding element + function( elem, context, xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + return matcher( elem, context, xml ); + } + } + return false; + } : + + // Check against all ancestor/preceding elements + function( elem, context, xml ) { + var oldCache, uniqueCache, outerCache, + newCache = [ dirruns, doneName ]; + + // We can't set arbitrary data on XML nodes, so they don't benefit from combinator caching + if ( xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + if ( matcher( elem, context, xml ) ) { + return true; + } + } + } + } else { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + outerCache = elem[ expando ] || ( elem[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ elem.uniqueID ] || + ( outerCache[ elem.uniqueID ] = {} ); + + if ( skip && skip === elem.nodeName.toLowerCase() ) { + elem = elem[ dir ] || elem; + } else if ( ( oldCache = uniqueCache[ key ] ) && + oldCache[ 0 ] === dirruns && oldCache[ 1 ] === doneName ) { + + // Assign to newCache so results back-propagate to previous elements + return ( newCache[ 2 ] = oldCache[ 2 ] ); + } else { + + // Reuse newcache so results back-propagate to previous elements + uniqueCache[ key ] = newCache; + + // A match means we're done; a fail means we have to keep checking + if ( ( newCache[ 2 ] = matcher( elem, context, xml ) ) ) { + return true; + } + } + } + } + } + return false; + }; +} + +function elementMatcher( matchers ) { + return matchers.length > 1 ? + function( elem, context, xml ) { + var i = matchers.length; + while ( i-- ) { + if ( !matchers[ i ]( elem, context, xml ) ) { + return false; + } + } + return true; + } : + matchers[ 0 ]; +} + +function multipleContexts( selector, contexts, results ) { + var i = 0, + len = contexts.length; + for ( ; i < len; i++ ) { + Sizzle( selector, contexts[ i ], results ); + } + return results; +} + +function condense( unmatched, map, filter, context, xml ) { + var elem, + newUnmatched = [], + i = 0, + len = unmatched.length, + mapped = map != null; + + for ( ; i < len; i++ ) { + if ( ( elem = unmatched[ i ] ) ) { + if ( !filter || filter( elem, context, xml ) ) { + newUnmatched.push( elem ); + if ( mapped ) { + map.push( i ); + } + } + } + } + + return newUnmatched; +} + +function setMatcher( preFilter, selector, matcher, postFilter, postFinder, postSelector ) { + if ( postFilter && !postFilter[ expando ] ) { + postFilter = setMatcher( postFilter ); + } + if ( postFinder && !postFinder[ expando ] ) { + postFinder = setMatcher( postFinder, postSelector ); + } + return markFunction( function( seed, results, context, xml ) { + var temp, i, elem, + preMap = [], + postMap = [], + preexisting = results.length, + + // Get initial elements from seed or context + elems = seed || multipleContexts( + selector || "*", + context.nodeType ? [ context ] : context, + [] + ), + + // Prefilter to get matcher input, preserving a map for seed-results synchronization + matcherIn = preFilter && ( seed || !selector ) ? + condense( elems, preMap, preFilter, context, xml ) : + elems, + + matcherOut = matcher ? + + // If we have a postFinder, or filtered seed, or non-seed postFilter or preexisting results, + postFinder || ( seed ? preFilter : preexisting || postFilter ) ? + + // ...intermediate processing is necessary + [] : + + // ...otherwise use results directly + results : + matcherIn; + + // Find primary matches + if ( matcher ) { + matcher( matcherIn, matcherOut, context, xml ); + } + + // Apply postFilter + if ( postFilter ) { + temp = condense( matcherOut, postMap ); + postFilter( temp, [], context, xml ); + + // Un-match failing elements by moving them back to matcherIn + i = temp.length; + while ( i-- ) { + if ( ( elem = temp[ i ] ) ) { + matcherOut[ postMap[ i ] ] = !( matcherIn[ postMap[ i ] ] = elem ); + } + } + } + + if ( seed ) { + if ( postFinder || preFilter ) { + if ( postFinder ) { + + // Get the final matcherOut by condensing this intermediate into postFinder contexts + temp = []; + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) ) { + + // Restore matcherIn since elem is not yet a final match + temp.push( ( matcherIn[ i ] = elem ) ); + } + } + postFinder( null, ( matcherOut = [] ), temp, xml ); + } + + // Move matched elements from seed to results to keep them synchronized + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) && + ( temp = postFinder ? indexOf( seed, elem ) : preMap[ i ] ) > -1 ) { + + seed[ temp ] = !( results[ temp ] = elem ); + } + } + } + + // Add elements to results, through postFinder if defined + } else { + matcherOut = condense( + matcherOut === results ? + matcherOut.splice( preexisting, matcherOut.length ) : + matcherOut + ); + if ( postFinder ) { + postFinder( null, results, matcherOut, xml ); + } else { + push.apply( results, matcherOut ); + } + } + } ); +} + +function matcherFromTokens( tokens ) { + var checkContext, matcher, j, + len = tokens.length, + leadingRelative = Expr.relative[ tokens[ 0 ].type ], + implicitRelative = leadingRelative || Expr.relative[ " " ], + i = leadingRelative ? 1 : 0, + + // The foundational matcher ensures that elements are reachable from top-level context(s) + matchContext = addCombinator( function( elem ) { + return elem === checkContext; + }, implicitRelative, true ), + matchAnyContext = addCombinator( function( elem ) { + return indexOf( checkContext, elem ) > -1; + }, implicitRelative, true ), + matchers = [ function( elem, context, xml ) { + var ret = ( !leadingRelative && ( xml || context !== outermostContext ) ) || ( + ( checkContext = context ).nodeType ? + matchContext( elem, context, xml ) : + matchAnyContext( elem, context, xml ) ); + + // Avoid hanging onto element (issue #299) + checkContext = null; + return ret; + } ]; + + for ( ; i < len; i++ ) { + if ( ( matcher = Expr.relative[ tokens[ i ].type ] ) ) { + matchers = [ addCombinator( elementMatcher( matchers ), matcher ) ]; + } else { + matcher = Expr.filter[ tokens[ i ].type ].apply( null, tokens[ i ].matches ); + + // Return special upon seeing a positional matcher + if ( matcher[ expando ] ) { + + // Find the next relative operator (if any) for proper handling + j = ++i; + for ( ; j < len; j++ ) { + if ( Expr.relative[ tokens[ j ].type ] ) { + break; + } + } + return setMatcher( + i > 1 && elementMatcher( matchers ), + i > 1 && toSelector( + + // If the preceding token was a descendant combinator, insert an implicit any-element `*` + tokens + .slice( 0, i - 1 ) + .concat( { value: tokens[ i - 2 ].type === " " ? "*" : "" } ) + ).replace( rtrim, "$1" ), + matcher, + i < j && matcherFromTokens( tokens.slice( i, j ) ), + j < len && matcherFromTokens( ( tokens = tokens.slice( j ) ) ), + j < len && toSelector( tokens ) + ); + } + matchers.push( matcher ); + } + } + + return elementMatcher( matchers ); +} + +function matcherFromGroupMatchers( elementMatchers, setMatchers ) { + var bySet = setMatchers.length > 0, + byElement = elementMatchers.length > 0, + superMatcher = function( seed, context, xml, results, outermost ) { + var elem, j, matcher, + matchedCount = 0, + i = "0", + unmatched = seed && [], + setMatched = [], + contextBackup = outermostContext, + + // We must always have either seed elements or outermost context + elems = seed || byElement && Expr.find[ "TAG" ]( "*", outermost ), + + // Use integer dirruns iff this is the outermost matcher + dirrunsUnique = ( dirruns += contextBackup == null ? 1 : Math.random() || 0.1 ), + len = elems.length; + + if ( outermost ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + outermostContext = context == document || context || outermost; + } + + // Add elements passing elementMatchers directly to results + // Support: IE<9, Safari + // Tolerate NodeList properties (IE: "length"; Safari: ) matching elements by id + for ( ; i !== len && ( elem = elems[ i ] ) != null; i++ ) { + if ( byElement && elem ) { + j = 0; + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( !context && elem.ownerDocument != document ) { + setDocument( elem ); + xml = !documentIsHTML; + } + while ( ( matcher = elementMatchers[ j++ ] ) ) { + if ( matcher( elem, context || document, xml ) ) { + results.push( elem ); + break; + } + } + if ( outermost ) { + dirruns = dirrunsUnique; + } + } + + // Track unmatched elements for set filters + if ( bySet ) { + + // They will have gone through all possible matchers + if ( ( elem = !matcher && elem ) ) { + matchedCount--; + } + + // Lengthen the array for every element, matched or not + if ( seed ) { + unmatched.push( elem ); + } + } + } + + // `i` is now the count of elements visited above, and adding it to `matchedCount` + // makes the latter nonnegative. + matchedCount += i; + + // Apply set filters to unmatched elements + // NOTE: This can be skipped if there are no unmatched elements (i.e., `matchedCount` + // equals `i`), unless we didn't visit _any_ elements in the above loop because we have + // no element matchers and no seed. + // Incrementing an initially-string "0" `i` allows `i` to remain a string only in that + // case, which will result in a "00" `matchedCount` that differs from `i` but is also + // numerically zero. + if ( bySet && i !== matchedCount ) { + j = 0; + while ( ( matcher = setMatchers[ j++ ] ) ) { + matcher( unmatched, setMatched, context, xml ); + } + + if ( seed ) { + + // Reintegrate element matches to eliminate the need for sorting + if ( matchedCount > 0 ) { + while ( i-- ) { + if ( !( unmatched[ i ] || setMatched[ i ] ) ) { + setMatched[ i ] = pop.call( results ); + } + } + } + + // Discard index placeholder values to get only actual matches + setMatched = condense( setMatched ); + } + + // Add matches to results + push.apply( results, setMatched ); + + // Seedless set matches succeeding multiple successful matchers stipulate sorting + if ( outermost && !seed && setMatched.length > 0 && + ( matchedCount + setMatchers.length ) > 1 ) { + + Sizzle.uniqueSort( results ); + } + } + + // Override manipulation of globals by nested matchers + if ( outermost ) { + dirruns = dirrunsUnique; + outermostContext = contextBackup; + } + + return unmatched; + }; + + return bySet ? + markFunction( superMatcher ) : + superMatcher; +} + +compile = Sizzle.compile = function( selector, match /* Internal Use Only */ ) { + var i, + setMatchers = [], + elementMatchers = [], + cached = compilerCache[ selector + " " ]; + + if ( !cached ) { + + // Generate a function of recursive functions that can be used to check each element + if ( !match ) { + match = tokenize( selector ); + } + i = match.length; + while ( i-- ) { + cached = matcherFromTokens( match[ i ] ); + if ( cached[ expando ] ) { + setMatchers.push( cached ); + } else { + elementMatchers.push( cached ); + } + } + + // Cache the compiled function + cached = compilerCache( + selector, + matcherFromGroupMatchers( elementMatchers, setMatchers ) + ); + + // Save selector and tokenization + cached.selector = selector; + } + return cached; +}; + +/** + * A low-level selection function that works with Sizzle's compiled + * selector functions + * @param {String|Function} selector A selector or a pre-compiled + * selector function built with Sizzle.compile + * @param {Element} context + * @param {Array} [results] + * @param {Array} [seed] A set of elements to match against + */ +select = Sizzle.select = function( selector, context, results, seed ) { + var i, tokens, token, type, find, + compiled = typeof selector === "function" && selector, + match = !seed && tokenize( ( selector = compiled.selector || selector ) ); + + results = results || []; + + // Try to minimize operations if there is only one selector in the list and no seed + // (the latter of which guarantees us context) + if ( match.length === 1 ) { + + // Reduce context if the leading compound selector is an ID + tokens = match[ 0 ] = match[ 0 ].slice( 0 ); + if ( tokens.length > 2 && ( token = tokens[ 0 ] ).type === "ID" && + context.nodeType === 9 && documentIsHTML && Expr.relative[ tokens[ 1 ].type ] ) { + + context = ( Expr.find[ "ID" ]( token.matches[ 0 ] + .replace( runescape, funescape ), context ) || [] )[ 0 ]; + if ( !context ) { + return results; + + // Precompiled matchers will still verify ancestry, so step up a level + } else if ( compiled ) { + context = context.parentNode; + } + + selector = selector.slice( tokens.shift().value.length ); + } + + // Fetch a seed set for right-to-left matching + i = matchExpr[ "needsContext" ].test( selector ) ? 0 : tokens.length; + while ( i-- ) { + token = tokens[ i ]; + + // Abort if we hit a combinator + if ( Expr.relative[ ( type = token.type ) ] ) { + break; + } + if ( ( find = Expr.find[ type ] ) ) { + + // Search, expanding context for leading sibling combinators + if ( ( seed = find( + token.matches[ 0 ].replace( runescape, funescape ), + rsibling.test( tokens[ 0 ].type ) && testContext( context.parentNode ) || + context + ) ) ) { + + // If seed is empty or no tokens remain, we can return early + tokens.splice( i, 1 ); + selector = seed.length && toSelector( tokens ); + if ( !selector ) { + push.apply( results, seed ); + return results; + } + + break; + } + } + } + } + + // Compile and execute a filtering function if one is not provided + // Provide `match` to avoid retokenization if we modified the selector above + ( compiled || compile( selector, match ) )( + seed, + context, + !documentIsHTML, + results, + !context || rsibling.test( selector ) && testContext( context.parentNode ) || context + ); + return results; +}; + +// One-time assignments + +// Sort stability +support.sortStable = expando.split( "" ).sort( sortOrder ).join( "" ) === expando; + +// Support: Chrome 14-35+ +// Always assume duplicates if they aren't passed to the comparison function +support.detectDuplicates = !!hasDuplicate; + +// Initialize against the default document +setDocument(); + +// Support: Webkit<537.32 - Safari 6.0.3/Chrome 25 (fixed in Chrome 27) +// Detached nodes confoundingly follow *each other* +support.sortDetached = assert( function( el ) { + + // Should return 1, but returns 4 (following) + return el.compareDocumentPosition( document.createElement( "fieldset" ) ) & 1; +} ); + +// Support: IE<8 +// Prevent attribute/property "interpolation" +// https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx +if ( !assert( function( el ) { + el.innerHTML = ""; + return el.firstChild.getAttribute( "href" ) === "#"; +} ) ) { + addHandle( "type|href|height|width", function( elem, name, isXML ) { + if ( !isXML ) { + return elem.getAttribute( name, name.toLowerCase() === "type" ? 1 : 2 ); + } + } ); +} + +// Support: IE<9 +// Use defaultValue in place of getAttribute("value") +if ( !support.attributes || !assert( function( el ) { + el.innerHTML = ""; + el.firstChild.setAttribute( "value", "" ); + return el.firstChild.getAttribute( "value" ) === ""; +} ) ) { + addHandle( "value", function( elem, _name, isXML ) { + if ( !isXML && elem.nodeName.toLowerCase() === "input" ) { + return elem.defaultValue; + } + } ); +} + +// Support: IE<9 +// Use getAttributeNode to fetch booleans when getAttribute lies +if ( !assert( function( el ) { + return el.getAttribute( "disabled" ) == null; +} ) ) { + addHandle( booleans, function( elem, name, isXML ) { + var val; + if ( !isXML ) { + return elem[ name ] === true ? name.toLowerCase() : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; + } + } ); +} + +return Sizzle; + +} )( window ); + + + +jQuery.find = Sizzle; +jQuery.expr = Sizzle.selectors; + +// Deprecated +jQuery.expr[ ":" ] = jQuery.expr.pseudos; +jQuery.uniqueSort = jQuery.unique = Sizzle.uniqueSort; +jQuery.text = Sizzle.getText; +jQuery.isXMLDoc = Sizzle.isXML; +jQuery.contains = Sizzle.contains; +jQuery.escapeSelector = Sizzle.escape; + + + + +var dir = function( elem, dir, until ) { + var matched = [], + truncate = until !== undefined; + + while ( ( elem = elem[ dir ] ) && elem.nodeType !== 9 ) { + if ( elem.nodeType === 1 ) { + if ( truncate && jQuery( elem ).is( until ) ) { + break; + } + matched.push( elem ); + } + } + return matched; +}; + + +var siblings = function( n, elem ) { + var matched = []; + + for ( ; n; n = n.nextSibling ) { + if ( n.nodeType === 1 && n !== elem ) { + matched.push( n ); + } + } + + return matched; +}; + + +var rneedsContext = jQuery.expr.match.needsContext; + + + +function nodeName( elem, name ) { + + return elem.nodeName && elem.nodeName.toLowerCase() === name.toLowerCase(); + +} +var rsingleTag = ( /^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i ); + + + +// Implement the identical functionality for filter and not +function winnow( elements, qualifier, not ) { + if ( isFunction( qualifier ) ) { + return jQuery.grep( elements, function( elem, i ) { + return !!qualifier.call( elem, i, elem ) !== not; + } ); + } + + // Single element + if ( qualifier.nodeType ) { + return jQuery.grep( elements, function( elem ) { + return ( elem === qualifier ) !== not; + } ); + } + + // Arraylike of elements (jQuery, arguments, Array) + if ( typeof qualifier !== "string" ) { + return jQuery.grep( elements, function( elem ) { + return ( indexOf.call( qualifier, elem ) > -1 ) !== not; + } ); + } + + // Filtered directly for both simple and complex selectors + return jQuery.filter( qualifier, elements, not ); +} + +jQuery.filter = function( expr, elems, not ) { + var elem = elems[ 0 ]; + + if ( not ) { + expr = ":not(" + expr + ")"; + } + + if ( elems.length === 1 && elem.nodeType === 1 ) { + return jQuery.find.matchesSelector( elem, expr ) ? [ elem ] : []; + } + + return jQuery.find.matches( expr, jQuery.grep( elems, function( elem ) { + return elem.nodeType === 1; + } ) ); +}; + +jQuery.fn.extend( { + find: function( selector ) { + var i, ret, + len = this.length, + self = this; + + if ( typeof selector !== "string" ) { + return this.pushStack( jQuery( selector ).filter( function() { + for ( i = 0; i < len; i++ ) { + if ( jQuery.contains( self[ i ], this ) ) { + return true; + } + } + } ) ); + } + + ret = this.pushStack( [] ); + + for ( i = 0; i < len; i++ ) { + jQuery.find( selector, self[ i ], ret ); + } + + return len > 1 ? jQuery.uniqueSort( ret ) : ret; + }, + filter: function( selector ) { + return this.pushStack( winnow( this, selector || [], false ) ); + }, + not: function( selector ) { + return this.pushStack( winnow( this, selector || [], true ) ); + }, + is: function( selector ) { + return !!winnow( + this, + + // If this is a positional/relative selector, check membership in the returned set + // so $("p:first").is("p:last") won't return true for a doc with two "p". + typeof selector === "string" && rneedsContext.test( selector ) ? + jQuery( selector ) : + selector || [], + false + ).length; + } +} ); + + +// Initialize a jQuery object + + +// A central reference to the root jQuery(document) +var rootjQuery, + + // A simple way to check for HTML strings + // Prioritize #id over to avoid XSS via location.hash (#9521) + // Strict HTML recognition (#11290: must start with <) + // Shortcut simple #id case for speed + rquickExpr = /^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/, + + init = jQuery.fn.init = function( selector, context, root ) { + var match, elem; + + // HANDLE: $(""), $(null), $(undefined), $(false) + if ( !selector ) { + return this; + } + + // Method init() accepts an alternate rootjQuery + // so migrate can support jQuery.sub (gh-2101) + root = root || rootjQuery; + + // Handle HTML strings + if ( typeof selector === "string" ) { + if ( selector[ 0 ] === "<" && + selector[ selector.length - 1 ] === ">" && + selector.length >= 3 ) { + + // Assume that strings that start and end with <> are HTML and skip the regex check + match = [ null, selector, null ]; + + } else { + match = rquickExpr.exec( selector ); + } + + // Match html or make sure no context is specified for #id + if ( match && ( match[ 1 ] || !context ) ) { + + // HANDLE: $(html) -> $(array) + if ( match[ 1 ] ) { + context = context instanceof jQuery ? context[ 0 ] : context; + + // Option to run scripts is true for back-compat + // Intentionally let the error be thrown if parseHTML is not present + jQuery.merge( this, jQuery.parseHTML( + match[ 1 ], + context && context.nodeType ? context.ownerDocument || context : document, + true + ) ); + + // HANDLE: $(html, props) + if ( rsingleTag.test( match[ 1 ] ) && jQuery.isPlainObject( context ) ) { + for ( match in context ) { + + // Properties of context are called as methods if possible + if ( isFunction( this[ match ] ) ) { + this[ match ]( context[ match ] ); + + // ...and otherwise set as attributes + } else { + this.attr( match, context[ match ] ); + } + } + } + + return this; + + // HANDLE: $(#id) + } else { + elem = document.getElementById( match[ 2 ] ); + + if ( elem ) { + + // Inject the element directly into the jQuery object + this[ 0 ] = elem; + this.length = 1; + } + return this; + } + + // HANDLE: $(expr, $(...)) + } else if ( !context || context.jquery ) { + return ( context || root ).find( selector ); + + // HANDLE: $(expr, context) + // (which is just equivalent to: $(context).find(expr) + } else { + return this.constructor( context ).find( selector ); + } + + // HANDLE: $(DOMElement) + } else if ( selector.nodeType ) { + this[ 0 ] = selector; + this.length = 1; + return this; + + // HANDLE: $(function) + // Shortcut for document ready + } else if ( isFunction( selector ) ) { + return root.ready !== undefined ? + root.ready( selector ) : + + // Execute immediately if ready is not present + selector( jQuery ); + } + + return jQuery.makeArray( selector, this ); + }; + +// Give the init function the jQuery prototype for later instantiation +init.prototype = jQuery.fn; + +// Initialize central reference +rootjQuery = jQuery( document ); + + +var rparentsprev = /^(?:parents|prev(?:Until|All))/, + + // Methods guaranteed to produce a unique set when starting from a unique set + guaranteedUnique = { + children: true, + contents: true, + next: true, + prev: true + }; + +jQuery.fn.extend( { + has: function( target ) { + var targets = jQuery( target, this ), + l = targets.length; + + return this.filter( function() { + var i = 0; + for ( ; i < l; i++ ) { + if ( jQuery.contains( this, targets[ i ] ) ) { + return true; + } + } + } ); + }, + + closest: function( selectors, context ) { + var cur, + i = 0, + l = this.length, + matched = [], + targets = typeof selectors !== "string" && jQuery( selectors ); + + // Positional selectors never match, since there's no _selection_ context + if ( !rneedsContext.test( selectors ) ) { + for ( ; i < l; i++ ) { + for ( cur = this[ i ]; cur && cur !== context; cur = cur.parentNode ) { + + // Always skip document fragments + if ( cur.nodeType < 11 && ( targets ? + targets.index( cur ) > -1 : + + // Don't pass non-elements to Sizzle + cur.nodeType === 1 && + jQuery.find.matchesSelector( cur, selectors ) ) ) { + + matched.push( cur ); + break; + } + } + } + } + + return this.pushStack( matched.length > 1 ? jQuery.uniqueSort( matched ) : matched ); + }, + + // Determine the position of an element within the set + index: function( elem ) { + + // No argument, return index in parent + if ( !elem ) { + return ( this[ 0 ] && this[ 0 ].parentNode ) ? this.first().prevAll().length : -1; + } + + // Index in selector + if ( typeof elem === "string" ) { + return indexOf.call( jQuery( elem ), this[ 0 ] ); + } + + // Locate the position of the desired element + return indexOf.call( this, + + // If it receives a jQuery object, the first element is used + elem.jquery ? elem[ 0 ] : elem + ); + }, + + add: function( selector, context ) { + return this.pushStack( + jQuery.uniqueSort( + jQuery.merge( this.get(), jQuery( selector, context ) ) + ) + ); + }, + + addBack: function( selector ) { + return this.add( selector == null ? + this.prevObject : this.prevObject.filter( selector ) + ); + } +} ); + +function sibling( cur, dir ) { + while ( ( cur = cur[ dir ] ) && cur.nodeType !== 1 ) {} + return cur; +} + +jQuery.each( { + parent: function( elem ) { + var parent = elem.parentNode; + return parent && parent.nodeType !== 11 ? parent : null; + }, + parents: function( elem ) { + return dir( elem, "parentNode" ); + }, + parentsUntil: function( elem, _i, until ) { + return dir( elem, "parentNode", until ); + }, + next: function( elem ) { + return sibling( elem, "nextSibling" ); + }, + prev: function( elem ) { + return sibling( elem, "previousSibling" ); + }, + nextAll: function( elem ) { + return dir( elem, "nextSibling" ); + }, + prevAll: function( elem ) { + return dir( elem, "previousSibling" ); + }, + nextUntil: function( elem, _i, until ) { + return dir( elem, "nextSibling", until ); + }, + prevUntil: function( elem, _i, until ) { + return dir( elem, "previousSibling", until ); + }, + siblings: function( elem ) { + return siblings( ( elem.parentNode || {} ).firstChild, elem ); + }, + children: function( elem ) { + return siblings( elem.firstChild ); + }, + contents: function( elem ) { + if ( elem.contentDocument != null && + + // Support: IE 11+ + // elements with no `data` attribute has an object + // `contentDocument` with a `null` prototype. + getProto( elem.contentDocument ) ) { + + return elem.contentDocument; + } + + // Support: IE 9 - 11 only, iOS 7 only, Android Browser <=4.3 only + // Treat the template element as a regular one in browsers that + // don't support it. + if ( nodeName( elem, "template" ) ) { + elem = elem.content || elem; + } + + return jQuery.merge( [], elem.childNodes ); + } +}, function( name, fn ) { + jQuery.fn[ name ] = function( until, selector ) { + var matched = jQuery.map( this, fn, until ); + + if ( name.slice( -5 ) !== "Until" ) { + selector = until; + } + + if ( selector && typeof selector === "string" ) { + matched = jQuery.filter( selector, matched ); + } + + if ( this.length > 1 ) { + + // Remove duplicates + if ( !guaranteedUnique[ name ] ) { + jQuery.uniqueSort( matched ); + } + + // Reverse order for parents* and prev-derivatives + if ( rparentsprev.test( name ) ) { + matched.reverse(); + } + } + + return this.pushStack( matched ); + }; +} ); +var rnothtmlwhite = ( /[^\x20\t\r\n\f]+/g ); + + + +// Convert String-formatted options into Object-formatted ones +function createOptions( options ) { + var object = {}; + jQuery.each( options.match( rnothtmlwhite ) || [], function( _, flag ) { + object[ flag ] = true; + } ); + return object; +} + +/* + * Create a callback list using the following parameters: + * + * options: an optional list of space-separated options that will change how + * the callback list behaves or a more traditional option object + * + * By default a callback list will act like an event callback list and can be + * "fired" multiple times. + * + * Possible options: + * + * once: will ensure the callback list can only be fired once (like a Deferred) + * + * memory: will keep track of previous values and will call any callback added + * after the list has been fired right away with the latest "memorized" + * values (like a Deferred) + * + * unique: will ensure a callback can only be added once (no duplicate in the list) + * + * stopOnFalse: interrupt callings when a callback returns false + * + */ +jQuery.Callbacks = function( options ) { + + // Convert options from String-formatted to Object-formatted if needed + // (we check in cache first) + options = typeof options === "string" ? + createOptions( options ) : + jQuery.extend( {}, options ); + + var // Flag to know if list is currently firing + firing, + + // Last fire value for non-forgettable lists + memory, + + // Flag to know if list was already fired + fired, + + // Flag to prevent firing + locked, + + // Actual callback list + list = [], + + // Queue of execution data for repeatable lists + queue = [], + + // Index of currently firing callback (modified by add/remove as needed) + firingIndex = -1, + + // Fire callbacks + fire = function() { + + // Enforce single-firing + locked = locked || options.once; + + // Execute callbacks for all pending executions, + // respecting firingIndex overrides and runtime changes + fired = firing = true; + for ( ; queue.length; firingIndex = -1 ) { + memory = queue.shift(); + while ( ++firingIndex < list.length ) { + + // Run callback and check for early termination + if ( list[ firingIndex ].apply( memory[ 0 ], memory[ 1 ] ) === false && + options.stopOnFalse ) { + + // Jump to end and forget the data so .add doesn't re-fire + firingIndex = list.length; + memory = false; + } + } + } + + // Forget the data if we're done with it + if ( !options.memory ) { + memory = false; + } + + firing = false; + + // Clean up if we're done firing for good + if ( locked ) { + + // Keep an empty list if we have data for future add calls + if ( memory ) { + list = []; + + // Otherwise, this object is spent + } else { + list = ""; + } + } + }, + + // Actual Callbacks object + self = { + + // Add a callback or a collection of callbacks to the list + add: function() { + if ( list ) { + + // If we have memory from a past run, we should fire after adding + if ( memory && !firing ) { + firingIndex = list.length - 1; + queue.push( memory ); + } + + ( function add( args ) { + jQuery.each( args, function( _, arg ) { + if ( isFunction( arg ) ) { + if ( !options.unique || !self.has( arg ) ) { + list.push( arg ); + } + } else if ( arg && arg.length && toType( arg ) !== "string" ) { + + // Inspect recursively + add( arg ); + } + } ); + } )( arguments ); + + if ( memory && !firing ) { + fire(); + } + } + return this; + }, + + // Remove a callback from the list + remove: function() { + jQuery.each( arguments, function( _, arg ) { + var index; + while ( ( index = jQuery.inArray( arg, list, index ) ) > -1 ) { + list.splice( index, 1 ); + + // Handle firing indexes + if ( index <= firingIndex ) { + firingIndex--; + } + } + } ); + return this; + }, + + // Check if a given callback is in the list. + // If no argument is given, return whether or not list has callbacks attached. + has: function( fn ) { + return fn ? + jQuery.inArray( fn, list ) > -1 : + list.length > 0; + }, + + // Remove all callbacks from the list + empty: function() { + if ( list ) { + list = []; + } + return this; + }, + + // Disable .fire and .add + // Abort any current/pending executions + // Clear all callbacks and values + disable: function() { + locked = queue = []; + list = memory = ""; + return this; + }, + disabled: function() { + return !list; + }, + + // Disable .fire + // Also disable .add unless we have memory (since it would have no effect) + // Abort any pending executions + lock: function() { + locked = queue = []; + if ( !memory && !firing ) { + list = memory = ""; + } + return this; + }, + locked: function() { + return !!locked; + }, + + // Call all callbacks with the given context and arguments + fireWith: function( context, args ) { + if ( !locked ) { + args = args || []; + args = [ context, args.slice ? args.slice() : args ]; + queue.push( args ); + if ( !firing ) { + fire(); + } + } + return this; + }, + + // Call all the callbacks with the given arguments + fire: function() { + self.fireWith( this, arguments ); + return this; + }, + + // To know if the callbacks have already been called at least once + fired: function() { + return !!fired; + } + }; + + return self; +}; + + +function Identity( v ) { + return v; +} +function Thrower( ex ) { + throw ex; +} + +function adoptValue( value, resolve, reject, noValue ) { + var method; + + try { + + // Check for promise aspect first to privilege synchronous behavior + if ( value && isFunction( ( method = value.promise ) ) ) { + method.call( value ).done( resolve ).fail( reject ); + + // Other thenables + } else if ( value && isFunction( ( method = value.then ) ) ) { + method.call( value, resolve, reject ); + + // Other non-thenables + } else { + + // Control `resolve` arguments by letting Array#slice cast boolean `noValue` to integer: + // * false: [ value ].slice( 0 ) => resolve( value ) + // * true: [ value ].slice( 1 ) => resolve() + resolve.apply( undefined, [ value ].slice( noValue ) ); + } + + // For Promises/A+, convert exceptions into rejections + // Since jQuery.when doesn't unwrap thenables, we can skip the extra checks appearing in + // Deferred#then to conditionally suppress rejection. + } catch ( value ) { + + // Support: Android 4.0 only + // Strict mode functions invoked without .call/.apply get global-object context + reject.apply( undefined, [ value ] ); + } +} + +jQuery.extend( { + + Deferred: function( func ) { + var tuples = [ + + // action, add listener, callbacks, + // ... .then handlers, argument index, [final state] + [ "notify", "progress", jQuery.Callbacks( "memory" ), + jQuery.Callbacks( "memory" ), 2 ], + [ "resolve", "done", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 0, "resolved" ], + [ "reject", "fail", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 1, "rejected" ] + ], + state = "pending", + promise = { + state: function() { + return state; + }, + always: function() { + deferred.done( arguments ).fail( arguments ); + return this; + }, + "catch": function( fn ) { + return promise.then( null, fn ); + }, + + // Keep pipe for back-compat + pipe: function( /* fnDone, fnFail, fnProgress */ ) { + var fns = arguments; + + return jQuery.Deferred( function( newDefer ) { + jQuery.each( tuples, function( _i, tuple ) { + + // Map tuples (progress, done, fail) to arguments (done, fail, progress) + var fn = isFunction( fns[ tuple[ 4 ] ] ) && fns[ tuple[ 4 ] ]; + + // deferred.progress(function() { bind to newDefer or newDefer.notify }) + // deferred.done(function() { bind to newDefer or newDefer.resolve }) + // deferred.fail(function() { bind to newDefer or newDefer.reject }) + deferred[ tuple[ 1 ] ]( function() { + var returned = fn && fn.apply( this, arguments ); + if ( returned && isFunction( returned.promise ) ) { + returned.promise() + .progress( newDefer.notify ) + .done( newDefer.resolve ) + .fail( newDefer.reject ); + } else { + newDefer[ tuple[ 0 ] + "With" ]( + this, + fn ? [ returned ] : arguments + ); + } + } ); + } ); + fns = null; + } ).promise(); + }, + then: function( onFulfilled, onRejected, onProgress ) { + var maxDepth = 0; + function resolve( depth, deferred, handler, special ) { + return function() { + var that = this, + args = arguments, + mightThrow = function() { + var returned, then; + + // Support: Promises/A+ section 2.3.3.3.3 + // https://promisesaplus.com/#point-59 + // Ignore double-resolution attempts + if ( depth < maxDepth ) { + return; + } + + returned = handler.apply( that, args ); + + // Support: Promises/A+ section 2.3.1 + // https://promisesaplus.com/#point-48 + if ( returned === deferred.promise() ) { + throw new TypeError( "Thenable self-resolution" ); + } + + // Support: Promises/A+ sections 2.3.3.1, 3.5 + // https://promisesaplus.com/#point-54 + // https://promisesaplus.com/#point-75 + // Retrieve `then` only once + then = returned && + + // Support: Promises/A+ section 2.3.4 + // https://promisesaplus.com/#point-64 + // Only check objects and functions for thenability + ( typeof returned === "object" || + typeof returned === "function" ) && + returned.then; + + // Handle a returned thenable + if ( isFunction( then ) ) { + + // Special processors (notify) just wait for resolution + if ( special ) { + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ) + ); + + // Normal processors (resolve) also hook into progress + } else { + + // ...and disregard older resolution values + maxDepth++; + + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ), + resolve( maxDepth, deferred, Identity, + deferred.notifyWith ) + ); + } + + // Handle all other returned values + } else { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Identity ) { + that = undefined; + args = [ returned ]; + } + + // Process the value(s) + // Default process is resolve + ( special || deferred.resolveWith )( that, args ); + } + }, + + // Only normal processors (resolve) catch and reject exceptions + process = special ? + mightThrow : + function() { + try { + mightThrow(); + } catch ( e ) { + + if ( jQuery.Deferred.exceptionHook ) { + jQuery.Deferred.exceptionHook( e, + process.stackTrace ); + } + + // Support: Promises/A+ section 2.3.3.3.4.1 + // https://promisesaplus.com/#point-61 + // Ignore post-resolution exceptions + if ( depth + 1 >= maxDepth ) { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Thrower ) { + that = undefined; + args = [ e ]; + } + + deferred.rejectWith( that, args ); + } + } + }; + + // Support: Promises/A+ section 2.3.3.3.1 + // https://promisesaplus.com/#point-57 + // Re-resolve promises immediately to dodge false rejection from + // subsequent errors + if ( depth ) { + process(); + } else { + + // Call an optional hook to record the stack, in case of exception + // since it's otherwise lost when execution goes async + if ( jQuery.Deferred.getStackHook ) { + process.stackTrace = jQuery.Deferred.getStackHook(); + } + window.setTimeout( process ); + } + }; + } + + return jQuery.Deferred( function( newDefer ) { + + // progress_handlers.add( ... ) + tuples[ 0 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onProgress ) ? + onProgress : + Identity, + newDefer.notifyWith + ) + ); + + // fulfilled_handlers.add( ... ) + tuples[ 1 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onFulfilled ) ? + onFulfilled : + Identity + ) + ); + + // rejected_handlers.add( ... ) + tuples[ 2 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onRejected ) ? + onRejected : + Thrower + ) + ); + } ).promise(); + }, + + // Get a promise for this deferred + // If obj is provided, the promise aspect is added to the object + promise: function( obj ) { + return obj != null ? jQuery.extend( obj, promise ) : promise; + } + }, + deferred = {}; + + // Add list-specific methods + jQuery.each( tuples, function( i, tuple ) { + var list = tuple[ 2 ], + stateString = tuple[ 5 ]; + + // promise.progress = list.add + // promise.done = list.add + // promise.fail = list.add + promise[ tuple[ 1 ] ] = list.add; + + // Handle state + if ( stateString ) { + list.add( + function() { + + // state = "resolved" (i.e., fulfilled) + // state = "rejected" + state = stateString; + }, + + // rejected_callbacks.disable + // fulfilled_callbacks.disable + tuples[ 3 - i ][ 2 ].disable, + + // rejected_handlers.disable + // fulfilled_handlers.disable + tuples[ 3 - i ][ 3 ].disable, + + // progress_callbacks.lock + tuples[ 0 ][ 2 ].lock, + + // progress_handlers.lock + tuples[ 0 ][ 3 ].lock + ); + } + + // progress_handlers.fire + // fulfilled_handlers.fire + // rejected_handlers.fire + list.add( tuple[ 3 ].fire ); + + // deferred.notify = function() { deferred.notifyWith(...) } + // deferred.resolve = function() { deferred.resolveWith(...) } + // deferred.reject = function() { deferred.rejectWith(...) } + deferred[ tuple[ 0 ] ] = function() { + deferred[ tuple[ 0 ] + "With" ]( this === deferred ? undefined : this, arguments ); + return this; + }; + + // deferred.notifyWith = list.fireWith + // deferred.resolveWith = list.fireWith + // deferred.rejectWith = list.fireWith + deferred[ tuple[ 0 ] + "With" ] = list.fireWith; + } ); + + // Make the deferred a promise + promise.promise( deferred ); + + // Call given func if any + if ( func ) { + func.call( deferred, deferred ); + } + + // All done! + return deferred; + }, + + // Deferred helper + when: function( singleValue ) { + var + + // count of uncompleted subordinates + remaining = arguments.length, + + // count of unprocessed arguments + i = remaining, + + // subordinate fulfillment data + resolveContexts = Array( i ), + resolveValues = slice.call( arguments ), + + // the primary Deferred + primary = jQuery.Deferred(), + + // subordinate callback factory + updateFunc = function( i ) { + return function( value ) { + resolveContexts[ i ] = this; + resolveValues[ i ] = arguments.length > 1 ? slice.call( arguments ) : value; + if ( !( --remaining ) ) { + primary.resolveWith( resolveContexts, resolveValues ); + } + }; + }; + + // Single- and empty arguments are adopted like Promise.resolve + if ( remaining <= 1 ) { + adoptValue( singleValue, primary.done( updateFunc( i ) ).resolve, primary.reject, + !remaining ); + + // Use .then() to unwrap secondary thenables (cf. gh-3000) + if ( primary.state() === "pending" || + isFunction( resolveValues[ i ] && resolveValues[ i ].then ) ) { + + return primary.then(); + } + } + + // Multiple arguments are aggregated like Promise.all array elements + while ( i-- ) { + adoptValue( resolveValues[ i ], updateFunc( i ), primary.reject ); + } + + return primary.promise(); + } +} ); + + +// These usually indicate a programmer mistake during development, +// warn about them ASAP rather than swallowing them by default. +var rerrorNames = /^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/; + +jQuery.Deferred.exceptionHook = function( error, stack ) { + + // Support: IE 8 - 9 only + // Console exists when dev tools are open, which can happen at any time + if ( window.console && window.console.warn && error && rerrorNames.test( error.name ) ) { + window.console.warn( "jQuery.Deferred exception: " + error.message, error.stack, stack ); + } +}; + + + + +jQuery.readyException = function( error ) { + window.setTimeout( function() { + throw error; + } ); +}; + + + + +// The deferred used on DOM ready +var readyList = jQuery.Deferred(); + +jQuery.fn.ready = function( fn ) { + + readyList + .then( fn ) + + // Wrap jQuery.readyException in a function so that the lookup + // happens at the time of error handling instead of callback + // registration. + .catch( function( error ) { + jQuery.readyException( error ); + } ); + + return this; +}; + +jQuery.extend( { + + // Is the DOM ready to be used? Set to true once it occurs. + isReady: false, + + // A counter to track how many items to wait for before + // the ready event fires. See #6781 + readyWait: 1, + + // Handle when the DOM is ready + ready: function( wait ) { + + // Abort if there are pending holds or we're already ready + if ( wait === true ? --jQuery.readyWait : jQuery.isReady ) { + return; + } + + // Remember that the DOM is ready + jQuery.isReady = true; + + // If a normal DOM Ready event fired, decrement, and wait if need be + if ( wait !== true && --jQuery.readyWait > 0 ) { + return; + } + + // If there are functions bound, to execute + readyList.resolveWith( document, [ jQuery ] ); + } +} ); + +jQuery.ready.then = readyList.then; + +// The ready event handler and self cleanup method +function completed() { + document.removeEventListener( "DOMContentLoaded", completed ); + window.removeEventListener( "load", completed ); + jQuery.ready(); +} + +// Catch cases where $(document).ready() is called +// after the browser event has already occurred. +// Support: IE <=9 - 10 only +// Older IE sometimes signals "interactive" too soon +if ( document.readyState === "complete" || + ( document.readyState !== "loading" && !document.documentElement.doScroll ) ) { + + // Handle it asynchronously to allow scripts the opportunity to delay ready + window.setTimeout( jQuery.ready ); + +} else { + + // Use the handy event callback + document.addEventListener( "DOMContentLoaded", completed ); + + // A fallback to window.onload, that will always work + window.addEventListener( "load", completed ); +} + + + + +// Multifunctional method to get and set values of a collection +// The value/s can optionally be executed if it's a function +var access = function( elems, fn, key, value, chainable, emptyGet, raw ) { + var i = 0, + len = elems.length, + bulk = key == null; + + // Sets many values + if ( toType( key ) === "object" ) { + chainable = true; + for ( i in key ) { + access( elems, fn, i, key[ i ], true, emptyGet, raw ); + } + + // Sets one value + } else if ( value !== undefined ) { + chainable = true; + + if ( !isFunction( value ) ) { + raw = true; + } + + if ( bulk ) { + + // Bulk operations run against the entire set + if ( raw ) { + fn.call( elems, value ); + fn = null; + + // ...except when executing function values + } else { + bulk = fn; + fn = function( elem, _key, value ) { + return bulk.call( jQuery( elem ), value ); + }; + } + } + + if ( fn ) { + for ( ; i < len; i++ ) { + fn( + elems[ i ], key, raw ? + value : + value.call( elems[ i ], i, fn( elems[ i ], key ) ) + ); + } + } + } + + if ( chainable ) { + return elems; + } + + // Gets + if ( bulk ) { + return fn.call( elems ); + } + + return len ? fn( elems[ 0 ], key ) : emptyGet; +}; + + +// Matches dashed string for camelizing +var rmsPrefix = /^-ms-/, + rdashAlpha = /-([a-z])/g; + +// Used by camelCase as callback to replace() +function fcamelCase( _all, letter ) { + return letter.toUpperCase(); +} + +// Convert dashed to camelCase; used by the css and data modules +// Support: IE <=9 - 11, Edge 12 - 15 +// Microsoft forgot to hump their vendor prefix (#9572) +function camelCase( string ) { + return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase ); +} +var acceptData = function( owner ) { + + // Accepts only: + // - Node + // - Node.ELEMENT_NODE + // - Node.DOCUMENT_NODE + // - Object + // - Any + return owner.nodeType === 1 || owner.nodeType === 9 || !( +owner.nodeType ); +}; + + + + +function Data() { + this.expando = jQuery.expando + Data.uid++; +} + +Data.uid = 1; + +Data.prototype = { + + cache: function( owner ) { + + // Check if the owner object already has a cache + var value = owner[ this.expando ]; + + // If not, create one + if ( !value ) { + value = {}; + + // We can accept data for non-element nodes in modern browsers, + // but we should not, see #8335. + // Always return an empty object. + if ( acceptData( owner ) ) { + + // If it is a node unlikely to be stringify-ed or looped over + // use plain assignment + if ( owner.nodeType ) { + owner[ this.expando ] = value; + + // Otherwise secure it in a non-enumerable property + // configurable must be true to allow the property to be + // deleted when data is removed + } else { + Object.defineProperty( owner, this.expando, { + value: value, + configurable: true + } ); + } + } + } + + return value; + }, + set: function( owner, data, value ) { + var prop, + cache = this.cache( owner ); + + // Handle: [ owner, key, value ] args + // Always use camelCase key (gh-2257) + if ( typeof data === "string" ) { + cache[ camelCase( data ) ] = value; + + // Handle: [ owner, { properties } ] args + } else { + + // Copy the properties one-by-one to the cache object + for ( prop in data ) { + cache[ camelCase( prop ) ] = data[ prop ]; + } + } + return cache; + }, + get: function( owner, key ) { + return key === undefined ? + this.cache( owner ) : + + // Always use camelCase key (gh-2257) + owner[ this.expando ] && owner[ this.expando ][ camelCase( key ) ]; + }, + access: function( owner, key, value ) { + + // In cases where either: + // + // 1. No key was specified + // 2. A string key was specified, but no value provided + // + // Take the "read" path and allow the get method to determine + // which value to return, respectively either: + // + // 1. The entire cache object + // 2. The data stored at the key + // + if ( key === undefined || + ( ( key && typeof key === "string" ) && value === undefined ) ) { + + return this.get( owner, key ); + } + + // When the key is not a string, or both a key and value + // are specified, set or extend (existing objects) with either: + // + // 1. An object of properties + // 2. A key and value + // + this.set( owner, key, value ); + + // Since the "set" path can have two possible entry points + // return the expected data based on which path was taken[*] + return value !== undefined ? value : key; + }, + remove: function( owner, key ) { + var i, + cache = owner[ this.expando ]; + + if ( cache === undefined ) { + return; + } + + if ( key !== undefined ) { + + // Support array or space separated string of keys + if ( Array.isArray( key ) ) { + + // If key is an array of keys... + // We always set camelCase keys, so remove that. + key = key.map( camelCase ); + } else { + key = camelCase( key ); + + // If a key with the spaces exists, use it. + // Otherwise, create an array by matching non-whitespace + key = key in cache ? + [ key ] : + ( key.match( rnothtmlwhite ) || [] ); + } + + i = key.length; + + while ( i-- ) { + delete cache[ key[ i ] ]; + } + } + + // Remove the expando if there's no more data + if ( key === undefined || jQuery.isEmptyObject( cache ) ) { + + // Support: Chrome <=35 - 45 + // Webkit & Blink performance suffers when deleting properties + // from DOM nodes, so set to undefined instead + // https://bugs.chromium.org/p/chromium/issues/detail?id=378607 (bug restricted) + if ( owner.nodeType ) { + owner[ this.expando ] = undefined; + } else { + delete owner[ this.expando ]; + } + } + }, + hasData: function( owner ) { + var cache = owner[ this.expando ]; + return cache !== undefined && !jQuery.isEmptyObject( cache ); + } +}; +var dataPriv = new Data(); + +var dataUser = new Data(); + + + +// Implementation Summary +// +// 1. Enforce API surface and semantic compatibility with 1.9.x branch +// 2. Improve the module's maintainability by reducing the storage +// paths to a single mechanism. +// 3. Use the same single mechanism to support "private" and "user" data. +// 4. _Never_ expose "private" data to user code (TODO: Drop _data, _removeData) +// 5. Avoid exposing implementation details on user objects (eg. expando properties) +// 6. Provide a clear path for implementation upgrade to WeakMap in 2014 + +var rbrace = /^(?:\{[\w\W]*\}|\[[\w\W]*\])$/, + rmultiDash = /[A-Z]/g; + +function getData( data ) { + if ( data === "true" ) { + return true; + } + + if ( data === "false" ) { + return false; + } + + if ( data === "null" ) { + return null; + } + + // Only convert to a number if it doesn't change the string + if ( data === +data + "" ) { + return +data; + } + + if ( rbrace.test( data ) ) { + return JSON.parse( data ); + } + + return data; +} + +function dataAttr( elem, key, data ) { + var name; + + // If nothing was found internally, try to fetch any + // data from the HTML5 data-* attribute + if ( data === undefined && elem.nodeType === 1 ) { + name = "data-" + key.replace( rmultiDash, "-$&" ).toLowerCase(); + data = elem.getAttribute( name ); + + if ( typeof data === "string" ) { + try { + data = getData( data ); + } catch ( e ) {} + + // Make sure we set the data so it isn't changed later + dataUser.set( elem, key, data ); + } else { + data = undefined; + } + } + return data; +} + +jQuery.extend( { + hasData: function( elem ) { + return dataUser.hasData( elem ) || dataPriv.hasData( elem ); + }, + + data: function( elem, name, data ) { + return dataUser.access( elem, name, data ); + }, + + removeData: function( elem, name ) { + dataUser.remove( elem, name ); + }, + + // TODO: Now that all calls to _data and _removeData have been replaced + // with direct calls to dataPriv methods, these can be deprecated. + _data: function( elem, name, data ) { + return dataPriv.access( elem, name, data ); + }, + + _removeData: function( elem, name ) { + dataPriv.remove( elem, name ); + } +} ); + +jQuery.fn.extend( { + data: function( key, value ) { + var i, name, data, + elem = this[ 0 ], + attrs = elem && elem.attributes; + + // Gets all values + if ( key === undefined ) { + if ( this.length ) { + data = dataUser.get( elem ); + + if ( elem.nodeType === 1 && !dataPriv.get( elem, "hasDataAttrs" ) ) { + i = attrs.length; + while ( i-- ) { + + // Support: IE 11 only + // The attrs elements can be null (#14894) + if ( attrs[ i ] ) { + name = attrs[ i ].name; + if ( name.indexOf( "data-" ) === 0 ) { + name = camelCase( name.slice( 5 ) ); + dataAttr( elem, name, data[ name ] ); + } + } + } + dataPriv.set( elem, "hasDataAttrs", true ); + } + } + + return data; + } + + // Sets multiple values + if ( typeof key === "object" ) { + return this.each( function() { + dataUser.set( this, key ); + } ); + } + + return access( this, function( value ) { + var data; + + // The calling jQuery object (element matches) is not empty + // (and therefore has an element appears at this[ 0 ]) and the + // `value` parameter was not undefined. An empty jQuery object + // will result in `undefined` for elem = this[ 0 ] which will + // throw an exception if an attempt to read a data cache is made. + if ( elem && value === undefined ) { + + // Attempt to get data from the cache + // The key will always be camelCased in Data + data = dataUser.get( elem, key ); + if ( data !== undefined ) { + return data; + } + + // Attempt to "discover" the data in + // HTML5 custom data-* attrs + data = dataAttr( elem, key ); + if ( data !== undefined ) { + return data; + } + + // We tried really hard, but the data doesn't exist. + return; + } + + // Set the data... + this.each( function() { + + // We always store the camelCased key + dataUser.set( this, key, value ); + } ); + }, null, value, arguments.length > 1, null, true ); + }, + + removeData: function( key ) { + return this.each( function() { + dataUser.remove( this, key ); + } ); + } +} ); + + +jQuery.extend( { + queue: function( elem, type, data ) { + var queue; + + if ( elem ) { + type = ( type || "fx" ) + "queue"; + queue = dataPriv.get( elem, type ); + + // Speed up dequeue by getting out quickly if this is just a lookup + if ( data ) { + if ( !queue || Array.isArray( data ) ) { + queue = dataPriv.access( elem, type, jQuery.makeArray( data ) ); + } else { + queue.push( data ); + } + } + return queue || []; + } + }, + + dequeue: function( elem, type ) { + type = type || "fx"; + + var queue = jQuery.queue( elem, type ), + startLength = queue.length, + fn = queue.shift(), + hooks = jQuery._queueHooks( elem, type ), + next = function() { + jQuery.dequeue( elem, type ); + }; + + // If the fx queue is dequeued, always remove the progress sentinel + if ( fn === "inprogress" ) { + fn = queue.shift(); + startLength--; + } + + if ( fn ) { + + // Add a progress sentinel to prevent the fx queue from being + // automatically dequeued + if ( type === "fx" ) { + queue.unshift( "inprogress" ); + } + + // Clear up the last queue stop function + delete hooks.stop; + fn.call( elem, next, hooks ); + } + + if ( !startLength && hooks ) { + hooks.empty.fire(); + } + }, + + // Not public - generate a queueHooks object, or return the current one + _queueHooks: function( elem, type ) { + var key = type + "queueHooks"; + return dataPriv.get( elem, key ) || dataPriv.access( elem, key, { + empty: jQuery.Callbacks( "once memory" ).add( function() { + dataPriv.remove( elem, [ type + "queue", key ] ); + } ) + } ); + } +} ); + +jQuery.fn.extend( { + queue: function( type, data ) { + var setter = 2; + + if ( typeof type !== "string" ) { + data = type; + type = "fx"; + setter--; + } + + if ( arguments.length < setter ) { + return jQuery.queue( this[ 0 ], type ); + } + + return data === undefined ? + this : + this.each( function() { + var queue = jQuery.queue( this, type, data ); + + // Ensure a hooks for this queue + jQuery._queueHooks( this, type ); + + if ( type === "fx" && queue[ 0 ] !== "inprogress" ) { + jQuery.dequeue( this, type ); + } + } ); + }, + dequeue: function( type ) { + return this.each( function() { + jQuery.dequeue( this, type ); + } ); + }, + clearQueue: function( type ) { + return this.queue( type || "fx", [] ); + }, + + // Get a promise resolved when queues of a certain type + // are emptied (fx is the type by default) + promise: function( type, obj ) { + var tmp, + count = 1, + defer = jQuery.Deferred(), + elements = this, + i = this.length, + resolve = function() { + if ( !( --count ) ) { + defer.resolveWith( elements, [ elements ] ); + } + }; + + if ( typeof type !== "string" ) { + obj = type; + type = undefined; + } + type = type || "fx"; + + while ( i-- ) { + tmp = dataPriv.get( elements[ i ], type + "queueHooks" ); + if ( tmp && tmp.empty ) { + count++; + tmp.empty.add( resolve ); + } + } + resolve(); + return defer.promise( obj ); + } +} ); +var pnum = ( /[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/ ).source; + +var rcssNum = new RegExp( "^(?:([+-])=|)(" + pnum + ")([a-z%]*)$", "i" ); + + +var cssExpand = [ "Top", "Right", "Bottom", "Left" ]; + +var documentElement = document.documentElement; + + + + var isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ); + }, + composed = { composed: true }; + + // Support: IE 9 - 11+, Edge 12 - 18+, iOS 10.0 - 10.2 only + // Check attachment across shadow DOM boundaries when possible (gh-3504) + // Support: iOS 10.0-10.2 only + // Early iOS 10 versions support `attachShadow` but not `getRootNode`, + // leading to errors. We need to check for `getRootNode`. + if ( documentElement.getRootNode ) { + isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ) || + elem.getRootNode( composed ) === elem.ownerDocument; + }; + } +var isHiddenWithinTree = function( elem, el ) { + + // isHiddenWithinTree might be called from jQuery#filter function; + // in that case, element will be second argument + elem = el || elem; + + // Inline style trumps all + return elem.style.display === "none" || + elem.style.display === "" && + + // Otherwise, check computed style + // Support: Firefox <=43 - 45 + // Disconnected elements can have computed display: none, so first confirm that elem is + // in the document. + isAttached( elem ) && + + jQuery.css( elem, "display" ) === "none"; + }; + + + +function adjustCSS( elem, prop, valueParts, tween ) { + var adjusted, scale, + maxIterations = 20, + currentValue = tween ? + function() { + return tween.cur(); + } : + function() { + return jQuery.css( elem, prop, "" ); + }, + initial = currentValue(), + unit = valueParts && valueParts[ 3 ] || ( jQuery.cssNumber[ prop ] ? "" : "px" ), + + // Starting value computation is required for potential unit mismatches + initialInUnit = elem.nodeType && + ( jQuery.cssNumber[ prop ] || unit !== "px" && +initial ) && + rcssNum.exec( jQuery.css( elem, prop ) ); + + if ( initialInUnit && initialInUnit[ 3 ] !== unit ) { + + // Support: Firefox <=54 + // Halve the iteration target value to prevent interference from CSS upper bounds (gh-2144) + initial = initial / 2; + + // Trust units reported by jQuery.css + unit = unit || initialInUnit[ 3 ]; + + // Iteratively approximate from a nonzero starting point + initialInUnit = +initial || 1; + + while ( maxIterations-- ) { + + // Evaluate and update our best guess (doubling guesses that zero out). + // Finish if the scale equals or crosses 1 (making the old*new product non-positive). + jQuery.style( elem, prop, initialInUnit + unit ); + if ( ( 1 - scale ) * ( 1 - ( scale = currentValue() / initial || 0.5 ) ) <= 0 ) { + maxIterations = 0; + } + initialInUnit = initialInUnit / scale; + + } + + initialInUnit = initialInUnit * 2; + jQuery.style( elem, prop, initialInUnit + unit ); + + // Make sure we update the tween properties later on + valueParts = valueParts || []; + } + + if ( valueParts ) { + initialInUnit = +initialInUnit || +initial || 0; + + // Apply relative offset (+=/-=) if specified + adjusted = valueParts[ 1 ] ? + initialInUnit + ( valueParts[ 1 ] + 1 ) * valueParts[ 2 ] : + +valueParts[ 2 ]; + if ( tween ) { + tween.unit = unit; + tween.start = initialInUnit; + tween.end = adjusted; + } + } + return adjusted; +} + + +var defaultDisplayMap = {}; + +function getDefaultDisplay( elem ) { + var temp, + doc = elem.ownerDocument, + nodeName = elem.nodeName, + display = defaultDisplayMap[ nodeName ]; + + if ( display ) { + return display; + } + + temp = doc.body.appendChild( doc.createElement( nodeName ) ); + display = jQuery.css( temp, "display" ); + + temp.parentNode.removeChild( temp ); + + if ( display === "none" ) { + display = "block"; + } + defaultDisplayMap[ nodeName ] = display; + + return display; +} + +function showHide( elements, show ) { + var display, elem, + values = [], + index = 0, + length = elements.length; + + // Determine new display value for elements that need to change + for ( ; index < length; index++ ) { + elem = elements[ index ]; + if ( !elem.style ) { + continue; + } + + display = elem.style.display; + if ( show ) { + + // Since we force visibility upon cascade-hidden elements, an immediate (and slow) + // check is required in this first loop unless we have a nonempty display value (either + // inline or about-to-be-restored) + if ( display === "none" ) { + values[ index ] = dataPriv.get( elem, "display" ) || null; + if ( !values[ index ] ) { + elem.style.display = ""; + } + } + if ( elem.style.display === "" && isHiddenWithinTree( elem ) ) { + values[ index ] = getDefaultDisplay( elem ); + } + } else { + if ( display !== "none" ) { + values[ index ] = "none"; + + // Remember what we're overwriting + dataPriv.set( elem, "display", display ); + } + } + } + + // Set the display of the elements in a second loop to avoid constant reflow + for ( index = 0; index < length; index++ ) { + if ( values[ index ] != null ) { + elements[ index ].style.display = values[ index ]; + } + } + + return elements; +} + +jQuery.fn.extend( { + show: function() { + return showHide( this, true ); + }, + hide: function() { + return showHide( this ); + }, + toggle: function( state ) { + if ( typeof state === "boolean" ) { + return state ? this.show() : this.hide(); + } + + return this.each( function() { + if ( isHiddenWithinTree( this ) ) { + jQuery( this ).show(); + } else { + jQuery( this ).hide(); + } + } ); + } +} ); +var rcheckableType = ( /^(?:checkbox|radio)$/i ); + +var rtagName = ( /<([a-z][^\/\0>\x20\t\r\n\f]*)/i ); + +var rscriptType = ( /^$|^module$|\/(?:java|ecma)script/i ); + + + +( function() { + var fragment = document.createDocumentFragment(), + div = fragment.appendChild( document.createElement( "div" ) ), + input = document.createElement( "input" ); + + // Support: Android 4.0 - 4.3 only + // Check state lost if the name is set (#11217) + // Support: Windows Web Apps (WWA) + // `name` and `type` must use .setAttribute for WWA (#14901) + input.setAttribute( "type", "radio" ); + input.setAttribute( "checked", "checked" ); + input.setAttribute( "name", "t" ); + + div.appendChild( input ); + + // Support: Android <=4.1 only + // Older WebKit doesn't clone checked state correctly in fragments + support.checkClone = div.cloneNode( true ).cloneNode( true ).lastChild.checked; + + // Support: IE <=11 only + // Make sure textarea (and checkbox) defaultValue is properly cloned + div.innerHTML = ""; + support.noCloneChecked = !!div.cloneNode( true ).lastChild.defaultValue; + + // Support: IE <=9 only + // IE <=9 replaces "; + support.option = !!div.lastChild; +} )(); + + +// We have to close these tags to support XHTML (#13200) +var wrapMap = { + + // XHTML parsers do not magically insert elements in the + // same way that tag soup parsers do. So we cannot shorten + // this by omitting or other required elements. + thead: [ 1, "", "
" ], + col: [ 2, "", "
" ], + tr: [ 2, "", "
" ], + td: [ 3, "", "
" ], + + _default: [ 0, "", "" ] +}; + +wrapMap.tbody = wrapMap.tfoot = wrapMap.colgroup = wrapMap.caption = wrapMap.thead; +wrapMap.th = wrapMap.td; + +// Support: IE <=9 only +if ( !support.option ) { + wrapMap.optgroup = wrapMap.option = [ 1, "" ]; +} + + +function getAll( context, tag ) { + + // Support: IE <=9 - 11 only + // Use typeof to avoid zero-argument method invocation on host objects (#15151) + var ret; + + if ( typeof context.getElementsByTagName !== "undefined" ) { + ret = context.getElementsByTagName( tag || "*" ); + + } else if ( typeof context.querySelectorAll !== "undefined" ) { + ret = context.querySelectorAll( tag || "*" ); + + } else { + ret = []; + } + + if ( tag === undefined || tag && nodeName( context, tag ) ) { + return jQuery.merge( [ context ], ret ); + } + + return ret; +} + + +// Mark scripts as having already been evaluated +function setGlobalEval( elems, refElements ) { + var i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + dataPriv.set( + elems[ i ], + "globalEval", + !refElements || dataPriv.get( refElements[ i ], "globalEval" ) + ); + } +} + + +var rhtml = /<|&#?\w+;/; + +function buildFragment( elems, context, scripts, selection, ignored ) { + var elem, tmp, tag, wrap, attached, j, + fragment = context.createDocumentFragment(), + nodes = [], + i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + elem = elems[ i ]; + + if ( elem || elem === 0 ) { + + // Add nodes directly + if ( toType( elem ) === "object" ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, elem.nodeType ? [ elem ] : elem ); + + // Convert non-html into a text node + } else if ( !rhtml.test( elem ) ) { + nodes.push( context.createTextNode( elem ) ); + + // Convert html into DOM nodes + } else { + tmp = tmp || fragment.appendChild( context.createElement( "div" ) ); + + // Deserialize a standard representation + tag = ( rtagName.exec( elem ) || [ "", "" ] )[ 1 ].toLowerCase(); + wrap = wrapMap[ tag ] || wrapMap._default; + tmp.innerHTML = wrap[ 1 ] + jQuery.htmlPrefilter( elem ) + wrap[ 2 ]; + + // Descend through wrappers to the right content + j = wrap[ 0 ]; + while ( j-- ) { + tmp = tmp.lastChild; + } + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, tmp.childNodes ); + + // Remember the top-level container + tmp = fragment.firstChild; + + // Ensure the created nodes are orphaned (#12392) + tmp.textContent = ""; + } + } + } + + // Remove wrapper from fragment + fragment.textContent = ""; + + i = 0; + while ( ( elem = nodes[ i++ ] ) ) { + + // Skip elements already in the context collection (trac-4087) + if ( selection && jQuery.inArray( elem, selection ) > -1 ) { + if ( ignored ) { + ignored.push( elem ); + } + continue; + } + + attached = isAttached( elem ); + + // Append to fragment + tmp = getAll( fragment.appendChild( elem ), "script" ); + + // Preserve script evaluation history + if ( attached ) { + setGlobalEval( tmp ); + } + + // Capture executables + if ( scripts ) { + j = 0; + while ( ( elem = tmp[ j++ ] ) ) { + if ( rscriptType.test( elem.type || "" ) ) { + scripts.push( elem ); + } + } + } + } + + return fragment; +} + + +var rtypenamespace = /^([^.]*)(?:\.(.+)|)/; + +function returnTrue() { + return true; +} + +function returnFalse() { + return false; +} + +// Support: IE <=9 - 11+ +// focus() and blur() are asynchronous, except when they are no-op. +// So expect focus to be synchronous when the element is already active, +// and blur to be synchronous when the element is not already active. +// (focus and blur are always synchronous in other supported browsers, +// this just defines when we can count on it). +function expectSync( elem, type ) { + return ( elem === safeActiveElement() ) === ( type === "focus" ); +} + +// Support: IE <=9 only +// Accessing document.activeElement can throw unexpectedly +// https://bugs.jquery.com/ticket/13393 +function safeActiveElement() { + try { + return document.activeElement; + } catch ( err ) { } +} + +function on( elem, types, selector, data, fn, one ) { + var origFn, type; + + // Types can be a map of types/handlers + if ( typeof types === "object" ) { + + // ( types-Object, selector, data ) + if ( typeof selector !== "string" ) { + + // ( types-Object, data ) + data = data || selector; + selector = undefined; + } + for ( type in types ) { + on( elem, type, selector, data, types[ type ], one ); + } + return elem; + } + + if ( data == null && fn == null ) { + + // ( types, fn ) + fn = selector; + data = selector = undefined; + } else if ( fn == null ) { + if ( typeof selector === "string" ) { + + // ( types, selector, fn ) + fn = data; + data = undefined; + } else { + + // ( types, data, fn ) + fn = data; + data = selector; + selector = undefined; + } + } + if ( fn === false ) { + fn = returnFalse; + } else if ( !fn ) { + return elem; + } + + if ( one === 1 ) { + origFn = fn; + fn = function( event ) { + + // Can use an empty set, since event contains the info + jQuery().off( event ); + return origFn.apply( this, arguments ); + }; + + // Use same guid so caller can remove using origFn + fn.guid = origFn.guid || ( origFn.guid = jQuery.guid++ ); + } + return elem.each( function() { + jQuery.event.add( this, types, fn, data, selector ); + } ); +} + +/* + * Helper functions for managing events -- not part of the public interface. + * Props to Dean Edwards' addEvent library for many of the ideas. + */ +jQuery.event = { + + global: {}, + + add: function( elem, types, handler, data, selector ) { + + var handleObjIn, eventHandle, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.get( elem ); + + // Only attach events to objects that accept data + if ( !acceptData( elem ) ) { + return; + } + + // Caller can pass in an object of custom data in lieu of the handler + if ( handler.handler ) { + handleObjIn = handler; + handler = handleObjIn.handler; + selector = handleObjIn.selector; + } + + // Ensure that invalid selectors throw exceptions at attach time + // Evaluate against documentElement in case elem is a non-element node (e.g., document) + if ( selector ) { + jQuery.find.matchesSelector( documentElement, selector ); + } + + // Make sure that the handler has a unique ID, used to find/remove it later + if ( !handler.guid ) { + handler.guid = jQuery.guid++; + } + + // Init the element's event structure and main handler, if this is the first + if ( !( events = elemData.events ) ) { + events = elemData.events = Object.create( null ); + } + if ( !( eventHandle = elemData.handle ) ) { + eventHandle = elemData.handle = function( e ) { + + // Discard the second event of a jQuery.event.trigger() and + // when an event is called after a page has unloaded + return typeof jQuery !== "undefined" && jQuery.event.triggered !== e.type ? + jQuery.event.dispatch.apply( elem, arguments ) : undefined; + }; + } + + // Handle multiple events separated by a space + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // There *must* be a type, no attaching namespace-only handlers + if ( !type ) { + continue; + } + + // If event changes its type, use the special event handlers for the changed type + special = jQuery.event.special[ type ] || {}; + + // If selector defined, determine special event api type, otherwise given type + type = ( selector ? special.delegateType : special.bindType ) || type; + + // Update special based on newly reset type + special = jQuery.event.special[ type ] || {}; + + // handleObj is passed to all event handlers + handleObj = jQuery.extend( { + type: type, + origType: origType, + data: data, + handler: handler, + guid: handler.guid, + selector: selector, + needsContext: selector && jQuery.expr.match.needsContext.test( selector ), + namespace: namespaces.join( "." ) + }, handleObjIn ); + + // Init the event handler queue if we're the first + if ( !( handlers = events[ type ] ) ) { + handlers = events[ type ] = []; + handlers.delegateCount = 0; + + // Only use addEventListener if the special events handler returns false + if ( !special.setup || + special.setup.call( elem, data, namespaces, eventHandle ) === false ) { + + if ( elem.addEventListener ) { + elem.addEventListener( type, eventHandle ); + } + } + } + + if ( special.add ) { + special.add.call( elem, handleObj ); + + if ( !handleObj.handler.guid ) { + handleObj.handler.guid = handler.guid; + } + } + + // Add to the element's handler list, delegates in front + if ( selector ) { + handlers.splice( handlers.delegateCount++, 0, handleObj ); + } else { + handlers.push( handleObj ); + } + + // Keep track of which events have ever been used, for event optimization + jQuery.event.global[ type ] = true; + } + + }, + + // Detach an event or set of events from an element + remove: function( elem, types, handler, selector, mappedTypes ) { + + var j, origCount, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.hasData( elem ) && dataPriv.get( elem ); + + if ( !elemData || !( events = elemData.events ) ) { + return; + } + + // Once for each type.namespace in types; type may be omitted + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // Unbind all events (on this namespace, if provided) for the element + if ( !type ) { + for ( type in events ) { + jQuery.event.remove( elem, type + types[ t ], handler, selector, true ); + } + continue; + } + + special = jQuery.event.special[ type ] || {}; + type = ( selector ? special.delegateType : special.bindType ) || type; + handlers = events[ type ] || []; + tmp = tmp[ 2 ] && + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ); + + // Remove matching events + origCount = j = handlers.length; + while ( j-- ) { + handleObj = handlers[ j ]; + + if ( ( mappedTypes || origType === handleObj.origType ) && + ( !handler || handler.guid === handleObj.guid ) && + ( !tmp || tmp.test( handleObj.namespace ) ) && + ( !selector || selector === handleObj.selector || + selector === "**" && handleObj.selector ) ) { + handlers.splice( j, 1 ); + + if ( handleObj.selector ) { + handlers.delegateCount--; + } + if ( special.remove ) { + special.remove.call( elem, handleObj ); + } + } + } + + // Remove generic event handler if we removed something and no more handlers exist + // (avoids potential for endless recursion during removal of special event handlers) + if ( origCount && !handlers.length ) { + if ( !special.teardown || + special.teardown.call( elem, namespaces, elemData.handle ) === false ) { + + jQuery.removeEvent( elem, type, elemData.handle ); + } + + delete events[ type ]; + } + } + + // Remove data and the expando if it's no longer used + if ( jQuery.isEmptyObject( events ) ) { + dataPriv.remove( elem, "handle events" ); + } + }, + + dispatch: function( nativeEvent ) { + + var i, j, ret, matched, handleObj, handlerQueue, + args = new Array( arguments.length ), + + // Make a writable jQuery.Event from the native event object + event = jQuery.event.fix( nativeEvent ), + + handlers = ( + dataPriv.get( this, "events" ) || Object.create( null ) + )[ event.type ] || [], + special = jQuery.event.special[ event.type ] || {}; + + // Use the fix-ed jQuery.Event rather than the (read-only) native event + args[ 0 ] = event; + + for ( i = 1; i < arguments.length; i++ ) { + args[ i ] = arguments[ i ]; + } + + event.delegateTarget = this; + + // Call the preDispatch hook for the mapped type, and let it bail if desired + if ( special.preDispatch && special.preDispatch.call( this, event ) === false ) { + return; + } + + // Determine handlers + handlerQueue = jQuery.event.handlers.call( this, event, handlers ); + + // Run delegates first; they may want to stop propagation beneath us + i = 0; + while ( ( matched = handlerQueue[ i++ ] ) && !event.isPropagationStopped() ) { + event.currentTarget = matched.elem; + + j = 0; + while ( ( handleObj = matched.handlers[ j++ ] ) && + !event.isImmediatePropagationStopped() ) { + + // If the event is namespaced, then each handler is only invoked if it is + // specially universal or its namespaces are a superset of the event's. + if ( !event.rnamespace || handleObj.namespace === false || + event.rnamespace.test( handleObj.namespace ) ) { + + event.handleObj = handleObj; + event.data = handleObj.data; + + ret = ( ( jQuery.event.special[ handleObj.origType ] || {} ).handle || + handleObj.handler ).apply( matched.elem, args ); + + if ( ret !== undefined ) { + if ( ( event.result = ret ) === false ) { + event.preventDefault(); + event.stopPropagation(); + } + } + } + } + } + + // Call the postDispatch hook for the mapped type + if ( special.postDispatch ) { + special.postDispatch.call( this, event ); + } + + return event.result; + }, + + handlers: function( event, handlers ) { + var i, handleObj, sel, matchedHandlers, matchedSelectors, + handlerQueue = [], + delegateCount = handlers.delegateCount, + cur = event.target; + + // Find delegate handlers + if ( delegateCount && + + // Support: IE <=9 + // Black-hole SVG instance trees (trac-13180) + cur.nodeType && + + // Support: Firefox <=42 + // Suppress spec-violating clicks indicating a non-primary pointer button (trac-3861) + // https://www.w3.org/TR/DOM-Level-3-Events/#event-type-click + // Support: IE 11 only + // ...but not arrow key "clicks" of radio inputs, which can have `button` -1 (gh-2343) + !( event.type === "click" && event.button >= 1 ) ) { + + for ( ; cur !== this; cur = cur.parentNode || this ) { + + // Don't check non-elements (#13208) + // Don't process clicks on disabled elements (#6911, #8165, #11382, #11764) + if ( cur.nodeType === 1 && !( event.type === "click" && cur.disabled === true ) ) { + matchedHandlers = []; + matchedSelectors = {}; + for ( i = 0; i < delegateCount; i++ ) { + handleObj = handlers[ i ]; + + // Don't conflict with Object.prototype properties (#13203) + sel = handleObj.selector + " "; + + if ( matchedSelectors[ sel ] === undefined ) { + matchedSelectors[ sel ] = handleObj.needsContext ? + jQuery( sel, this ).index( cur ) > -1 : + jQuery.find( sel, this, null, [ cur ] ).length; + } + if ( matchedSelectors[ sel ] ) { + matchedHandlers.push( handleObj ); + } + } + if ( matchedHandlers.length ) { + handlerQueue.push( { elem: cur, handlers: matchedHandlers } ); + } + } + } + } + + // Add the remaining (directly-bound) handlers + cur = this; + if ( delegateCount < handlers.length ) { + handlerQueue.push( { elem: cur, handlers: handlers.slice( delegateCount ) } ); + } + + return handlerQueue; + }, + + addProp: function( name, hook ) { + Object.defineProperty( jQuery.Event.prototype, name, { + enumerable: true, + configurable: true, + + get: isFunction( hook ) ? + function() { + if ( this.originalEvent ) { + return hook( this.originalEvent ); + } + } : + function() { + if ( this.originalEvent ) { + return this.originalEvent[ name ]; + } + }, + + set: function( value ) { + Object.defineProperty( this, name, { + enumerable: true, + configurable: true, + writable: true, + value: value + } ); + } + } ); + }, + + fix: function( originalEvent ) { + return originalEvent[ jQuery.expando ] ? + originalEvent : + new jQuery.Event( originalEvent ); + }, + + special: { + load: { + + // Prevent triggered image.load events from bubbling to window.load + noBubble: true + }, + click: { + + // Utilize native event to ensure correct state for checkable inputs + setup: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Claim the first handler + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + // dataPriv.set( el, "click", ... ) + leverageNative( el, "click", returnTrue ); + } + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Force setup before triggering a click + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + leverageNative( el, "click" ); + } + + // Return non-false to allow normal event-path propagation + return true; + }, + + // For cross-browser consistency, suppress native .click() on links + // Also prevent it if we're currently inside a leveraged native-event stack + _default: function( event ) { + var target = event.target; + return rcheckableType.test( target.type ) && + target.click && nodeName( target, "input" ) && + dataPriv.get( target, "click" ) || + nodeName( target, "a" ); + } + }, + + beforeunload: { + postDispatch: function( event ) { + + // Support: Firefox 20+ + // Firefox doesn't alert if the returnValue field is not set. + if ( event.result !== undefined && event.originalEvent ) { + event.originalEvent.returnValue = event.result; + } + } + } + } +}; + +// Ensure the presence of an event listener that handles manually-triggered +// synthetic events by interrupting progress until reinvoked in response to +// *native* events that it fires directly, ensuring that state changes have +// already occurred before other listeners are invoked. +function leverageNative( el, type, expectSync ) { + + // Missing expectSync indicates a trigger call, which must force setup through jQuery.event.add + if ( !expectSync ) { + if ( dataPriv.get( el, type ) === undefined ) { + jQuery.event.add( el, type, returnTrue ); + } + return; + } + + // Register the controller as a special universal handler for all event namespaces + dataPriv.set( el, type, false ); + jQuery.event.add( el, type, { + namespace: false, + handler: function( event ) { + var notAsync, result, + saved = dataPriv.get( this, type ); + + if ( ( event.isTrigger & 1 ) && this[ type ] ) { + + // Interrupt processing of the outer synthetic .trigger()ed event + // Saved data should be false in such cases, but might be a leftover capture object + // from an async native handler (gh-4350) + if ( !saved.length ) { + + // Store arguments for use when handling the inner native event + // There will always be at least one argument (an event object), so this array + // will not be confused with a leftover capture object. + saved = slice.call( arguments ); + dataPriv.set( this, type, saved ); + + // Trigger the native event and capture its result + // Support: IE <=9 - 11+ + // focus() and blur() are asynchronous + notAsync = expectSync( this, type ); + this[ type ](); + result = dataPriv.get( this, type ); + if ( saved !== result || notAsync ) { + dataPriv.set( this, type, false ); + } else { + result = {}; + } + if ( saved !== result ) { + + // Cancel the outer synthetic event + event.stopImmediatePropagation(); + event.preventDefault(); + + // Support: Chrome 86+ + // In Chrome, if an element having a focusout handler is blurred by + // clicking outside of it, it invokes the handler synchronously. If + // that handler calls `.remove()` on the element, the data is cleared, + // leaving `result` undefined. We need to guard against this. + return result && result.value; + } + + // If this is an inner synthetic event for an event with a bubbling surrogate + // (focus or blur), assume that the surrogate already propagated from triggering the + // native event and prevent that from happening again here. + // This technically gets the ordering wrong w.r.t. to `.trigger()` (in which the + // bubbling surrogate propagates *after* the non-bubbling base), but that seems + // less bad than duplication. + } else if ( ( jQuery.event.special[ type ] || {} ).delegateType ) { + event.stopPropagation(); + } + + // If this is a native event triggered above, everything is now in order + // Fire an inner synthetic event with the original arguments + } else if ( saved.length ) { + + // ...and capture the result + dataPriv.set( this, type, { + value: jQuery.event.trigger( + + // Support: IE <=9 - 11+ + // Extend with the prototype to reset the above stopImmediatePropagation() + jQuery.extend( saved[ 0 ], jQuery.Event.prototype ), + saved.slice( 1 ), + this + ) + } ); + + // Abort handling of the native event + event.stopImmediatePropagation(); + } + } + } ); +} + +jQuery.removeEvent = function( elem, type, handle ) { + + // This "if" is needed for plain objects + if ( elem.removeEventListener ) { + elem.removeEventListener( type, handle ); + } +}; + +jQuery.Event = function( src, props ) { + + // Allow instantiation without the 'new' keyword + if ( !( this instanceof jQuery.Event ) ) { + return new jQuery.Event( src, props ); + } + + // Event object + if ( src && src.type ) { + this.originalEvent = src; + this.type = src.type; + + // Events bubbling up the document may have been marked as prevented + // by a handler lower down the tree; reflect the correct value. + this.isDefaultPrevented = src.defaultPrevented || + src.defaultPrevented === undefined && + + // Support: Android <=2.3 only + src.returnValue === false ? + returnTrue : + returnFalse; + + // Create target properties + // Support: Safari <=6 - 7 only + // Target should not be a text node (#504, #13143) + this.target = ( src.target && src.target.nodeType === 3 ) ? + src.target.parentNode : + src.target; + + this.currentTarget = src.currentTarget; + this.relatedTarget = src.relatedTarget; + + // Event type + } else { + this.type = src; + } + + // Put explicitly provided properties onto the event object + if ( props ) { + jQuery.extend( this, props ); + } + + // Create a timestamp if incoming event doesn't have one + this.timeStamp = src && src.timeStamp || Date.now(); + + // Mark it as fixed + this[ jQuery.expando ] = true; +}; + +// jQuery.Event is based on DOM3 Events as specified by the ECMAScript Language Binding +// https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html +jQuery.Event.prototype = { + constructor: jQuery.Event, + isDefaultPrevented: returnFalse, + isPropagationStopped: returnFalse, + isImmediatePropagationStopped: returnFalse, + isSimulated: false, + + preventDefault: function() { + var e = this.originalEvent; + + this.isDefaultPrevented = returnTrue; + + if ( e && !this.isSimulated ) { + e.preventDefault(); + } + }, + stopPropagation: function() { + var e = this.originalEvent; + + this.isPropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopPropagation(); + } + }, + stopImmediatePropagation: function() { + var e = this.originalEvent; + + this.isImmediatePropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopImmediatePropagation(); + } + + this.stopPropagation(); + } +}; + +// Includes all common event props including KeyEvent and MouseEvent specific props +jQuery.each( { + altKey: true, + bubbles: true, + cancelable: true, + changedTouches: true, + ctrlKey: true, + detail: true, + eventPhase: true, + metaKey: true, + pageX: true, + pageY: true, + shiftKey: true, + view: true, + "char": true, + code: true, + charCode: true, + key: true, + keyCode: true, + button: true, + buttons: true, + clientX: true, + clientY: true, + offsetX: true, + offsetY: true, + pointerId: true, + pointerType: true, + screenX: true, + screenY: true, + targetTouches: true, + toElement: true, + touches: true, + which: true +}, jQuery.event.addProp ); + +jQuery.each( { focus: "focusin", blur: "focusout" }, function( type, delegateType ) { + jQuery.event.special[ type ] = { + + // Utilize native event if possible so blur/focus sequence is correct + setup: function() { + + // Claim the first handler + // dataPriv.set( this, "focus", ... ) + // dataPriv.set( this, "blur", ... ) + leverageNative( this, type, expectSync ); + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function() { + + // Force setup before trigger + leverageNative( this, type ); + + // Return non-false to allow normal event-path propagation + return true; + }, + + // Suppress native focus or blur as it's already being fired + // in leverageNative. + _default: function() { + return true; + }, + + delegateType: delegateType + }; +} ); + +// Create mouseenter/leave events using mouseover/out and event-time checks +// so that event delegation works in jQuery. +// Do the same for pointerenter/pointerleave and pointerover/pointerout +// +// Support: Safari 7 only +// Safari sends mouseenter too often; see: +// https://bugs.chromium.org/p/chromium/issues/detail?id=470258 +// for the description of the bug (it existed in older Chrome versions as well). +jQuery.each( { + mouseenter: "mouseover", + mouseleave: "mouseout", + pointerenter: "pointerover", + pointerleave: "pointerout" +}, function( orig, fix ) { + jQuery.event.special[ orig ] = { + delegateType: fix, + bindType: fix, + + handle: function( event ) { + var ret, + target = this, + related = event.relatedTarget, + handleObj = event.handleObj; + + // For mouseenter/leave call the handler if related is outside the target. + // NB: No relatedTarget if the mouse left/entered the browser window + if ( !related || ( related !== target && !jQuery.contains( target, related ) ) ) { + event.type = handleObj.origType; + ret = handleObj.handler.apply( this, arguments ); + event.type = fix; + } + return ret; + } + }; +} ); + +jQuery.fn.extend( { + + on: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn ); + }, + one: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn, 1 ); + }, + off: function( types, selector, fn ) { + var handleObj, type; + if ( types && types.preventDefault && types.handleObj ) { + + // ( event ) dispatched jQuery.Event + handleObj = types.handleObj; + jQuery( types.delegateTarget ).off( + handleObj.namespace ? + handleObj.origType + "." + handleObj.namespace : + handleObj.origType, + handleObj.selector, + handleObj.handler + ); + return this; + } + if ( typeof types === "object" ) { + + // ( types-object [, selector] ) + for ( type in types ) { + this.off( type, selector, types[ type ] ); + } + return this; + } + if ( selector === false || typeof selector === "function" ) { + + // ( types [, fn] ) + fn = selector; + selector = undefined; + } + if ( fn === false ) { + fn = returnFalse; + } + return this.each( function() { + jQuery.event.remove( this, types, fn, selector ); + } ); + } +} ); + + +var + + // Support: IE <=10 - 11, Edge 12 - 13 only + // In IE/Edge using regex groups here causes severe slowdowns. + // See https://connect.microsoft.com/IE/feedback/details/1736512/ + rnoInnerhtml = /\s*$/g; + +// Prefer a tbody over its parent table for containing new rows +function manipulationTarget( elem, content ) { + if ( nodeName( elem, "table" ) && + nodeName( content.nodeType !== 11 ? content : content.firstChild, "tr" ) ) { + + return jQuery( elem ).children( "tbody" )[ 0 ] || elem; + } + + return elem; +} + +// Replace/restore the type attribute of script elements for safe DOM manipulation +function disableScript( elem ) { + elem.type = ( elem.getAttribute( "type" ) !== null ) + "/" + elem.type; + return elem; +} +function restoreScript( elem ) { + if ( ( elem.type || "" ).slice( 0, 5 ) === "true/" ) { + elem.type = elem.type.slice( 5 ); + } else { + elem.removeAttribute( "type" ); + } + + return elem; +} + +function cloneCopyEvent( src, dest ) { + var i, l, type, pdataOld, udataOld, udataCur, events; + + if ( dest.nodeType !== 1 ) { + return; + } + + // 1. Copy private data: events, handlers, etc. + if ( dataPriv.hasData( src ) ) { + pdataOld = dataPriv.get( src ); + events = pdataOld.events; + + if ( events ) { + dataPriv.remove( dest, "handle events" ); + + for ( type in events ) { + for ( i = 0, l = events[ type ].length; i < l; i++ ) { + jQuery.event.add( dest, type, events[ type ][ i ] ); + } + } + } + } + + // 2. Copy user data + if ( dataUser.hasData( src ) ) { + udataOld = dataUser.access( src ); + udataCur = jQuery.extend( {}, udataOld ); + + dataUser.set( dest, udataCur ); + } +} + +// Fix IE bugs, see support tests +function fixInput( src, dest ) { + var nodeName = dest.nodeName.toLowerCase(); + + // Fails to persist the checked state of a cloned checkbox or radio button. + if ( nodeName === "input" && rcheckableType.test( src.type ) ) { + dest.checked = src.checked; + + // Fails to return the selected option to the default selected state when cloning options + } else if ( nodeName === "input" || nodeName === "textarea" ) { + dest.defaultValue = src.defaultValue; + } +} + +function domManip( collection, args, callback, ignored ) { + + // Flatten any nested arrays + args = flat( args ); + + var fragment, first, scripts, hasScripts, node, doc, + i = 0, + l = collection.length, + iNoClone = l - 1, + value = args[ 0 ], + valueIsFunction = isFunction( value ); + + // We can't cloneNode fragments that contain checked, in WebKit + if ( valueIsFunction || + ( l > 1 && typeof value === "string" && + !support.checkClone && rchecked.test( value ) ) ) { + return collection.each( function( index ) { + var self = collection.eq( index ); + if ( valueIsFunction ) { + args[ 0 ] = value.call( this, index, self.html() ); + } + domManip( self, args, callback, ignored ); + } ); + } + + if ( l ) { + fragment = buildFragment( args, collection[ 0 ].ownerDocument, false, collection, ignored ); + first = fragment.firstChild; + + if ( fragment.childNodes.length === 1 ) { + fragment = first; + } + + // Require either new content or an interest in ignored elements to invoke the callback + if ( first || ignored ) { + scripts = jQuery.map( getAll( fragment, "script" ), disableScript ); + hasScripts = scripts.length; + + // Use the original fragment for the last item + // instead of the first because it can end up + // being emptied incorrectly in certain situations (#8070). + for ( ; i < l; i++ ) { + node = fragment; + + if ( i !== iNoClone ) { + node = jQuery.clone( node, true, true ); + + // Keep references to cloned scripts for later restoration + if ( hasScripts ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( scripts, getAll( node, "script" ) ); + } + } + + callback.call( collection[ i ], node, i ); + } + + if ( hasScripts ) { + doc = scripts[ scripts.length - 1 ].ownerDocument; + + // Reenable scripts + jQuery.map( scripts, restoreScript ); + + // Evaluate executable scripts on first document insertion + for ( i = 0; i < hasScripts; i++ ) { + node = scripts[ i ]; + if ( rscriptType.test( node.type || "" ) && + !dataPriv.access( node, "globalEval" ) && + jQuery.contains( doc, node ) ) { + + if ( node.src && ( node.type || "" ).toLowerCase() !== "module" ) { + + // Optional AJAX dependency, but won't run scripts if not present + if ( jQuery._evalUrl && !node.noModule ) { + jQuery._evalUrl( node.src, { + nonce: node.nonce || node.getAttribute( "nonce" ) + }, doc ); + } + } else { + DOMEval( node.textContent.replace( rcleanScript, "" ), node, doc ); + } + } + } + } + } + } + + return collection; +} + +function remove( elem, selector, keepData ) { + var node, + nodes = selector ? jQuery.filter( selector, elem ) : elem, + i = 0; + + for ( ; ( node = nodes[ i ] ) != null; i++ ) { + if ( !keepData && node.nodeType === 1 ) { + jQuery.cleanData( getAll( node ) ); + } + + if ( node.parentNode ) { + if ( keepData && isAttached( node ) ) { + setGlobalEval( getAll( node, "script" ) ); + } + node.parentNode.removeChild( node ); + } + } + + return elem; +} + +jQuery.extend( { + htmlPrefilter: function( html ) { + return html; + }, + + clone: function( elem, dataAndEvents, deepDataAndEvents ) { + var i, l, srcElements, destElements, + clone = elem.cloneNode( true ), + inPage = isAttached( elem ); + + // Fix IE cloning issues + if ( !support.noCloneChecked && ( elem.nodeType === 1 || elem.nodeType === 11 ) && + !jQuery.isXMLDoc( elem ) ) { + + // We eschew Sizzle here for performance reasons: https://jsperf.com/getall-vs-sizzle/2 + destElements = getAll( clone ); + srcElements = getAll( elem ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + fixInput( srcElements[ i ], destElements[ i ] ); + } + } + + // Copy the events from the original to the clone + if ( dataAndEvents ) { + if ( deepDataAndEvents ) { + srcElements = srcElements || getAll( elem ); + destElements = destElements || getAll( clone ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + cloneCopyEvent( srcElements[ i ], destElements[ i ] ); + } + } else { + cloneCopyEvent( elem, clone ); + } + } + + // Preserve script evaluation history + destElements = getAll( clone, "script" ); + if ( destElements.length > 0 ) { + setGlobalEval( destElements, !inPage && getAll( elem, "script" ) ); + } + + // Return the cloned set + return clone; + }, + + cleanData: function( elems ) { + var data, elem, type, + special = jQuery.event.special, + i = 0; + + for ( ; ( elem = elems[ i ] ) !== undefined; i++ ) { + if ( acceptData( elem ) ) { + if ( ( data = elem[ dataPriv.expando ] ) ) { + if ( data.events ) { + for ( type in data.events ) { + if ( special[ type ] ) { + jQuery.event.remove( elem, type ); + + // This is a shortcut to avoid jQuery.event.remove's overhead + } else { + jQuery.removeEvent( elem, type, data.handle ); + } + } + } + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataPriv.expando ] = undefined; + } + if ( elem[ dataUser.expando ] ) { + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataUser.expando ] = undefined; + } + } + } + } +} ); + +jQuery.fn.extend( { + detach: function( selector ) { + return remove( this, selector, true ); + }, + + remove: function( selector ) { + return remove( this, selector ); + }, + + text: function( value ) { + return access( this, function( value ) { + return value === undefined ? + jQuery.text( this ) : + this.empty().each( function() { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + this.textContent = value; + } + } ); + }, null, value, arguments.length ); + }, + + append: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.appendChild( elem ); + } + } ); + }, + + prepend: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.insertBefore( elem, target.firstChild ); + } + } ); + }, + + before: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this ); + } + } ); + }, + + after: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this.nextSibling ); + } + } ); + }, + + empty: function() { + var elem, + i = 0; + + for ( ; ( elem = this[ i ] ) != null; i++ ) { + if ( elem.nodeType === 1 ) { + + // Prevent memory leaks + jQuery.cleanData( getAll( elem, false ) ); + + // Remove any remaining nodes + elem.textContent = ""; + } + } + + return this; + }, + + clone: function( dataAndEvents, deepDataAndEvents ) { + dataAndEvents = dataAndEvents == null ? false : dataAndEvents; + deepDataAndEvents = deepDataAndEvents == null ? dataAndEvents : deepDataAndEvents; + + return this.map( function() { + return jQuery.clone( this, dataAndEvents, deepDataAndEvents ); + } ); + }, + + html: function( value ) { + return access( this, function( value ) { + var elem = this[ 0 ] || {}, + i = 0, + l = this.length; + + if ( value === undefined && elem.nodeType === 1 ) { + return elem.innerHTML; + } + + // See if we can take a shortcut and just use innerHTML + if ( typeof value === "string" && !rnoInnerhtml.test( value ) && + !wrapMap[ ( rtagName.exec( value ) || [ "", "" ] )[ 1 ].toLowerCase() ] ) { + + value = jQuery.htmlPrefilter( value ); + + try { + for ( ; i < l; i++ ) { + elem = this[ i ] || {}; + + // Remove element nodes and prevent memory leaks + if ( elem.nodeType === 1 ) { + jQuery.cleanData( getAll( elem, false ) ); + elem.innerHTML = value; + } + } + + elem = 0; + + // If using innerHTML throws an exception, use the fallback method + } catch ( e ) {} + } + + if ( elem ) { + this.empty().append( value ); + } + }, null, value, arguments.length ); + }, + + replaceWith: function() { + var ignored = []; + + // Make the changes, replacing each non-ignored context element with the new content + return domManip( this, arguments, function( elem ) { + var parent = this.parentNode; + + if ( jQuery.inArray( this, ignored ) < 0 ) { + jQuery.cleanData( getAll( this ) ); + if ( parent ) { + parent.replaceChild( elem, this ); + } + } + + // Force callback invocation + }, ignored ); + } +} ); + +jQuery.each( { + appendTo: "append", + prependTo: "prepend", + insertBefore: "before", + insertAfter: "after", + replaceAll: "replaceWith" +}, function( name, original ) { + jQuery.fn[ name ] = function( selector ) { + var elems, + ret = [], + insert = jQuery( selector ), + last = insert.length - 1, + i = 0; + + for ( ; i <= last; i++ ) { + elems = i === last ? this : this.clone( true ); + jQuery( insert[ i ] )[ original ]( elems ); + + // Support: Android <=4.0 only, PhantomJS 1 only + // .get() because push.apply(_, arraylike) throws on ancient WebKit + push.apply( ret, elems.get() ); + } + + return this.pushStack( ret ); + }; +} ); +var rnumnonpx = new RegExp( "^(" + pnum + ")(?!px)[a-z%]+$", "i" ); + +var getStyles = function( elem ) { + + // Support: IE <=11 only, Firefox <=30 (#15098, #14150) + // IE throws on elements created in popups + // FF meanwhile throws on frame elements through "defaultView.getComputedStyle" + var view = elem.ownerDocument.defaultView; + + if ( !view || !view.opener ) { + view = window; + } + + return view.getComputedStyle( elem ); + }; + +var swap = function( elem, options, callback ) { + var ret, name, + old = {}; + + // Remember the old values, and insert the new ones + for ( name in options ) { + old[ name ] = elem.style[ name ]; + elem.style[ name ] = options[ name ]; + } + + ret = callback.call( elem ); + + // Revert the old values + for ( name in options ) { + elem.style[ name ] = old[ name ]; + } + + return ret; +}; + + +var rboxStyle = new RegExp( cssExpand.join( "|" ), "i" ); + + + +( function() { + + // Executing both pixelPosition & boxSizingReliable tests require only one layout + // so they're executed at the same time to save the second computation. + function computeStyleTests() { + + // This is a singleton, we need to execute it only once + if ( !div ) { + return; + } + + container.style.cssText = "position:absolute;left:-11111px;width:60px;" + + "margin-top:1px;padding:0;border:0"; + div.style.cssText = + "position:relative;display:block;box-sizing:border-box;overflow:scroll;" + + "margin:auto;border:1px;padding:1px;" + + "width:60%;top:1%"; + documentElement.appendChild( container ).appendChild( div ); + + var divStyle = window.getComputedStyle( div ); + pixelPositionVal = divStyle.top !== "1%"; + + // Support: Android 4.0 - 4.3 only, Firefox <=3 - 44 + reliableMarginLeftVal = roundPixelMeasures( divStyle.marginLeft ) === 12; + + // Support: Android 4.0 - 4.3 only, Safari <=9.1 - 10.1, iOS <=7.0 - 9.3 + // Some styles come back with percentage values, even though they shouldn't + div.style.right = "60%"; + pixelBoxStylesVal = roundPixelMeasures( divStyle.right ) === 36; + + // Support: IE 9 - 11 only + // Detect misreporting of content dimensions for box-sizing:border-box elements + boxSizingReliableVal = roundPixelMeasures( divStyle.width ) === 36; + + // Support: IE 9 only + // Detect overflow:scroll screwiness (gh-3699) + // Support: Chrome <=64 + // Don't get tricked when zoom affects offsetWidth (gh-4029) + div.style.position = "absolute"; + scrollboxSizeVal = roundPixelMeasures( div.offsetWidth / 3 ) === 12; + + documentElement.removeChild( container ); + + // Nullify the div so it wouldn't be stored in the memory and + // it will also be a sign that checks already performed + div = null; + } + + function roundPixelMeasures( measure ) { + return Math.round( parseFloat( measure ) ); + } + + var pixelPositionVal, boxSizingReliableVal, scrollboxSizeVal, pixelBoxStylesVal, + reliableTrDimensionsVal, reliableMarginLeftVal, + container = document.createElement( "div" ), + div = document.createElement( "div" ); + + // Finish early in limited (non-browser) environments + if ( !div.style ) { + return; + } + + // Support: IE <=9 - 11 only + // Style of cloned element affects source element cloned (#8908) + div.style.backgroundClip = "content-box"; + div.cloneNode( true ).style.backgroundClip = ""; + support.clearCloneStyle = div.style.backgroundClip === "content-box"; + + jQuery.extend( support, { + boxSizingReliable: function() { + computeStyleTests(); + return boxSizingReliableVal; + }, + pixelBoxStyles: function() { + computeStyleTests(); + return pixelBoxStylesVal; + }, + pixelPosition: function() { + computeStyleTests(); + return pixelPositionVal; + }, + reliableMarginLeft: function() { + computeStyleTests(); + return reliableMarginLeftVal; + }, + scrollboxSize: function() { + computeStyleTests(); + return scrollboxSizeVal; + }, + + // Support: IE 9 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Behavior in IE 9 is more subtle than in newer versions & it passes + // some versions of this test; make sure not to make it pass there! + // + // Support: Firefox 70+ + // Only Firefox includes border widths + // in computed dimensions. (gh-4529) + reliableTrDimensions: function() { + var table, tr, trChild, trStyle; + if ( reliableTrDimensionsVal == null ) { + table = document.createElement( "table" ); + tr = document.createElement( "tr" ); + trChild = document.createElement( "div" ); + + table.style.cssText = "position:absolute;left:-11111px;border-collapse:separate"; + tr.style.cssText = "border:1px solid"; + + // Support: Chrome 86+ + // Height set through cssText does not get applied. + // Computed height then comes back as 0. + tr.style.height = "1px"; + trChild.style.height = "9px"; + + // Support: Android 8 Chrome 86+ + // In our bodyBackground.html iframe, + // display for all div elements is set to "inline", + // which causes a problem only in Android 8 Chrome 86. + // Ensuring the div is display: block + // gets around this issue. + trChild.style.display = "block"; + + documentElement + .appendChild( table ) + .appendChild( tr ) + .appendChild( trChild ); + + trStyle = window.getComputedStyle( tr ); + reliableTrDimensionsVal = ( parseInt( trStyle.height, 10 ) + + parseInt( trStyle.borderTopWidth, 10 ) + + parseInt( trStyle.borderBottomWidth, 10 ) ) === tr.offsetHeight; + + documentElement.removeChild( table ); + } + return reliableTrDimensionsVal; + } + } ); +} )(); + + +function curCSS( elem, name, computed ) { + var width, minWidth, maxWidth, ret, + + // Support: Firefox 51+ + // Retrieving style before computed somehow + // fixes an issue with getting wrong values + // on detached elements + style = elem.style; + + computed = computed || getStyles( elem ); + + // getPropertyValue is needed for: + // .css('filter') (IE 9 only, #12537) + // .css('--customProperty) (#3144) + if ( computed ) { + ret = computed.getPropertyValue( name ) || computed[ name ]; + + if ( ret === "" && !isAttached( elem ) ) { + ret = jQuery.style( elem, name ); + } + + // A tribute to the "awesome hack by Dean Edwards" + // Android Browser returns percentage for some values, + // but width seems to be reliably pixels. + // This is against the CSSOM draft spec: + // https://drafts.csswg.org/cssom/#resolved-values + if ( !support.pixelBoxStyles() && rnumnonpx.test( ret ) && rboxStyle.test( name ) ) { + + // Remember the original values + width = style.width; + minWidth = style.minWidth; + maxWidth = style.maxWidth; + + // Put in the new values to get a computed value out + style.minWidth = style.maxWidth = style.width = ret; + ret = computed.width; + + // Revert the changed values + style.width = width; + style.minWidth = minWidth; + style.maxWidth = maxWidth; + } + } + + return ret !== undefined ? + + // Support: IE <=9 - 11 only + // IE returns zIndex value as an integer. + ret + "" : + ret; +} + + +function addGetHookIf( conditionFn, hookFn ) { + + // Define the hook, we'll check on the first run if it's really needed. + return { + get: function() { + if ( conditionFn() ) { + + // Hook not needed (or it's not possible to use it due + // to missing dependency), remove it. + delete this.get; + return; + } + + // Hook needed; redefine it so that the support test is not executed again. + return ( this.get = hookFn ).apply( this, arguments ); + } + }; +} + + +var cssPrefixes = [ "Webkit", "Moz", "ms" ], + emptyStyle = document.createElement( "div" ).style, + vendorProps = {}; + +// Return a vendor-prefixed property or undefined +function vendorPropName( name ) { + + // Check for vendor prefixed names + var capName = name[ 0 ].toUpperCase() + name.slice( 1 ), + i = cssPrefixes.length; + + while ( i-- ) { + name = cssPrefixes[ i ] + capName; + if ( name in emptyStyle ) { + return name; + } + } +} + +// Return a potentially-mapped jQuery.cssProps or vendor prefixed property +function finalPropName( name ) { + var final = jQuery.cssProps[ name ] || vendorProps[ name ]; + + if ( final ) { + return final; + } + if ( name in emptyStyle ) { + return name; + } + return vendorProps[ name ] = vendorPropName( name ) || name; +} + + +var + + // Swappable if display is none or starts with table + // except "table", "table-cell", or "table-caption" + // See here for display values: https://developer.mozilla.org/en-US/docs/CSS/display + rdisplayswap = /^(none|table(?!-c[ea]).+)/, + rcustomProp = /^--/, + cssShow = { position: "absolute", visibility: "hidden", display: "block" }, + cssNormalTransform = { + letterSpacing: "0", + fontWeight: "400" + }; + +function setPositiveNumber( _elem, value, subtract ) { + + // Any relative (+/-) values have already been + // normalized at this point + var matches = rcssNum.exec( value ); + return matches ? + + // Guard against undefined "subtract", e.g., when used as in cssHooks + Math.max( 0, matches[ 2 ] - ( subtract || 0 ) ) + ( matches[ 3 ] || "px" ) : + value; +} + +function boxModelAdjustment( elem, dimension, box, isBorderBox, styles, computedVal ) { + var i = dimension === "width" ? 1 : 0, + extra = 0, + delta = 0; + + // Adjustment may not be necessary + if ( box === ( isBorderBox ? "border" : "content" ) ) { + return 0; + } + + for ( ; i < 4; i += 2 ) { + + // Both box models exclude margin + if ( box === "margin" ) { + delta += jQuery.css( elem, box + cssExpand[ i ], true, styles ); + } + + // If we get here with a content-box, we're seeking "padding" or "border" or "margin" + if ( !isBorderBox ) { + + // Add padding + delta += jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + + // For "border" or "margin", add border + if ( box !== "padding" ) { + delta += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + + // But still keep track of it otherwise + } else { + extra += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + + // If we get here with a border-box (content + padding + border), we're seeking "content" or + // "padding" or "margin" + } else { + + // For "content", subtract padding + if ( box === "content" ) { + delta -= jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + } + + // For "content" or "padding", subtract border + if ( box !== "margin" ) { + delta -= jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + } + } + + // Account for positive content-box scroll gutter when requested by providing computedVal + if ( !isBorderBox && computedVal >= 0 ) { + + // offsetWidth/offsetHeight is a rounded sum of content, padding, scroll gutter, and border + // Assuming integer scroll gutter, subtract the rest and round down + delta += Math.max( 0, Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + computedVal - + delta - + extra - + 0.5 + + // If offsetWidth/offsetHeight is unknown, then we can't determine content-box scroll gutter + // Use an explicit zero to avoid NaN (gh-3964) + ) ) || 0; + } + + return delta; +} + +function getWidthOrHeight( elem, dimension, extra ) { + + // Start with computed style + var styles = getStyles( elem ), + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-4322). + // Fake content-box until we know it's needed to know the true value. + boxSizingNeeded = !support.boxSizingReliable() || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + valueIsBorderBox = isBorderBox, + + val = curCSS( elem, dimension, styles ), + offsetProp = "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ); + + // Support: Firefox <=54 + // Return a confounding non-pixel value or feign ignorance, as appropriate. + if ( rnumnonpx.test( val ) ) { + if ( !extra ) { + return val; + } + val = "auto"; + } + + + // Support: IE 9 - 11 only + // Use offsetWidth/offsetHeight for when box sizing is unreliable. + // In those cases, the computed value can be trusted to be border-box. + if ( ( !support.boxSizingReliable() && isBorderBox || + + // Support: IE 10 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Interestingly, in some cases IE 9 doesn't suffer from this issue. + !support.reliableTrDimensions() && nodeName( elem, "tr" ) || + + // Fall back to offsetWidth/offsetHeight when value is "auto" + // This happens for inline elements with no explicit setting (gh-3571) + val === "auto" || + + // Support: Android <=4.1 - 4.3 only + // Also use offsetWidth/offsetHeight for misreported inline dimensions (gh-3602) + !parseFloat( val ) && jQuery.css( elem, "display", false, styles ) === "inline" ) && + + // Make sure the element is visible & connected + elem.getClientRects().length ) { + + isBorderBox = jQuery.css( elem, "boxSizing", false, styles ) === "border-box"; + + // Where available, offsetWidth/offsetHeight approximate border box dimensions. + // Where not available (e.g., SVG), assume unreliable box-sizing and interpret the + // retrieved value as a content box dimension. + valueIsBorderBox = offsetProp in elem; + if ( valueIsBorderBox ) { + val = elem[ offsetProp ]; + } + } + + // Normalize "" and auto + val = parseFloat( val ) || 0; + + // Adjust for the element's box model + return ( val + + boxModelAdjustment( + elem, + dimension, + extra || ( isBorderBox ? "border" : "content" ), + valueIsBorderBox, + styles, + + // Provide the current computed size to request scroll gutter calculation (gh-3589) + val + ) + ) + "px"; +} + +jQuery.extend( { + + // Add in style property hooks for overriding the default + // behavior of getting and setting a style property + cssHooks: { + opacity: { + get: function( elem, computed ) { + if ( computed ) { + + // We should always get a number back from opacity + var ret = curCSS( elem, "opacity" ); + return ret === "" ? "1" : ret; + } + } + } + }, + + // Don't automatically add "px" to these possibly-unitless properties + cssNumber: { + "animationIterationCount": true, + "columnCount": true, + "fillOpacity": true, + "flexGrow": true, + "flexShrink": true, + "fontWeight": true, + "gridArea": true, + "gridColumn": true, + "gridColumnEnd": true, + "gridColumnStart": true, + "gridRow": true, + "gridRowEnd": true, + "gridRowStart": true, + "lineHeight": true, + "opacity": true, + "order": true, + "orphans": true, + "widows": true, + "zIndex": true, + "zoom": true + }, + + // Add in properties whose names you wish to fix before + // setting or getting the value + cssProps: {}, + + // Get and set the style property on a DOM Node + style: function( elem, name, value, extra ) { + + // Don't set styles on text and comment nodes + if ( !elem || elem.nodeType === 3 || elem.nodeType === 8 || !elem.style ) { + return; + } + + // Make sure that we're working with the right name + var ret, type, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ), + style = elem.style; + + // Make sure that we're working with the right name. We don't + // want to query the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Gets hook for the prefixed version, then unprefixed version + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // Check if we're setting a value + if ( value !== undefined ) { + type = typeof value; + + // Convert "+=" or "-=" to relative numbers (#7345) + if ( type === "string" && ( ret = rcssNum.exec( value ) ) && ret[ 1 ] ) { + value = adjustCSS( elem, name, ret ); + + // Fixes bug #9237 + type = "number"; + } + + // Make sure that null and NaN values aren't set (#7116) + if ( value == null || value !== value ) { + return; + } + + // If a number was passed in, add the unit (except for certain CSS properties) + // The isCustomProp check can be removed in jQuery 4.0 when we only auto-append + // "px" to a few hardcoded values. + if ( type === "number" && !isCustomProp ) { + value += ret && ret[ 3 ] || ( jQuery.cssNumber[ origName ] ? "" : "px" ); + } + + // background-* props affect original clone's values + if ( !support.clearCloneStyle && value === "" && name.indexOf( "background" ) === 0 ) { + style[ name ] = "inherit"; + } + + // If a hook was provided, use that value, otherwise just set the specified value + if ( !hooks || !( "set" in hooks ) || + ( value = hooks.set( elem, value, extra ) ) !== undefined ) { + + if ( isCustomProp ) { + style.setProperty( name, value ); + } else { + style[ name ] = value; + } + } + + } else { + + // If a hook was provided get the non-computed value from there + if ( hooks && "get" in hooks && + ( ret = hooks.get( elem, false, extra ) ) !== undefined ) { + + return ret; + } + + // Otherwise just get the value from the style object + return style[ name ]; + } + }, + + css: function( elem, name, extra, styles ) { + var val, num, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ); + + // Make sure that we're working with the right name. We don't + // want to modify the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Try prefixed name followed by the unprefixed name + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // If a hook was provided get the computed value from there + if ( hooks && "get" in hooks ) { + val = hooks.get( elem, true, extra ); + } + + // Otherwise, if a way to get the computed value exists, use that + if ( val === undefined ) { + val = curCSS( elem, name, styles ); + } + + // Convert "normal" to computed value + if ( val === "normal" && name in cssNormalTransform ) { + val = cssNormalTransform[ name ]; + } + + // Make numeric if forced or a qualifier was provided and val looks numeric + if ( extra === "" || extra ) { + num = parseFloat( val ); + return extra === true || isFinite( num ) ? num || 0 : val; + } + + return val; + } +} ); + +jQuery.each( [ "height", "width" ], function( _i, dimension ) { + jQuery.cssHooks[ dimension ] = { + get: function( elem, computed, extra ) { + if ( computed ) { + + // Certain elements can have dimension info if we invisibly show them + // but it must have a current display style that would benefit + return rdisplayswap.test( jQuery.css( elem, "display" ) ) && + + // Support: Safari 8+ + // Table columns in Safari have non-zero offsetWidth & zero + // getBoundingClientRect().width unless display is changed. + // Support: IE <=11 only + // Running getBoundingClientRect on a disconnected node + // in IE throws an error. + ( !elem.getClientRects().length || !elem.getBoundingClientRect().width ) ? + swap( elem, cssShow, function() { + return getWidthOrHeight( elem, dimension, extra ); + } ) : + getWidthOrHeight( elem, dimension, extra ); + } + }, + + set: function( elem, value, extra ) { + var matches, + styles = getStyles( elem ), + + // Only read styles.position if the test has a chance to fail + // to avoid forcing a reflow. + scrollboxSizeBuggy = !support.scrollboxSize() && + styles.position === "absolute", + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-3991) + boxSizingNeeded = scrollboxSizeBuggy || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + subtract = extra ? + boxModelAdjustment( + elem, + dimension, + extra, + isBorderBox, + styles + ) : + 0; + + // Account for unreliable border-box dimensions by comparing offset* to computed and + // faking a content-box to get border and padding (gh-3699) + if ( isBorderBox && scrollboxSizeBuggy ) { + subtract -= Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + parseFloat( styles[ dimension ] ) - + boxModelAdjustment( elem, dimension, "border", false, styles ) - + 0.5 + ); + } + + // Convert to pixels if value adjustment is needed + if ( subtract && ( matches = rcssNum.exec( value ) ) && + ( matches[ 3 ] || "px" ) !== "px" ) { + + elem.style[ dimension ] = value; + value = jQuery.css( elem, dimension ); + } + + return setPositiveNumber( elem, value, subtract ); + } + }; +} ); + +jQuery.cssHooks.marginLeft = addGetHookIf( support.reliableMarginLeft, + function( elem, computed ) { + if ( computed ) { + return ( parseFloat( curCSS( elem, "marginLeft" ) ) || + elem.getBoundingClientRect().left - + swap( elem, { marginLeft: 0 }, function() { + return elem.getBoundingClientRect().left; + } ) + ) + "px"; + } + } +); + +// These hooks are used by animate to expand properties +jQuery.each( { + margin: "", + padding: "", + border: "Width" +}, function( prefix, suffix ) { + jQuery.cssHooks[ prefix + suffix ] = { + expand: function( value ) { + var i = 0, + expanded = {}, + + // Assumes a single number if not a string + parts = typeof value === "string" ? value.split( " " ) : [ value ]; + + for ( ; i < 4; i++ ) { + expanded[ prefix + cssExpand[ i ] + suffix ] = + parts[ i ] || parts[ i - 2 ] || parts[ 0 ]; + } + + return expanded; + } + }; + + if ( prefix !== "margin" ) { + jQuery.cssHooks[ prefix + suffix ].set = setPositiveNumber; + } +} ); + +jQuery.fn.extend( { + css: function( name, value ) { + return access( this, function( elem, name, value ) { + var styles, len, + map = {}, + i = 0; + + if ( Array.isArray( name ) ) { + styles = getStyles( elem ); + len = name.length; + + for ( ; i < len; i++ ) { + map[ name[ i ] ] = jQuery.css( elem, name[ i ], false, styles ); + } + + return map; + } + + return value !== undefined ? + jQuery.style( elem, name, value ) : + jQuery.css( elem, name ); + }, name, value, arguments.length > 1 ); + } +} ); + + +function Tween( elem, options, prop, end, easing ) { + return new Tween.prototype.init( elem, options, prop, end, easing ); +} +jQuery.Tween = Tween; + +Tween.prototype = { + constructor: Tween, + init: function( elem, options, prop, end, easing, unit ) { + this.elem = elem; + this.prop = prop; + this.easing = easing || jQuery.easing._default; + this.options = options; + this.start = this.now = this.cur(); + this.end = end; + this.unit = unit || ( jQuery.cssNumber[ prop ] ? "" : "px" ); + }, + cur: function() { + var hooks = Tween.propHooks[ this.prop ]; + + return hooks && hooks.get ? + hooks.get( this ) : + Tween.propHooks._default.get( this ); + }, + run: function( percent ) { + var eased, + hooks = Tween.propHooks[ this.prop ]; + + if ( this.options.duration ) { + this.pos = eased = jQuery.easing[ this.easing ]( + percent, this.options.duration * percent, 0, 1, this.options.duration + ); + } else { + this.pos = eased = percent; + } + this.now = ( this.end - this.start ) * eased + this.start; + + if ( this.options.step ) { + this.options.step.call( this.elem, this.now, this ); + } + + if ( hooks && hooks.set ) { + hooks.set( this ); + } else { + Tween.propHooks._default.set( this ); + } + return this; + } +}; + +Tween.prototype.init.prototype = Tween.prototype; + +Tween.propHooks = { + _default: { + get: function( tween ) { + var result; + + // Use a property on the element directly when it is not a DOM element, + // or when there is no matching style property that exists. + if ( tween.elem.nodeType !== 1 || + tween.elem[ tween.prop ] != null && tween.elem.style[ tween.prop ] == null ) { + return tween.elem[ tween.prop ]; + } + + // Passing an empty string as a 3rd parameter to .css will automatically + // attempt a parseFloat and fallback to a string if the parse fails. + // Simple values such as "10px" are parsed to Float; + // complex values such as "rotate(1rad)" are returned as-is. + result = jQuery.css( tween.elem, tween.prop, "" ); + + // Empty strings, null, undefined and "auto" are converted to 0. + return !result || result === "auto" ? 0 : result; + }, + set: function( tween ) { + + // Use step hook for back compat. + // Use cssHook if its there. + // Use .style if available and use plain properties where available. + if ( jQuery.fx.step[ tween.prop ] ) { + jQuery.fx.step[ tween.prop ]( tween ); + } else if ( tween.elem.nodeType === 1 && ( + jQuery.cssHooks[ tween.prop ] || + tween.elem.style[ finalPropName( tween.prop ) ] != null ) ) { + jQuery.style( tween.elem, tween.prop, tween.now + tween.unit ); + } else { + tween.elem[ tween.prop ] = tween.now; + } + } + } +}; + +// Support: IE <=9 only +// Panic based approach to setting things on disconnected nodes +Tween.propHooks.scrollTop = Tween.propHooks.scrollLeft = { + set: function( tween ) { + if ( tween.elem.nodeType && tween.elem.parentNode ) { + tween.elem[ tween.prop ] = tween.now; + } + } +}; + +jQuery.easing = { + linear: function( p ) { + return p; + }, + swing: function( p ) { + return 0.5 - Math.cos( p * Math.PI ) / 2; + }, + _default: "swing" +}; + +jQuery.fx = Tween.prototype.init; + +// Back compat <1.8 extension point +jQuery.fx.step = {}; + + + + +var + fxNow, inProgress, + rfxtypes = /^(?:toggle|show|hide)$/, + rrun = /queueHooks$/; + +function schedule() { + if ( inProgress ) { + if ( document.hidden === false && window.requestAnimationFrame ) { + window.requestAnimationFrame( schedule ); + } else { + window.setTimeout( schedule, jQuery.fx.interval ); + } + + jQuery.fx.tick(); + } +} + +// Animations created synchronously will run synchronously +function createFxNow() { + window.setTimeout( function() { + fxNow = undefined; + } ); + return ( fxNow = Date.now() ); +} + +// Generate parameters to create a standard animation +function genFx( type, includeWidth ) { + var which, + i = 0, + attrs = { height: type }; + + // If we include width, step value is 1 to do all cssExpand values, + // otherwise step value is 2 to skip over Left and Right + includeWidth = includeWidth ? 1 : 0; + for ( ; i < 4; i += 2 - includeWidth ) { + which = cssExpand[ i ]; + attrs[ "margin" + which ] = attrs[ "padding" + which ] = type; + } + + if ( includeWidth ) { + attrs.opacity = attrs.width = type; + } + + return attrs; +} + +function createTween( value, prop, animation ) { + var tween, + collection = ( Animation.tweeners[ prop ] || [] ).concat( Animation.tweeners[ "*" ] ), + index = 0, + length = collection.length; + for ( ; index < length; index++ ) { + if ( ( tween = collection[ index ].call( animation, prop, value ) ) ) { + + // We're done with this property + return tween; + } + } +} + +function defaultPrefilter( elem, props, opts ) { + var prop, value, toggle, hooks, oldfire, propTween, restoreDisplay, display, + isBox = "width" in props || "height" in props, + anim = this, + orig = {}, + style = elem.style, + hidden = elem.nodeType && isHiddenWithinTree( elem ), + dataShow = dataPriv.get( elem, "fxshow" ); + + // Queue-skipping animations hijack the fx hooks + if ( !opts.queue ) { + hooks = jQuery._queueHooks( elem, "fx" ); + if ( hooks.unqueued == null ) { + hooks.unqueued = 0; + oldfire = hooks.empty.fire; + hooks.empty.fire = function() { + if ( !hooks.unqueued ) { + oldfire(); + } + }; + } + hooks.unqueued++; + + anim.always( function() { + + // Ensure the complete handler is called before this completes + anim.always( function() { + hooks.unqueued--; + if ( !jQuery.queue( elem, "fx" ).length ) { + hooks.empty.fire(); + } + } ); + } ); + } + + // Detect show/hide animations + for ( prop in props ) { + value = props[ prop ]; + if ( rfxtypes.test( value ) ) { + delete props[ prop ]; + toggle = toggle || value === "toggle"; + if ( value === ( hidden ? "hide" : "show" ) ) { + + // Pretend to be hidden if this is a "show" and + // there is still data from a stopped show/hide + if ( value === "show" && dataShow && dataShow[ prop ] !== undefined ) { + hidden = true; + + // Ignore all other no-op show/hide data + } else { + continue; + } + } + orig[ prop ] = dataShow && dataShow[ prop ] || jQuery.style( elem, prop ); + } + } + + // Bail out if this is a no-op like .hide().hide() + propTween = !jQuery.isEmptyObject( props ); + if ( !propTween && jQuery.isEmptyObject( orig ) ) { + return; + } + + // Restrict "overflow" and "display" styles during box animations + if ( isBox && elem.nodeType === 1 ) { + + // Support: IE <=9 - 11, Edge 12 - 15 + // Record all 3 overflow attributes because IE does not infer the shorthand + // from identically-valued overflowX and overflowY and Edge just mirrors + // the overflowX value there. + opts.overflow = [ style.overflow, style.overflowX, style.overflowY ]; + + // Identify a display type, preferring old show/hide data over the CSS cascade + restoreDisplay = dataShow && dataShow.display; + if ( restoreDisplay == null ) { + restoreDisplay = dataPriv.get( elem, "display" ); + } + display = jQuery.css( elem, "display" ); + if ( display === "none" ) { + if ( restoreDisplay ) { + display = restoreDisplay; + } else { + + // Get nonempty value(s) by temporarily forcing visibility + showHide( [ elem ], true ); + restoreDisplay = elem.style.display || restoreDisplay; + display = jQuery.css( elem, "display" ); + showHide( [ elem ] ); + } + } + + // Animate inline elements as inline-block + if ( display === "inline" || display === "inline-block" && restoreDisplay != null ) { + if ( jQuery.css( elem, "float" ) === "none" ) { + + // Restore the original display value at the end of pure show/hide animations + if ( !propTween ) { + anim.done( function() { + style.display = restoreDisplay; + } ); + if ( restoreDisplay == null ) { + display = style.display; + restoreDisplay = display === "none" ? "" : display; + } + } + style.display = "inline-block"; + } + } + } + + if ( opts.overflow ) { + style.overflow = "hidden"; + anim.always( function() { + style.overflow = opts.overflow[ 0 ]; + style.overflowX = opts.overflow[ 1 ]; + style.overflowY = opts.overflow[ 2 ]; + } ); + } + + // Implement show/hide animations + propTween = false; + for ( prop in orig ) { + + // General show/hide setup for this element animation + if ( !propTween ) { + if ( dataShow ) { + if ( "hidden" in dataShow ) { + hidden = dataShow.hidden; + } + } else { + dataShow = dataPriv.access( elem, "fxshow", { display: restoreDisplay } ); + } + + // Store hidden/visible for toggle so `.stop().toggle()` "reverses" + if ( toggle ) { + dataShow.hidden = !hidden; + } + + // Show elements before animating them + if ( hidden ) { + showHide( [ elem ], true ); + } + + /* eslint-disable no-loop-func */ + + anim.done( function() { + + /* eslint-enable no-loop-func */ + + // The final step of a "hide" animation is actually hiding the element + if ( !hidden ) { + showHide( [ elem ] ); + } + dataPriv.remove( elem, "fxshow" ); + for ( prop in orig ) { + jQuery.style( elem, prop, orig[ prop ] ); + } + } ); + } + + // Per-property setup + propTween = createTween( hidden ? dataShow[ prop ] : 0, prop, anim ); + if ( !( prop in dataShow ) ) { + dataShow[ prop ] = propTween.start; + if ( hidden ) { + propTween.end = propTween.start; + propTween.start = 0; + } + } + } +} + +function propFilter( props, specialEasing ) { + var index, name, easing, value, hooks; + + // camelCase, specialEasing and expand cssHook pass + for ( index in props ) { + name = camelCase( index ); + easing = specialEasing[ name ]; + value = props[ index ]; + if ( Array.isArray( value ) ) { + easing = value[ 1 ]; + value = props[ index ] = value[ 0 ]; + } + + if ( index !== name ) { + props[ name ] = value; + delete props[ index ]; + } + + hooks = jQuery.cssHooks[ name ]; + if ( hooks && "expand" in hooks ) { + value = hooks.expand( value ); + delete props[ name ]; + + // Not quite $.extend, this won't overwrite existing keys. + // Reusing 'index' because we have the correct "name" + for ( index in value ) { + if ( !( index in props ) ) { + props[ index ] = value[ index ]; + specialEasing[ index ] = easing; + } + } + } else { + specialEasing[ name ] = easing; + } + } +} + +function Animation( elem, properties, options ) { + var result, + stopped, + index = 0, + length = Animation.prefilters.length, + deferred = jQuery.Deferred().always( function() { + + // Don't match elem in the :animated selector + delete tick.elem; + } ), + tick = function() { + if ( stopped ) { + return false; + } + var currentTime = fxNow || createFxNow(), + remaining = Math.max( 0, animation.startTime + animation.duration - currentTime ), + + // Support: Android 2.3 only + // Archaic crash bug won't allow us to use `1 - ( 0.5 || 0 )` (#12497) + temp = remaining / animation.duration || 0, + percent = 1 - temp, + index = 0, + length = animation.tweens.length; + + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( percent ); + } + + deferred.notifyWith( elem, [ animation, percent, remaining ] ); + + // If there's more to do, yield + if ( percent < 1 && length ) { + return remaining; + } + + // If this was an empty animation, synthesize a final progress notification + if ( !length ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + } + + // Resolve the animation and report its conclusion + deferred.resolveWith( elem, [ animation ] ); + return false; + }, + animation = deferred.promise( { + elem: elem, + props: jQuery.extend( {}, properties ), + opts: jQuery.extend( true, { + specialEasing: {}, + easing: jQuery.easing._default + }, options ), + originalProperties: properties, + originalOptions: options, + startTime: fxNow || createFxNow(), + duration: options.duration, + tweens: [], + createTween: function( prop, end ) { + var tween = jQuery.Tween( elem, animation.opts, prop, end, + animation.opts.specialEasing[ prop ] || animation.opts.easing ); + animation.tweens.push( tween ); + return tween; + }, + stop: function( gotoEnd ) { + var index = 0, + + // If we are going to the end, we want to run all the tweens + // otherwise we skip this part + length = gotoEnd ? animation.tweens.length : 0; + if ( stopped ) { + return this; + } + stopped = true; + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( 1 ); + } + + // Resolve when we played the last frame; otherwise, reject + if ( gotoEnd ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + deferred.resolveWith( elem, [ animation, gotoEnd ] ); + } else { + deferred.rejectWith( elem, [ animation, gotoEnd ] ); + } + return this; + } + } ), + props = animation.props; + + propFilter( props, animation.opts.specialEasing ); + + for ( ; index < length; index++ ) { + result = Animation.prefilters[ index ].call( animation, elem, props, animation.opts ); + if ( result ) { + if ( isFunction( result.stop ) ) { + jQuery._queueHooks( animation.elem, animation.opts.queue ).stop = + result.stop.bind( result ); + } + return result; + } + } + + jQuery.map( props, createTween, animation ); + + if ( isFunction( animation.opts.start ) ) { + animation.opts.start.call( elem, animation ); + } + + // Attach callbacks from options + animation + .progress( animation.opts.progress ) + .done( animation.opts.done, animation.opts.complete ) + .fail( animation.opts.fail ) + .always( animation.opts.always ); + + jQuery.fx.timer( + jQuery.extend( tick, { + elem: elem, + anim: animation, + queue: animation.opts.queue + } ) + ); + + return animation; +} + +jQuery.Animation = jQuery.extend( Animation, { + + tweeners: { + "*": [ function( prop, value ) { + var tween = this.createTween( prop, value ); + adjustCSS( tween.elem, prop, rcssNum.exec( value ), tween ); + return tween; + } ] + }, + + tweener: function( props, callback ) { + if ( isFunction( props ) ) { + callback = props; + props = [ "*" ]; + } else { + props = props.match( rnothtmlwhite ); + } + + var prop, + index = 0, + length = props.length; + + for ( ; index < length; index++ ) { + prop = props[ index ]; + Animation.tweeners[ prop ] = Animation.tweeners[ prop ] || []; + Animation.tweeners[ prop ].unshift( callback ); + } + }, + + prefilters: [ defaultPrefilter ], + + prefilter: function( callback, prepend ) { + if ( prepend ) { + Animation.prefilters.unshift( callback ); + } else { + Animation.prefilters.push( callback ); + } + } +} ); + +jQuery.speed = function( speed, easing, fn ) { + var opt = speed && typeof speed === "object" ? jQuery.extend( {}, speed ) : { + complete: fn || !fn && easing || + isFunction( speed ) && speed, + duration: speed, + easing: fn && easing || easing && !isFunction( easing ) && easing + }; + + // Go to the end state if fx are off + if ( jQuery.fx.off ) { + opt.duration = 0; + + } else { + if ( typeof opt.duration !== "number" ) { + if ( opt.duration in jQuery.fx.speeds ) { + opt.duration = jQuery.fx.speeds[ opt.duration ]; + + } else { + opt.duration = jQuery.fx.speeds._default; + } + } + } + + // Normalize opt.queue - true/undefined/null -> "fx" + if ( opt.queue == null || opt.queue === true ) { + opt.queue = "fx"; + } + + // Queueing + opt.old = opt.complete; + + opt.complete = function() { + if ( isFunction( opt.old ) ) { + opt.old.call( this ); + } + + if ( opt.queue ) { + jQuery.dequeue( this, opt.queue ); + } + }; + + return opt; +}; + +jQuery.fn.extend( { + fadeTo: function( speed, to, easing, callback ) { + + // Show any hidden elements after setting opacity to 0 + return this.filter( isHiddenWithinTree ).css( "opacity", 0 ).show() + + // Animate to the value specified + .end().animate( { opacity: to }, speed, easing, callback ); + }, + animate: function( prop, speed, easing, callback ) { + var empty = jQuery.isEmptyObject( prop ), + optall = jQuery.speed( speed, easing, callback ), + doAnimation = function() { + + // Operate on a copy of prop so per-property easing won't be lost + var anim = Animation( this, jQuery.extend( {}, prop ), optall ); + + // Empty animations, or finishing resolves immediately + if ( empty || dataPriv.get( this, "finish" ) ) { + anim.stop( true ); + } + }; + + doAnimation.finish = doAnimation; + + return empty || optall.queue === false ? + this.each( doAnimation ) : + this.queue( optall.queue, doAnimation ); + }, + stop: function( type, clearQueue, gotoEnd ) { + var stopQueue = function( hooks ) { + var stop = hooks.stop; + delete hooks.stop; + stop( gotoEnd ); + }; + + if ( typeof type !== "string" ) { + gotoEnd = clearQueue; + clearQueue = type; + type = undefined; + } + if ( clearQueue ) { + this.queue( type || "fx", [] ); + } + + return this.each( function() { + var dequeue = true, + index = type != null && type + "queueHooks", + timers = jQuery.timers, + data = dataPriv.get( this ); + + if ( index ) { + if ( data[ index ] && data[ index ].stop ) { + stopQueue( data[ index ] ); + } + } else { + for ( index in data ) { + if ( data[ index ] && data[ index ].stop && rrun.test( index ) ) { + stopQueue( data[ index ] ); + } + } + } + + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && + ( type == null || timers[ index ].queue === type ) ) { + + timers[ index ].anim.stop( gotoEnd ); + dequeue = false; + timers.splice( index, 1 ); + } + } + + // Start the next in the queue if the last step wasn't forced. + // Timers currently will call their complete callbacks, which + // will dequeue but only if they were gotoEnd. + if ( dequeue || !gotoEnd ) { + jQuery.dequeue( this, type ); + } + } ); + }, + finish: function( type ) { + if ( type !== false ) { + type = type || "fx"; + } + return this.each( function() { + var index, + data = dataPriv.get( this ), + queue = data[ type + "queue" ], + hooks = data[ type + "queueHooks" ], + timers = jQuery.timers, + length = queue ? queue.length : 0; + + // Enable finishing flag on private data + data.finish = true; + + // Empty the queue first + jQuery.queue( this, type, [] ); + + if ( hooks && hooks.stop ) { + hooks.stop.call( this, true ); + } + + // Look for any active animations, and finish them + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && timers[ index ].queue === type ) { + timers[ index ].anim.stop( true ); + timers.splice( index, 1 ); + } + } + + // Look for any animations in the old queue and finish them + for ( index = 0; index < length; index++ ) { + if ( queue[ index ] && queue[ index ].finish ) { + queue[ index ].finish.call( this ); + } + } + + // Turn off finishing flag + delete data.finish; + } ); + } +} ); + +jQuery.each( [ "toggle", "show", "hide" ], function( _i, name ) { + var cssFn = jQuery.fn[ name ]; + jQuery.fn[ name ] = function( speed, easing, callback ) { + return speed == null || typeof speed === "boolean" ? + cssFn.apply( this, arguments ) : + this.animate( genFx( name, true ), speed, easing, callback ); + }; +} ); + +// Generate shortcuts for custom animations +jQuery.each( { + slideDown: genFx( "show" ), + slideUp: genFx( "hide" ), + slideToggle: genFx( "toggle" ), + fadeIn: { opacity: "show" }, + fadeOut: { opacity: "hide" }, + fadeToggle: { opacity: "toggle" } +}, function( name, props ) { + jQuery.fn[ name ] = function( speed, easing, callback ) { + return this.animate( props, speed, easing, callback ); + }; +} ); + +jQuery.timers = []; +jQuery.fx.tick = function() { + var timer, + i = 0, + timers = jQuery.timers; + + fxNow = Date.now(); + + for ( ; i < timers.length; i++ ) { + timer = timers[ i ]; + + // Run the timer and safely remove it when done (allowing for external removal) + if ( !timer() && timers[ i ] === timer ) { + timers.splice( i--, 1 ); + } + } + + if ( !timers.length ) { + jQuery.fx.stop(); + } + fxNow = undefined; +}; + +jQuery.fx.timer = function( timer ) { + jQuery.timers.push( timer ); + jQuery.fx.start(); +}; + +jQuery.fx.interval = 13; +jQuery.fx.start = function() { + if ( inProgress ) { + return; + } + + inProgress = true; + schedule(); +}; + +jQuery.fx.stop = function() { + inProgress = null; +}; + +jQuery.fx.speeds = { + slow: 600, + fast: 200, + + // Default speed + _default: 400 +}; + + +// Based off of the plugin by Clint Helfers, with permission. +// https://web.archive.org/web/20100324014747/http://blindsignals.com/index.php/2009/07/jquery-delay/ +jQuery.fn.delay = function( time, type ) { + time = jQuery.fx ? jQuery.fx.speeds[ time ] || time : time; + type = type || "fx"; + + return this.queue( type, function( next, hooks ) { + var timeout = window.setTimeout( next, time ); + hooks.stop = function() { + window.clearTimeout( timeout ); + }; + } ); +}; + + +( function() { + var input = document.createElement( "input" ), + select = document.createElement( "select" ), + opt = select.appendChild( document.createElement( "option" ) ); + + input.type = "checkbox"; + + // Support: Android <=4.3 only + // Default value for a checkbox should be "on" + support.checkOn = input.value !== ""; + + // Support: IE <=11 only + // Must access selectedIndex to make default options select + support.optSelected = opt.selected; + + // Support: IE <=11 only + // An input loses its value after becoming a radio + input = document.createElement( "input" ); + input.value = "t"; + input.type = "radio"; + support.radioValue = input.value === "t"; +} )(); + + +var boolHook, + attrHandle = jQuery.expr.attrHandle; + +jQuery.fn.extend( { + attr: function( name, value ) { + return access( this, jQuery.attr, name, value, arguments.length > 1 ); + }, + + removeAttr: function( name ) { + return this.each( function() { + jQuery.removeAttr( this, name ); + } ); + } +} ); + +jQuery.extend( { + attr: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set attributes on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + // Fallback to prop when attributes are not supported + if ( typeof elem.getAttribute === "undefined" ) { + return jQuery.prop( elem, name, value ); + } + + // Attribute hooks are determined by the lowercase version + // Grab necessary hook if one is defined + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + hooks = jQuery.attrHooks[ name.toLowerCase() ] || + ( jQuery.expr.match.bool.test( name ) ? boolHook : undefined ); + } + + if ( value !== undefined ) { + if ( value === null ) { + jQuery.removeAttr( elem, name ); + return; + } + + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + elem.setAttribute( name, value + "" ); + return value; + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + ret = jQuery.find.attr( elem, name ); + + // Non-existent attributes return null, we normalize to undefined + return ret == null ? undefined : ret; + }, + + attrHooks: { + type: { + set: function( elem, value ) { + if ( !support.radioValue && value === "radio" && + nodeName( elem, "input" ) ) { + var val = elem.value; + elem.setAttribute( "type", value ); + if ( val ) { + elem.value = val; + } + return value; + } + } + } + }, + + removeAttr: function( elem, value ) { + var name, + i = 0, + + // Attribute names can contain non-HTML whitespace characters + // https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 + attrNames = value && value.match( rnothtmlwhite ); + + if ( attrNames && elem.nodeType === 1 ) { + while ( ( name = attrNames[ i++ ] ) ) { + elem.removeAttribute( name ); + } + } + } +} ); + +// Hooks for boolean attributes +boolHook = { + set: function( elem, value, name ) { + if ( value === false ) { + + // Remove boolean attributes when set to false + jQuery.removeAttr( elem, name ); + } else { + elem.setAttribute( name, name ); + } + return name; + } +}; + +jQuery.each( jQuery.expr.match.bool.source.match( /\w+/g ), function( _i, name ) { + var getter = attrHandle[ name ] || jQuery.find.attr; + + attrHandle[ name ] = function( elem, name, isXML ) { + var ret, handle, + lowercaseName = name.toLowerCase(); + + if ( !isXML ) { + + // Avoid an infinite loop by temporarily removing this function from the getter + handle = attrHandle[ lowercaseName ]; + attrHandle[ lowercaseName ] = ret; + ret = getter( elem, name, isXML ) != null ? + lowercaseName : + null; + attrHandle[ lowercaseName ] = handle; + } + return ret; + }; +} ); + + + + +var rfocusable = /^(?:input|select|textarea|button)$/i, + rclickable = /^(?:a|area)$/i; + +jQuery.fn.extend( { + prop: function( name, value ) { + return access( this, jQuery.prop, name, value, arguments.length > 1 ); + }, + + removeProp: function( name ) { + return this.each( function() { + delete this[ jQuery.propFix[ name ] || name ]; + } ); + } +} ); + +jQuery.extend( { + prop: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set properties on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + + // Fix name and attach hooks + name = jQuery.propFix[ name ] || name; + hooks = jQuery.propHooks[ name ]; + } + + if ( value !== undefined ) { + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + return ( elem[ name ] = value ); + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + return elem[ name ]; + }, + + propHooks: { + tabIndex: { + get: function( elem ) { + + // Support: IE <=9 - 11 only + // elem.tabIndex doesn't always return the + // correct value when it hasn't been explicitly set + // https://web.archive.org/web/20141116233347/http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ + // Use proper attribute retrieval(#12072) + var tabindex = jQuery.find.attr( elem, "tabindex" ); + + if ( tabindex ) { + return parseInt( tabindex, 10 ); + } + + if ( + rfocusable.test( elem.nodeName ) || + rclickable.test( elem.nodeName ) && + elem.href + ) { + return 0; + } + + return -1; + } + } + }, + + propFix: { + "for": "htmlFor", + "class": "className" + } +} ); + +// Support: IE <=11 only +// Accessing the selectedIndex property +// forces the browser to respect setting selected +// on the option +// The getter ensures a default option is selected +// when in an optgroup +// eslint rule "no-unused-expressions" is disabled for this code +// since it considers such accessions noop +if ( !support.optSelected ) { + jQuery.propHooks.selected = { + get: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent && parent.parentNode ) { + parent.parentNode.selectedIndex; + } + return null; + }, + set: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent ) { + parent.selectedIndex; + + if ( parent.parentNode ) { + parent.parentNode.selectedIndex; + } + } + } + }; +} + +jQuery.each( [ + "tabIndex", + "readOnly", + "maxLength", + "cellSpacing", + "cellPadding", + "rowSpan", + "colSpan", + "useMap", + "frameBorder", + "contentEditable" +], function() { + jQuery.propFix[ this.toLowerCase() ] = this; +} ); + + + + + // Strip and collapse whitespace according to HTML spec + // https://infra.spec.whatwg.org/#strip-and-collapse-ascii-whitespace + function stripAndCollapse( value ) { + var tokens = value.match( rnothtmlwhite ) || []; + return tokens.join( " " ); + } + + +function getClass( elem ) { + return elem.getAttribute && elem.getAttribute( "class" ) || ""; +} + +function classesToArray( value ) { + if ( Array.isArray( value ) ) { + return value; + } + if ( typeof value === "string" ) { + return value.match( rnothtmlwhite ) || []; + } + return []; +} + +jQuery.fn.extend( { + addClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).addClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + if ( cur.indexOf( " " + clazz + " " ) < 0 ) { + cur += clazz + " "; + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + removeClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).removeClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + if ( !arguments.length ) { + return this.attr( "class", "" ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + + // This expression is here for better compressibility (see addClass) + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + + // Remove *all* instances + while ( cur.indexOf( " " + clazz + " " ) > -1 ) { + cur = cur.replace( " " + clazz + " ", " " ); + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + toggleClass: function( value, stateVal ) { + var type = typeof value, + isValidValue = type === "string" || Array.isArray( value ); + + if ( typeof stateVal === "boolean" && isValidValue ) { + return stateVal ? this.addClass( value ) : this.removeClass( value ); + } + + if ( isFunction( value ) ) { + return this.each( function( i ) { + jQuery( this ).toggleClass( + value.call( this, i, getClass( this ), stateVal ), + stateVal + ); + } ); + } + + return this.each( function() { + var className, i, self, classNames; + + if ( isValidValue ) { + + // Toggle individual class names + i = 0; + self = jQuery( this ); + classNames = classesToArray( value ); + + while ( ( className = classNames[ i++ ] ) ) { + + // Check each className given, space separated list + if ( self.hasClass( className ) ) { + self.removeClass( className ); + } else { + self.addClass( className ); + } + } + + // Toggle whole class name + } else if ( value === undefined || type === "boolean" ) { + className = getClass( this ); + if ( className ) { + + // Store className if set + dataPriv.set( this, "__className__", className ); + } + + // If the element has a class name or if we're passed `false`, + // then remove the whole classname (if there was one, the above saved it). + // Otherwise bring back whatever was previously saved (if anything), + // falling back to the empty string if nothing was stored. + if ( this.setAttribute ) { + this.setAttribute( "class", + className || value === false ? + "" : + dataPriv.get( this, "__className__" ) || "" + ); + } + } + } ); + }, + + hasClass: function( selector ) { + var className, elem, + i = 0; + + className = " " + selector + " "; + while ( ( elem = this[ i++ ] ) ) { + if ( elem.nodeType === 1 && + ( " " + stripAndCollapse( getClass( elem ) ) + " " ).indexOf( className ) > -1 ) { + return true; + } + } + + return false; + } +} ); + + + + +var rreturn = /\r/g; + +jQuery.fn.extend( { + val: function( value ) { + var hooks, ret, valueIsFunction, + elem = this[ 0 ]; + + if ( !arguments.length ) { + if ( elem ) { + hooks = jQuery.valHooks[ elem.type ] || + jQuery.valHooks[ elem.nodeName.toLowerCase() ]; + + if ( hooks && + "get" in hooks && + ( ret = hooks.get( elem, "value" ) ) !== undefined + ) { + return ret; + } + + ret = elem.value; + + // Handle most common string cases + if ( typeof ret === "string" ) { + return ret.replace( rreturn, "" ); + } + + // Handle cases where value is null/undef or number + return ret == null ? "" : ret; + } + + return; + } + + valueIsFunction = isFunction( value ); + + return this.each( function( i ) { + var val; + + if ( this.nodeType !== 1 ) { + return; + } + + if ( valueIsFunction ) { + val = value.call( this, i, jQuery( this ).val() ); + } else { + val = value; + } + + // Treat null/undefined as ""; convert numbers to string + if ( val == null ) { + val = ""; + + } else if ( typeof val === "number" ) { + val += ""; + + } else if ( Array.isArray( val ) ) { + val = jQuery.map( val, function( value ) { + return value == null ? "" : value + ""; + } ); + } + + hooks = jQuery.valHooks[ this.type ] || jQuery.valHooks[ this.nodeName.toLowerCase() ]; + + // If set returns undefined, fall back to normal setting + if ( !hooks || !( "set" in hooks ) || hooks.set( this, val, "value" ) === undefined ) { + this.value = val; + } + } ); + } +} ); + +jQuery.extend( { + valHooks: { + option: { + get: function( elem ) { + + var val = jQuery.find.attr( elem, "value" ); + return val != null ? + val : + + // Support: IE <=10 - 11 only + // option.text throws exceptions (#14686, #14858) + // Strip and collapse whitespace + // https://html.spec.whatwg.org/#strip-and-collapse-whitespace + stripAndCollapse( jQuery.text( elem ) ); + } + }, + select: { + get: function( elem ) { + var value, option, i, + options = elem.options, + index = elem.selectedIndex, + one = elem.type === "select-one", + values = one ? null : [], + max = one ? index + 1 : options.length; + + if ( index < 0 ) { + i = max; + + } else { + i = one ? index : 0; + } + + // Loop through all the selected options + for ( ; i < max; i++ ) { + option = options[ i ]; + + // Support: IE <=9 only + // IE8-9 doesn't update selected after form reset (#2551) + if ( ( option.selected || i === index ) && + + // Don't return options that are disabled or in a disabled optgroup + !option.disabled && + ( !option.parentNode.disabled || + !nodeName( option.parentNode, "optgroup" ) ) ) { + + // Get the specific value for the option + value = jQuery( option ).val(); + + // We don't need an array for one selects + if ( one ) { + return value; + } + + // Multi-Selects return an array + values.push( value ); + } + } + + return values; + }, + + set: function( elem, value ) { + var optionSet, option, + options = elem.options, + values = jQuery.makeArray( value ), + i = options.length; + + while ( i-- ) { + option = options[ i ]; + + /* eslint-disable no-cond-assign */ + + if ( option.selected = + jQuery.inArray( jQuery.valHooks.option.get( option ), values ) > -1 + ) { + optionSet = true; + } + + /* eslint-enable no-cond-assign */ + } + + // Force browsers to behave consistently when non-matching value is set + if ( !optionSet ) { + elem.selectedIndex = -1; + } + return values; + } + } + } +} ); + +// Radios and checkboxes getter/setter +jQuery.each( [ "radio", "checkbox" ], function() { + jQuery.valHooks[ this ] = { + set: function( elem, value ) { + if ( Array.isArray( value ) ) { + return ( elem.checked = jQuery.inArray( jQuery( elem ).val(), value ) > -1 ); + } + } + }; + if ( !support.checkOn ) { + jQuery.valHooks[ this ].get = function( elem ) { + return elem.getAttribute( "value" ) === null ? "on" : elem.value; + }; + } +} ); + + + + +// Return jQuery for attributes-only inclusion + + +support.focusin = "onfocusin" in window; + + +var rfocusMorph = /^(?:focusinfocus|focusoutblur)$/, + stopPropagationCallback = function( e ) { + e.stopPropagation(); + }; + +jQuery.extend( jQuery.event, { + + trigger: function( event, data, elem, onlyHandlers ) { + + var i, cur, tmp, bubbleType, ontype, handle, special, lastElement, + eventPath = [ elem || document ], + type = hasOwn.call( event, "type" ) ? event.type : event, + namespaces = hasOwn.call( event, "namespace" ) ? event.namespace.split( "." ) : []; + + cur = lastElement = tmp = elem = elem || document; + + // Don't do events on text and comment nodes + if ( elem.nodeType === 3 || elem.nodeType === 8 ) { + return; + } + + // focus/blur morphs to focusin/out; ensure we're not firing them right now + if ( rfocusMorph.test( type + jQuery.event.triggered ) ) { + return; + } + + if ( type.indexOf( "." ) > -1 ) { + + // Namespaced trigger; create a regexp to match event type in handle() + namespaces = type.split( "." ); + type = namespaces.shift(); + namespaces.sort(); + } + ontype = type.indexOf( ":" ) < 0 && "on" + type; + + // Caller can pass in a jQuery.Event object, Object, or just an event type string + event = event[ jQuery.expando ] ? + event : + new jQuery.Event( type, typeof event === "object" && event ); + + // Trigger bitmask: & 1 for native handlers; & 2 for jQuery (always true) + event.isTrigger = onlyHandlers ? 2 : 3; + event.namespace = namespaces.join( "." ); + event.rnamespace = event.namespace ? + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ) : + null; + + // Clean up the event in case it is being reused + event.result = undefined; + if ( !event.target ) { + event.target = elem; + } + + // Clone any incoming data and prepend the event, creating the handler arg list + data = data == null ? + [ event ] : + jQuery.makeArray( data, [ event ] ); + + // Allow special events to draw outside the lines + special = jQuery.event.special[ type ] || {}; + if ( !onlyHandlers && special.trigger && special.trigger.apply( elem, data ) === false ) { + return; + } + + // Determine event propagation path in advance, per W3C events spec (#9951) + // Bubble up to document, then to window; watch for a global ownerDocument var (#9724) + if ( !onlyHandlers && !special.noBubble && !isWindow( elem ) ) { + + bubbleType = special.delegateType || type; + if ( !rfocusMorph.test( bubbleType + type ) ) { + cur = cur.parentNode; + } + for ( ; cur; cur = cur.parentNode ) { + eventPath.push( cur ); + tmp = cur; + } + + // Only add window if we got to document (e.g., not plain obj or detached DOM) + if ( tmp === ( elem.ownerDocument || document ) ) { + eventPath.push( tmp.defaultView || tmp.parentWindow || window ); + } + } + + // Fire handlers on the event path + i = 0; + while ( ( cur = eventPath[ i++ ] ) && !event.isPropagationStopped() ) { + lastElement = cur; + event.type = i > 1 ? + bubbleType : + special.bindType || type; + + // jQuery handler + handle = ( dataPriv.get( cur, "events" ) || Object.create( null ) )[ event.type ] && + dataPriv.get( cur, "handle" ); + if ( handle ) { + handle.apply( cur, data ); + } + + // Native handler + handle = ontype && cur[ ontype ]; + if ( handle && handle.apply && acceptData( cur ) ) { + event.result = handle.apply( cur, data ); + if ( event.result === false ) { + event.preventDefault(); + } + } + } + event.type = type; + + // If nobody prevented the default action, do it now + if ( !onlyHandlers && !event.isDefaultPrevented() ) { + + if ( ( !special._default || + special._default.apply( eventPath.pop(), data ) === false ) && + acceptData( elem ) ) { + + // Call a native DOM method on the target with the same name as the event. + // Don't do default actions on window, that's where global variables be (#6170) + if ( ontype && isFunction( elem[ type ] ) && !isWindow( elem ) ) { + + // Don't re-trigger an onFOO event when we call its FOO() method + tmp = elem[ ontype ]; + + if ( tmp ) { + elem[ ontype ] = null; + } + + // Prevent re-triggering of the same event, since we already bubbled it above + jQuery.event.triggered = type; + + if ( event.isPropagationStopped() ) { + lastElement.addEventListener( type, stopPropagationCallback ); + } + + elem[ type ](); + + if ( event.isPropagationStopped() ) { + lastElement.removeEventListener( type, stopPropagationCallback ); + } + + jQuery.event.triggered = undefined; + + if ( tmp ) { + elem[ ontype ] = tmp; + } + } + } + } + + return event.result; + }, + + // Piggyback on a donor event to simulate a different one + // Used only for `focus(in | out)` events + simulate: function( type, elem, event ) { + var e = jQuery.extend( + new jQuery.Event(), + event, + { + type: type, + isSimulated: true + } + ); + + jQuery.event.trigger( e, null, elem ); + } + +} ); + +jQuery.fn.extend( { + + trigger: function( type, data ) { + return this.each( function() { + jQuery.event.trigger( type, data, this ); + } ); + }, + triggerHandler: function( type, data ) { + var elem = this[ 0 ]; + if ( elem ) { + return jQuery.event.trigger( type, data, elem, true ); + } + } +} ); + + +// Support: Firefox <=44 +// Firefox doesn't have focus(in | out) events +// Related ticket - https://bugzilla.mozilla.org/show_bug.cgi?id=687787 +// +// Support: Chrome <=48 - 49, Safari <=9.0 - 9.1 +// focus(in | out) events fire after focus & blur events, +// which is spec violation - http://www.w3.org/TR/DOM-Level-3-Events/#events-focusevent-event-order +// Related ticket - https://bugs.chromium.org/p/chromium/issues/detail?id=449857 +if ( !support.focusin ) { + jQuery.each( { focus: "focusin", blur: "focusout" }, function( orig, fix ) { + + // Attach a single capturing handler on the document while someone wants focusin/focusout + var handler = function( event ) { + jQuery.event.simulate( fix, event.target, jQuery.event.fix( event ) ); + }; + + jQuery.event.special[ fix ] = { + setup: function() { + + // Handle: regular nodes (via `this.ownerDocument`), window + // (via `this.document`) & document (via `this`). + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ); + + if ( !attaches ) { + doc.addEventListener( orig, handler, true ); + } + dataPriv.access( doc, fix, ( attaches || 0 ) + 1 ); + }, + teardown: function() { + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ) - 1; + + if ( !attaches ) { + doc.removeEventListener( orig, handler, true ); + dataPriv.remove( doc, fix ); + + } else { + dataPriv.access( doc, fix, attaches ); + } + } + }; + } ); +} +var location = window.location; + +var nonce = { guid: Date.now() }; + +var rquery = ( /\?/ ); + + + +// Cross-browser xml parsing +jQuery.parseXML = function( data ) { + var xml, parserErrorElem; + if ( !data || typeof data !== "string" ) { + return null; + } + + // Support: IE 9 - 11 only + // IE throws on parseFromString with invalid input. + try { + xml = ( new window.DOMParser() ).parseFromString( data, "text/xml" ); + } catch ( e ) {} + + parserErrorElem = xml && xml.getElementsByTagName( "parsererror" )[ 0 ]; + if ( !xml || parserErrorElem ) { + jQuery.error( "Invalid XML: " + ( + parserErrorElem ? + jQuery.map( parserErrorElem.childNodes, function( el ) { + return el.textContent; + } ).join( "\n" ) : + data + ) ); + } + return xml; +}; + + +var + rbracket = /\[\]$/, + rCRLF = /\r?\n/g, + rsubmitterTypes = /^(?:submit|button|image|reset|file)$/i, + rsubmittable = /^(?:input|select|textarea|keygen)/i; + +function buildParams( prefix, obj, traditional, add ) { + var name; + + if ( Array.isArray( obj ) ) { + + // Serialize array item. + jQuery.each( obj, function( i, v ) { + if ( traditional || rbracket.test( prefix ) ) { + + // Treat each array item as a scalar. + add( prefix, v ); + + } else { + + // Item is non-scalar (array or object), encode its numeric index. + buildParams( + prefix + "[" + ( typeof v === "object" && v != null ? i : "" ) + "]", + v, + traditional, + add + ); + } + } ); + + } else if ( !traditional && toType( obj ) === "object" ) { + + // Serialize object item. + for ( name in obj ) { + buildParams( prefix + "[" + name + "]", obj[ name ], traditional, add ); + } + + } else { + + // Serialize scalar item. + add( prefix, obj ); + } +} + +// Serialize an array of form elements or a set of +// key/values into a query string +jQuery.param = function( a, traditional ) { + var prefix, + s = [], + add = function( key, valueOrFunction ) { + + // If value is a function, invoke it and use its return value + var value = isFunction( valueOrFunction ) ? + valueOrFunction() : + valueOrFunction; + + s[ s.length ] = encodeURIComponent( key ) + "=" + + encodeURIComponent( value == null ? "" : value ); + }; + + if ( a == null ) { + return ""; + } + + // If an array was passed in, assume that it is an array of form elements. + if ( Array.isArray( a ) || ( a.jquery && !jQuery.isPlainObject( a ) ) ) { + + // Serialize the form elements + jQuery.each( a, function() { + add( this.name, this.value ); + } ); + + } else { + + // If traditional, encode the "old" way (the way 1.3.2 or older + // did it), otherwise encode params recursively. + for ( prefix in a ) { + buildParams( prefix, a[ prefix ], traditional, add ); + } + } + + // Return the resulting serialization + return s.join( "&" ); +}; + +jQuery.fn.extend( { + serialize: function() { + return jQuery.param( this.serializeArray() ); + }, + serializeArray: function() { + return this.map( function() { + + // Can add propHook for "elements" to filter or add form elements + var elements = jQuery.prop( this, "elements" ); + return elements ? jQuery.makeArray( elements ) : this; + } ).filter( function() { + var type = this.type; + + // Use .is( ":disabled" ) so that fieldset[disabled] works + return this.name && !jQuery( this ).is( ":disabled" ) && + rsubmittable.test( this.nodeName ) && !rsubmitterTypes.test( type ) && + ( this.checked || !rcheckableType.test( type ) ); + } ).map( function( _i, elem ) { + var val = jQuery( this ).val(); + + if ( val == null ) { + return null; + } + + if ( Array.isArray( val ) ) { + return jQuery.map( val, function( val ) { + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ); + } + + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ).get(); + } +} ); + + +var + r20 = /%20/g, + rhash = /#.*$/, + rantiCache = /([?&])_=[^&]*/, + rheaders = /^(.*?):[ \t]*([^\r\n]*)$/mg, + + // #7653, #8125, #8152: local protocol detection + rlocalProtocol = /^(?:about|app|app-storage|.+-extension|file|res|widget):$/, + rnoContent = /^(?:GET|HEAD)$/, + rprotocol = /^\/\//, + + /* Prefilters + * 1) They are useful to introduce custom dataTypes (see ajax/jsonp.js for an example) + * 2) These are called: + * - BEFORE asking for a transport + * - AFTER param serialization (s.data is a string if s.processData is true) + * 3) key is the dataType + * 4) the catchall symbol "*" can be used + * 5) execution will start with transport dataType and THEN continue down to "*" if needed + */ + prefilters = {}, + + /* Transports bindings + * 1) key is the dataType + * 2) the catchall symbol "*" can be used + * 3) selection will start with transport dataType and THEN go to "*" if needed + */ + transports = {}, + + // Avoid comment-prolog char sequence (#10098); must appease lint and evade compression + allTypes = "*/".concat( "*" ), + + // Anchor tag for parsing the document origin + originAnchor = document.createElement( "a" ); + +originAnchor.href = location.href; + +// Base "constructor" for jQuery.ajaxPrefilter and jQuery.ajaxTransport +function addToPrefiltersOrTransports( structure ) { + + // dataTypeExpression is optional and defaults to "*" + return function( dataTypeExpression, func ) { + + if ( typeof dataTypeExpression !== "string" ) { + func = dataTypeExpression; + dataTypeExpression = "*"; + } + + var dataType, + i = 0, + dataTypes = dataTypeExpression.toLowerCase().match( rnothtmlwhite ) || []; + + if ( isFunction( func ) ) { + + // For each dataType in the dataTypeExpression + while ( ( dataType = dataTypes[ i++ ] ) ) { + + // Prepend if requested + if ( dataType[ 0 ] === "+" ) { + dataType = dataType.slice( 1 ) || "*"; + ( structure[ dataType ] = structure[ dataType ] || [] ).unshift( func ); + + // Otherwise append + } else { + ( structure[ dataType ] = structure[ dataType ] || [] ).push( func ); + } + } + } + }; +} + +// Base inspection function for prefilters and transports +function inspectPrefiltersOrTransports( structure, options, originalOptions, jqXHR ) { + + var inspected = {}, + seekingTransport = ( structure === transports ); + + function inspect( dataType ) { + var selected; + inspected[ dataType ] = true; + jQuery.each( structure[ dataType ] || [], function( _, prefilterOrFactory ) { + var dataTypeOrTransport = prefilterOrFactory( options, originalOptions, jqXHR ); + if ( typeof dataTypeOrTransport === "string" && + !seekingTransport && !inspected[ dataTypeOrTransport ] ) { + + options.dataTypes.unshift( dataTypeOrTransport ); + inspect( dataTypeOrTransport ); + return false; + } else if ( seekingTransport ) { + return !( selected = dataTypeOrTransport ); + } + } ); + return selected; + } + + return inspect( options.dataTypes[ 0 ] ) || !inspected[ "*" ] && inspect( "*" ); +} + +// A special extend for ajax options +// that takes "flat" options (not to be deep extended) +// Fixes #9887 +function ajaxExtend( target, src ) { + var key, deep, + flatOptions = jQuery.ajaxSettings.flatOptions || {}; + + for ( key in src ) { + if ( src[ key ] !== undefined ) { + ( flatOptions[ key ] ? target : ( deep || ( deep = {} ) ) )[ key ] = src[ key ]; + } + } + if ( deep ) { + jQuery.extend( true, target, deep ); + } + + return target; +} + +/* Handles responses to an ajax request: + * - finds the right dataType (mediates between content-type and expected dataType) + * - returns the corresponding response + */ +function ajaxHandleResponses( s, jqXHR, responses ) { + + var ct, type, finalDataType, firstDataType, + contents = s.contents, + dataTypes = s.dataTypes; + + // Remove auto dataType and get content-type in the process + while ( dataTypes[ 0 ] === "*" ) { + dataTypes.shift(); + if ( ct === undefined ) { + ct = s.mimeType || jqXHR.getResponseHeader( "Content-Type" ); + } + } + + // Check if we're dealing with a known content-type + if ( ct ) { + for ( type in contents ) { + if ( contents[ type ] && contents[ type ].test( ct ) ) { + dataTypes.unshift( type ); + break; + } + } + } + + // Check to see if we have a response for the expected dataType + if ( dataTypes[ 0 ] in responses ) { + finalDataType = dataTypes[ 0 ]; + } else { + + // Try convertible dataTypes + for ( type in responses ) { + if ( !dataTypes[ 0 ] || s.converters[ type + " " + dataTypes[ 0 ] ] ) { + finalDataType = type; + break; + } + if ( !firstDataType ) { + firstDataType = type; + } + } + + // Or just use first one + finalDataType = finalDataType || firstDataType; + } + + // If we found a dataType + // We add the dataType to the list if needed + // and return the corresponding response + if ( finalDataType ) { + if ( finalDataType !== dataTypes[ 0 ] ) { + dataTypes.unshift( finalDataType ); + } + return responses[ finalDataType ]; + } +} + +/* Chain conversions given the request and the original response + * Also sets the responseXXX fields on the jqXHR instance + */ +function ajaxConvert( s, response, jqXHR, isSuccess ) { + var conv2, current, conv, tmp, prev, + converters = {}, + + // Work with a copy of dataTypes in case we need to modify it for conversion + dataTypes = s.dataTypes.slice(); + + // Create converters map with lowercased keys + if ( dataTypes[ 1 ] ) { + for ( conv in s.converters ) { + converters[ conv.toLowerCase() ] = s.converters[ conv ]; + } + } + + current = dataTypes.shift(); + + // Convert to each sequential dataType + while ( current ) { + + if ( s.responseFields[ current ] ) { + jqXHR[ s.responseFields[ current ] ] = response; + } + + // Apply the dataFilter if provided + if ( !prev && isSuccess && s.dataFilter ) { + response = s.dataFilter( response, s.dataType ); + } + + prev = current; + current = dataTypes.shift(); + + if ( current ) { + + // There's only work to do if current dataType is non-auto + if ( current === "*" ) { + + current = prev; + + // Convert response if prev dataType is non-auto and differs from current + } else if ( prev !== "*" && prev !== current ) { + + // Seek a direct converter + conv = converters[ prev + " " + current ] || converters[ "* " + current ]; + + // If none found, seek a pair + if ( !conv ) { + for ( conv2 in converters ) { + + // If conv2 outputs current + tmp = conv2.split( " " ); + if ( tmp[ 1 ] === current ) { + + // If prev can be converted to accepted input + conv = converters[ prev + " " + tmp[ 0 ] ] || + converters[ "* " + tmp[ 0 ] ]; + if ( conv ) { + + // Condense equivalence converters + if ( conv === true ) { + conv = converters[ conv2 ]; + + // Otherwise, insert the intermediate dataType + } else if ( converters[ conv2 ] !== true ) { + current = tmp[ 0 ]; + dataTypes.unshift( tmp[ 1 ] ); + } + break; + } + } + } + } + + // Apply converter (if not an equivalence) + if ( conv !== true ) { + + // Unless errors are allowed to bubble, catch and return them + if ( conv && s.throws ) { + response = conv( response ); + } else { + try { + response = conv( response ); + } catch ( e ) { + return { + state: "parsererror", + error: conv ? e : "No conversion from " + prev + " to " + current + }; + } + } + } + } + } + } + + return { state: "success", data: response }; +} + +jQuery.extend( { + + // Counter for holding the number of active queries + active: 0, + + // Last-Modified header cache for next request + lastModified: {}, + etag: {}, + + ajaxSettings: { + url: location.href, + type: "GET", + isLocal: rlocalProtocol.test( location.protocol ), + global: true, + processData: true, + async: true, + contentType: "application/x-www-form-urlencoded; charset=UTF-8", + + /* + timeout: 0, + data: null, + dataType: null, + username: null, + password: null, + cache: null, + throws: false, + traditional: false, + headers: {}, + */ + + accepts: { + "*": allTypes, + text: "text/plain", + html: "text/html", + xml: "application/xml, text/xml", + json: "application/json, text/javascript" + }, + + contents: { + xml: /\bxml\b/, + html: /\bhtml/, + json: /\bjson\b/ + }, + + responseFields: { + xml: "responseXML", + text: "responseText", + json: "responseJSON" + }, + + // Data converters + // Keys separate source (or catchall "*") and destination types with a single space + converters: { + + // Convert anything to text + "* text": String, + + // Text to html (true = no transformation) + "text html": true, + + // Evaluate text as a json expression + "text json": JSON.parse, + + // Parse text as xml + "text xml": jQuery.parseXML + }, + + // For options that shouldn't be deep extended: + // you can add your own custom options here if + // and when you create one that shouldn't be + // deep extended (see ajaxExtend) + flatOptions: { + url: true, + context: true + } + }, + + // Creates a full fledged settings object into target + // with both ajaxSettings and settings fields. + // If target is omitted, writes into ajaxSettings. + ajaxSetup: function( target, settings ) { + return settings ? + + // Building a settings object + ajaxExtend( ajaxExtend( target, jQuery.ajaxSettings ), settings ) : + + // Extending ajaxSettings + ajaxExtend( jQuery.ajaxSettings, target ); + }, + + ajaxPrefilter: addToPrefiltersOrTransports( prefilters ), + ajaxTransport: addToPrefiltersOrTransports( transports ), + + // Main method + ajax: function( url, options ) { + + // If url is an object, simulate pre-1.5 signature + if ( typeof url === "object" ) { + options = url; + url = undefined; + } + + // Force options to be an object + options = options || {}; + + var transport, + + // URL without anti-cache param + cacheURL, + + // Response headers + responseHeadersString, + responseHeaders, + + // timeout handle + timeoutTimer, + + // Url cleanup var + urlAnchor, + + // Request state (becomes false upon send and true upon completion) + completed, + + // To know if global events are to be dispatched + fireGlobals, + + // Loop variable + i, + + // uncached part of the url + uncached, + + // Create the final options object + s = jQuery.ajaxSetup( {}, options ), + + // Callbacks context + callbackContext = s.context || s, + + // Context for global events is callbackContext if it is a DOM node or jQuery collection + globalEventContext = s.context && + ( callbackContext.nodeType || callbackContext.jquery ) ? + jQuery( callbackContext ) : + jQuery.event, + + // Deferreds + deferred = jQuery.Deferred(), + completeDeferred = jQuery.Callbacks( "once memory" ), + + // Status-dependent callbacks + statusCode = s.statusCode || {}, + + // Headers (they are sent all at once) + requestHeaders = {}, + requestHeadersNames = {}, + + // Default abort message + strAbort = "canceled", + + // Fake xhr + jqXHR = { + readyState: 0, + + // Builds headers hashtable if needed + getResponseHeader: function( key ) { + var match; + if ( completed ) { + if ( !responseHeaders ) { + responseHeaders = {}; + while ( ( match = rheaders.exec( responseHeadersString ) ) ) { + responseHeaders[ match[ 1 ].toLowerCase() + " " ] = + ( responseHeaders[ match[ 1 ].toLowerCase() + " " ] || [] ) + .concat( match[ 2 ] ); + } + } + match = responseHeaders[ key.toLowerCase() + " " ]; + } + return match == null ? null : match.join( ", " ); + }, + + // Raw string + getAllResponseHeaders: function() { + return completed ? responseHeadersString : null; + }, + + // Caches the header + setRequestHeader: function( name, value ) { + if ( completed == null ) { + name = requestHeadersNames[ name.toLowerCase() ] = + requestHeadersNames[ name.toLowerCase() ] || name; + requestHeaders[ name ] = value; + } + return this; + }, + + // Overrides response content-type header + overrideMimeType: function( type ) { + if ( completed == null ) { + s.mimeType = type; + } + return this; + }, + + // Status-dependent callbacks + statusCode: function( map ) { + var code; + if ( map ) { + if ( completed ) { + + // Execute the appropriate callbacks + jqXHR.always( map[ jqXHR.status ] ); + } else { + + // Lazy-add the new callbacks in a way that preserves old ones + for ( code in map ) { + statusCode[ code ] = [ statusCode[ code ], map[ code ] ]; + } + } + } + return this; + }, + + // Cancel the request + abort: function( statusText ) { + var finalText = statusText || strAbort; + if ( transport ) { + transport.abort( finalText ); + } + done( 0, finalText ); + return this; + } + }; + + // Attach deferreds + deferred.promise( jqXHR ); + + // Add protocol if not provided (prefilters might expect it) + // Handle falsy url in the settings object (#10093: consistency with old signature) + // We also use the url parameter if available + s.url = ( ( url || s.url || location.href ) + "" ) + .replace( rprotocol, location.protocol + "//" ); + + // Alias method option to type as per ticket #12004 + s.type = options.method || options.type || s.method || s.type; + + // Extract dataTypes list + s.dataTypes = ( s.dataType || "*" ).toLowerCase().match( rnothtmlwhite ) || [ "" ]; + + // A cross-domain request is in order when the origin doesn't match the current origin. + if ( s.crossDomain == null ) { + urlAnchor = document.createElement( "a" ); + + // Support: IE <=8 - 11, Edge 12 - 15 + // IE throws exception on accessing the href property if url is malformed, + // e.g. http://example.com:80x/ + try { + urlAnchor.href = s.url; + + // Support: IE <=8 - 11 only + // Anchor's host property isn't correctly set when s.url is relative + urlAnchor.href = urlAnchor.href; + s.crossDomain = originAnchor.protocol + "//" + originAnchor.host !== + urlAnchor.protocol + "//" + urlAnchor.host; + } catch ( e ) { + + // If there is an error parsing the URL, assume it is crossDomain, + // it can be rejected by the transport if it is invalid + s.crossDomain = true; + } + } + + // Convert data if not already a string + if ( s.data && s.processData && typeof s.data !== "string" ) { + s.data = jQuery.param( s.data, s.traditional ); + } + + // Apply prefilters + inspectPrefiltersOrTransports( prefilters, s, options, jqXHR ); + + // If request was aborted inside a prefilter, stop there + if ( completed ) { + return jqXHR; + } + + // We can fire global events as of now if asked to + // Don't fire events if jQuery.event is undefined in an AMD-usage scenario (#15118) + fireGlobals = jQuery.event && s.global; + + // Watch for a new set of requests + if ( fireGlobals && jQuery.active++ === 0 ) { + jQuery.event.trigger( "ajaxStart" ); + } + + // Uppercase the type + s.type = s.type.toUpperCase(); + + // Determine if request has content + s.hasContent = !rnoContent.test( s.type ); + + // Save the URL in case we're toying with the If-Modified-Since + // and/or If-None-Match header later on + // Remove hash to simplify url manipulation + cacheURL = s.url.replace( rhash, "" ); + + // More options handling for requests with no content + if ( !s.hasContent ) { + + // Remember the hash so we can put it back + uncached = s.url.slice( cacheURL.length ); + + // If data is available and should be processed, append data to url + if ( s.data && ( s.processData || typeof s.data === "string" ) ) { + cacheURL += ( rquery.test( cacheURL ) ? "&" : "?" ) + s.data; + + // #9682: remove data so that it's not used in an eventual retry + delete s.data; + } + + // Add or update anti-cache param if needed + if ( s.cache === false ) { + cacheURL = cacheURL.replace( rantiCache, "$1" ); + uncached = ( rquery.test( cacheURL ) ? "&" : "?" ) + "_=" + ( nonce.guid++ ) + + uncached; + } + + // Put hash and anti-cache on the URL that will be requested (gh-1732) + s.url = cacheURL + uncached; + + // Change '%20' to '+' if this is encoded form body content (gh-2658) + } else if ( s.data && s.processData && + ( s.contentType || "" ).indexOf( "application/x-www-form-urlencoded" ) === 0 ) { + s.data = s.data.replace( r20, "+" ); + } + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + if ( jQuery.lastModified[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-Modified-Since", jQuery.lastModified[ cacheURL ] ); + } + if ( jQuery.etag[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-None-Match", jQuery.etag[ cacheURL ] ); + } + } + + // Set the correct header, if data is being sent + if ( s.data && s.hasContent && s.contentType !== false || options.contentType ) { + jqXHR.setRequestHeader( "Content-Type", s.contentType ); + } + + // Set the Accepts header for the server, depending on the dataType + jqXHR.setRequestHeader( + "Accept", + s.dataTypes[ 0 ] && s.accepts[ s.dataTypes[ 0 ] ] ? + s.accepts[ s.dataTypes[ 0 ] ] + + ( s.dataTypes[ 0 ] !== "*" ? ", " + allTypes + "; q=0.01" : "" ) : + s.accepts[ "*" ] + ); + + // Check for headers option + for ( i in s.headers ) { + jqXHR.setRequestHeader( i, s.headers[ i ] ); + } + + // Allow custom headers/mimetypes and early abort + if ( s.beforeSend && + ( s.beforeSend.call( callbackContext, jqXHR, s ) === false || completed ) ) { + + // Abort if not done already and return + return jqXHR.abort(); + } + + // Aborting is no longer a cancellation + strAbort = "abort"; + + // Install callbacks on deferreds + completeDeferred.add( s.complete ); + jqXHR.done( s.success ); + jqXHR.fail( s.error ); + + // Get transport + transport = inspectPrefiltersOrTransports( transports, s, options, jqXHR ); + + // If no transport, we auto-abort + if ( !transport ) { + done( -1, "No Transport" ); + } else { + jqXHR.readyState = 1; + + // Send global event + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxSend", [ jqXHR, s ] ); + } + + // If request was aborted inside ajaxSend, stop there + if ( completed ) { + return jqXHR; + } + + // Timeout + if ( s.async && s.timeout > 0 ) { + timeoutTimer = window.setTimeout( function() { + jqXHR.abort( "timeout" ); + }, s.timeout ); + } + + try { + completed = false; + transport.send( requestHeaders, done ); + } catch ( e ) { + + // Rethrow post-completion exceptions + if ( completed ) { + throw e; + } + + // Propagate others as results + done( -1, e ); + } + } + + // Callback for when everything is done + function done( status, nativeStatusText, responses, headers ) { + var isSuccess, success, error, response, modified, + statusText = nativeStatusText; + + // Ignore repeat invocations + if ( completed ) { + return; + } + + completed = true; + + // Clear timeout if it exists + if ( timeoutTimer ) { + window.clearTimeout( timeoutTimer ); + } + + // Dereference transport for early garbage collection + // (no matter how long the jqXHR object will be used) + transport = undefined; + + // Cache response headers + responseHeadersString = headers || ""; + + // Set readyState + jqXHR.readyState = status > 0 ? 4 : 0; + + // Determine if successful + isSuccess = status >= 200 && status < 300 || status === 304; + + // Get response data + if ( responses ) { + response = ajaxHandleResponses( s, jqXHR, responses ); + } + + // Use a noop converter for missing script but not if jsonp + if ( !isSuccess && + jQuery.inArray( "script", s.dataTypes ) > -1 && + jQuery.inArray( "json", s.dataTypes ) < 0 ) { + s.converters[ "text script" ] = function() {}; + } + + // Convert no matter what (that way responseXXX fields are always set) + response = ajaxConvert( s, response, jqXHR, isSuccess ); + + // If successful, handle type chaining + if ( isSuccess ) { + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + modified = jqXHR.getResponseHeader( "Last-Modified" ); + if ( modified ) { + jQuery.lastModified[ cacheURL ] = modified; + } + modified = jqXHR.getResponseHeader( "etag" ); + if ( modified ) { + jQuery.etag[ cacheURL ] = modified; + } + } + + // if no content + if ( status === 204 || s.type === "HEAD" ) { + statusText = "nocontent"; + + // if not modified + } else if ( status === 304 ) { + statusText = "notmodified"; + + // If we have data, let's convert it + } else { + statusText = response.state; + success = response.data; + error = response.error; + isSuccess = !error; + } + } else { + + // Extract error from statusText and normalize for non-aborts + error = statusText; + if ( status || !statusText ) { + statusText = "error"; + if ( status < 0 ) { + status = 0; + } + } + } + + // Set data for the fake xhr object + jqXHR.status = status; + jqXHR.statusText = ( nativeStatusText || statusText ) + ""; + + // Success/Error + if ( isSuccess ) { + deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] ); + } else { + deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] ); + } + + // Status-dependent callbacks + jqXHR.statusCode( statusCode ); + statusCode = undefined; + + if ( fireGlobals ) { + globalEventContext.trigger( isSuccess ? "ajaxSuccess" : "ajaxError", + [ jqXHR, s, isSuccess ? success : error ] ); + } + + // Complete + completeDeferred.fireWith( callbackContext, [ jqXHR, statusText ] ); + + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxComplete", [ jqXHR, s ] ); + + // Handle the global AJAX counter + if ( !( --jQuery.active ) ) { + jQuery.event.trigger( "ajaxStop" ); + } + } + } + + return jqXHR; + }, + + getJSON: function( url, data, callback ) { + return jQuery.get( url, data, callback, "json" ); + }, + + getScript: function( url, callback ) { + return jQuery.get( url, undefined, callback, "script" ); + } +} ); + +jQuery.each( [ "get", "post" ], function( _i, method ) { + jQuery[ method ] = function( url, data, callback, type ) { + + // Shift arguments if data argument was omitted + if ( isFunction( data ) ) { + type = type || callback; + callback = data; + data = undefined; + } + + // The url can be an options object (which then must have .url) + return jQuery.ajax( jQuery.extend( { + url: url, + type: method, + dataType: type, + data: data, + success: callback + }, jQuery.isPlainObject( url ) && url ) ); + }; +} ); + +jQuery.ajaxPrefilter( function( s ) { + var i; + for ( i in s.headers ) { + if ( i.toLowerCase() === "content-type" ) { + s.contentType = s.headers[ i ] || ""; + } + } +} ); + + +jQuery._evalUrl = function( url, options, doc ) { + return jQuery.ajax( { + url: url, + + // Make this explicit, since user can override this through ajaxSetup (#11264) + type: "GET", + dataType: "script", + cache: true, + async: false, + global: false, + + // Only evaluate the response if it is successful (gh-4126) + // dataFilter is not invoked for failure responses, so using it instead + // of the default converter is kludgy but it works. + converters: { + "text script": function() {} + }, + dataFilter: function( response ) { + jQuery.globalEval( response, options, doc ); + } + } ); +}; + + +jQuery.fn.extend( { + wrapAll: function( html ) { + var wrap; + + if ( this[ 0 ] ) { + if ( isFunction( html ) ) { + html = html.call( this[ 0 ] ); + } + + // The elements to wrap the target around + wrap = jQuery( html, this[ 0 ].ownerDocument ).eq( 0 ).clone( true ); + + if ( this[ 0 ].parentNode ) { + wrap.insertBefore( this[ 0 ] ); + } + + wrap.map( function() { + var elem = this; + + while ( elem.firstElementChild ) { + elem = elem.firstElementChild; + } + + return elem; + } ).append( this ); + } + + return this; + }, + + wrapInner: function( html ) { + if ( isFunction( html ) ) { + return this.each( function( i ) { + jQuery( this ).wrapInner( html.call( this, i ) ); + } ); + } + + return this.each( function() { + var self = jQuery( this ), + contents = self.contents(); + + if ( contents.length ) { + contents.wrapAll( html ); + + } else { + self.append( html ); + } + } ); + }, + + wrap: function( html ) { + var htmlIsFunction = isFunction( html ); + + return this.each( function( i ) { + jQuery( this ).wrapAll( htmlIsFunction ? html.call( this, i ) : html ); + } ); + }, + + unwrap: function( selector ) { + this.parent( selector ).not( "body" ).each( function() { + jQuery( this ).replaceWith( this.childNodes ); + } ); + return this; + } +} ); + + +jQuery.expr.pseudos.hidden = function( elem ) { + return !jQuery.expr.pseudos.visible( elem ); +}; +jQuery.expr.pseudos.visible = function( elem ) { + return !!( elem.offsetWidth || elem.offsetHeight || elem.getClientRects().length ); +}; + + + + +jQuery.ajaxSettings.xhr = function() { + try { + return new window.XMLHttpRequest(); + } catch ( e ) {} +}; + +var xhrSuccessStatus = { + + // File protocol always yields status code 0, assume 200 + 0: 200, + + // Support: IE <=9 only + // #1450: sometimes IE returns 1223 when it should be 204 + 1223: 204 + }, + xhrSupported = jQuery.ajaxSettings.xhr(); + +support.cors = !!xhrSupported && ( "withCredentials" in xhrSupported ); +support.ajax = xhrSupported = !!xhrSupported; + +jQuery.ajaxTransport( function( options ) { + var callback, errorCallback; + + // Cross domain only allowed if supported through XMLHttpRequest + if ( support.cors || xhrSupported && !options.crossDomain ) { + return { + send: function( headers, complete ) { + var i, + xhr = options.xhr(); + + xhr.open( + options.type, + options.url, + options.async, + options.username, + options.password + ); + + // Apply custom fields if provided + if ( options.xhrFields ) { + for ( i in options.xhrFields ) { + xhr[ i ] = options.xhrFields[ i ]; + } + } + + // Override mime type if needed + if ( options.mimeType && xhr.overrideMimeType ) { + xhr.overrideMimeType( options.mimeType ); + } + + // X-Requested-With header + // For cross-domain requests, seeing as conditions for a preflight are + // akin to a jigsaw puzzle, we simply never set it to be sure. + // (it can always be set on a per-request basis or even using ajaxSetup) + // For same-domain requests, won't change header if already provided. + if ( !options.crossDomain && !headers[ "X-Requested-With" ] ) { + headers[ "X-Requested-With" ] = "XMLHttpRequest"; + } + + // Set headers + for ( i in headers ) { + xhr.setRequestHeader( i, headers[ i ] ); + } + + // Callback + callback = function( type ) { + return function() { + if ( callback ) { + callback = errorCallback = xhr.onload = + xhr.onerror = xhr.onabort = xhr.ontimeout = + xhr.onreadystatechange = null; + + if ( type === "abort" ) { + xhr.abort(); + } else if ( type === "error" ) { + + // Support: IE <=9 only + // On a manual native abort, IE9 throws + // errors on any property access that is not readyState + if ( typeof xhr.status !== "number" ) { + complete( 0, "error" ); + } else { + complete( + + // File: protocol always yields status 0; see #8605, #14207 + xhr.status, + xhr.statusText + ); + } + } else { + complete( + xhrSuccessStatus[ xhr.status ] || xhr.status, + xhr.statusText, + + // Support: IE <=9 only + // IE9 has no XHR2 but throws on binary (trac-11426) + // For XHR2 non-text, let the caller handle it (gh-2498) + ( xhr.responseType || "text" ) !== "text" || + typeof xhr.responseText !== "string" ? + { binary: xhr.response } : + { text: xhr.responseText }, + xhr.getAllResponseHeaders() + ); + } + } + }; + }; + + // Listen to events + xhr.onload = callback(); + errorCallback = xhr.onerror = xhr.ontimeout = callback( "error" ); + + // Support: IE 9 only + // Use onreadystatechange to replace onabort + // to handle uncaught aborts + if ( xhr.onabort !== undefined ) { + xhr.onabort = errorCallback; + } else { + xhr.onreadystatechange = function() { + + // Check readyState before timeout as it changes + if ( xhr.readyState === 4 ) { + + // Allow onerror to be called first, + // but that will not handle a native abort + // Also, save errorCallback to a variable + // as xhr.onerror cannot be accessed + window.setTimeout( function() { + if ( callback ) { + errorCallback(); + } + } ); + } + }; + } + + // Create the abort callback + callback = callback( "abort" ); + + try { + + // Do send the request (this may raise an exception) + xhr.send( options.hasContent && options.data || null ); + } catch ( e ) { + + // #14683: Only rethrow if this hasn't been notified as an error yet + if ( callback ) { + throw e; + } + } + }, + + abort: function() { + if ( callback ) { + callback(); + } + } + }; + } +} ); + + + + +// Prevent auto-execution of scripts when no explicit dataType was provided (See gh-2432) +jQuery.ajaxPrefilter( function( s ) { + if ( s.crossDomain ) { + s.contents.script = false; + } +} ); + +// Install script dataType +jQuery.ajaxSetup( { + accepts: { + script: "text/javascript, application/javascript, " + + "application/ecmascript, application/x-ecmascript" + }, + contents: { + script: /\b(?:java|ecma)script\b/ + }, + converters: { + "text script": function( text ) { + jQuery.globalEval( text ); + return text; + } + } +} ); + +// Handle cache's special case and crossDomain +jQuery.ajaxPrefilter( "script", function( s ) { + if ( s.cache === undefined ) { + s.cache = false; + } + if ( s.crossDomain ) { + s.type = "GET"; + } +} ); + +// Bind script tag hack transport +jQuery.ajaxTransport( "script", function( s ) { + + // This transport only deals with cross domain or forced-by-attrs requests + if ( s.crossDomain || s.scriptAttrs ) { + var script, callback; + return { + send: function( _, complete ) { + script = jQuery( " + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Accounts

+

The accounts section is for the management of user accounts, access permissions, and tenants.

+
+

Users

+

Description of User account fields:

+
    +
  • Username - Unique username.

  • +
  • Full Name - Full Name of the user.

  • +
  • E-mail - The email address of the user. Also used for system notifications and for password retrieval.

  • +
  • E-mail CC - Send copies of email notifications to this address.

  • +
  • Phone Number - User’s phone number.

  • +
  • Company - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers.

  • +
  • Position - Position within the company.

  • +
  • User Role - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate.

  • +
  • Permission Group - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used)

  • +
  • +Tenant - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used)

  • +
+

Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin.

+_images/users.png +

Password: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side.

+

Example: Listing of user accounts.

+_images/password.png +
+
+

Tenants

+

IP addresses and Switch Ports are network resources that can be assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs) as part of DevOps/NetOps pipeline.

+

A Tenant has just two fields, the unique name and custom description.

+

Example: Adding a tenant.

+_images/tenants.png +
+
+

Permission Groups

+

Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections.

+

Example: Permission Group.

+_images/permission_group.png +
+
+

User Roles

+

Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username.

+_images/user_role.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/concepts.html b/en/2.9/concepts.html new file mode 100644 index 0000000000..5352580841 --- /dev/null +++ b/en/2.9/concepts.html @@ -0,0 +1,477 @@ + + + + + + + + + + + Concepts — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Concepts

+
+

Introduction to Netris

+

Netris is an automatic netops software for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network’s health and either apply software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation.

+
+
+

What is Netris Controller

+

Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud.

+

Diagram: High level Netris architecture.

+_images/netris_controller_diagram.png +
    +
  • Controller HA. We highly recommend running more than one copy of the controller for database replication.

  • +
  • Multiple sites. Netris is designed to operate multiple sites with just a single controller with HA.

  • +
  • What if the controller is unreachable. Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers’ core operation will not be affected.

  • +
+
+
+

Netris Switch Agent

+

Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet.

+
+
+

Netris SoftGate

+

Netris SoftGate is automatic configuration software and reference architecture for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), and site-to-site VPN function on a regular x86 server with a SmartNIC card.

+

Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2.

+

The server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities.

+

Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels.

+

Diagram: Netris SoftGate high level architecture

+_images/softgate_diagram.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/controller-initial-configuration.html b/en/2.9/controller-initial-configuration.html new file mode 100644 index 0000000000..307e42eaba --- /dev/null +++ b/en/2.9/controller-initial-configuration.html @@ -0,0 +1,576 @@ + + + + + + + + + + + Controller initial configuration — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Controller initial configuration
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Controller initial configuration

+
+

Definitions

+
    +
  • User - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is netris, with password newNet0ps.

  • +
  • Tenant - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request & manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs) DevOps/NetOps pipeline.

  • +
  • Permission Group - List of permissions on a per section basis can be attached individually to a User or a User Role.

  • +
  • User Role - Group of user permissions and tenants for role-based access control.

  • +
  • Site - Each separate deployment (each data center) should be defined as a Site. All network units and resources are attached to a site. Netris Controller comes with a “default” site preconfigured. Site entry defines global attributes such as; AS numbers, default ACL policy, Site Mesh (site to site VPN) type.

  • +
  • Subnet - IPv4/IPv6 address resources linked to Sites and Tenants.

  • +
  • Switch Port - Physical ports of all switches attached to the system. Switch port objects represent statuses, take basic parameters, and are assigned to Tenants.

  • +
  • Inventory - This is an inventory of all network units that are operated using Netris Agent.

  • +
  • E-BGP - Is for defining all External BGP peers (iBGP and eBGP).

  • +
+
+
+

Subnets

+

It is required to define at least two subnets to get started. One subnet is for the management interfaces, another for the loopback addresses. Every network unit managed with Netris should have at least one management IP and at least one loopback IP. Loopback IP addresses are used for network unit identification by network protocols and by Netris Agent/Controller. There’s no need for defining any IP addresses for the switch-to-switch links. Netris is using IPv6 link-local addresses for all switch-to-switch communication.

+
+

Example: (IP addresses used are just examples, please replace them following your IP planning.)

+

In NET->Subnets section of the Netris Controller GUI, you can add new subnet entries. Subnets are of 2 types of allocation and assignment. Allocations are the large blocks of IP resources assigned to the organization. Assignments are IP blocks that are smaller blocks inside the allocation and can be used by services or policies that yet to be defined.

+
    +
  1. Adding a new allocation. In this example, 10.0.0.0/8 is used as a large block of allocation. You can add as many allocations as required.

  2. +
+_images/allocation.png +
    +
  1. Adding two new assignments.

  2. +
+
    +
  • 10.254.96.0/24 (netManagement) assigned to the tenant “Admin” and available for the site “Default”.

  • +
  • 10.254.97.0/24 (netLoopbacks) assigned to the tenant “Admin” and available for the site “Default”.

  • +
+_images/assignment.png +_images/assignment2.png +

Screenshot: Listing of the Subnets section after adding the new objects.

+_images/subnet_listing.png +
+

+

+
+
+
+
+

Inventory Profiles

+

Inventory profiles define access security, timezone, DNS, NTP settings profiles for network switches and SoftGate nodes. +To create a new Inventory profile, click +Add under the Net→Inventory Profiles section.

+
+

Fields descriptions:

+
    +
  • Name - Profile name.

  • +
  • Description - Free text description.

  • +
  • Allow SSH from IPv4 - List of IPv4 subnets allowed to ssh (one address per line)

  • +
  • Allow SSH from IPv6 - List of IPv6 subnets allowed to ssh (one address per line)

  • +
  • Timezone - Devices using this inventory profile will adjust their system time to the selected timezone.

  • +
  • NTP servers - List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate.

  • +
  • DNS servers - List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate.

  • +
+

Example: In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup).

+_images/inventory_profile.png +
+
+
+

Adding Switches to Topology

+

You need to define every switch in the Net→Topology section. To add a switch, please go to Net→Topology and click +Add.

+
    +
  • Name - Descriptive name.

  • +
  • Owner Tenant - Tenant(typically Admin) who administers this node.

  • +
  • Description - Free text description.

  • +
  • Hardware Type - For switches: Spine Switch or Leaf Switch.

  • +
  • NOS* - Network operating system. Cumulus Linux, Ubuntu SwitchDev (Nvidia Mellanox only), SONiC (not for production use yet)

  • +
  • Site* - The site where the switch belongs.

  • +
  • Inventory Profile - Reference to Timezone, DNS, NTP, and Security features profile.

  • +
  • IP Address* - IPv4 address for the loopback interface.

  • +
  • Management IP address - IPv4 address for the out of band management interface.

  • +
  • Zero-touch provisioning - Automatically install the NOS. (Experimental in this version)

  • +
  • MAC address - Out of band management interface MAC address used for zero-touch provisioning. (Experimental in this version)

  • +
  • The number of ports - It is required for the topology manager. WIll be synced to the real number of Switch Ports when Netris Switch Agent establishes the very first connection with the Netris Controller.

  • +
+

Example: Adding a spine switch w/ Cumulus Linux.

+_images/new_hardware.png +

Tip: You can drag/move the units to your desired positions and click “Save positions”.

+

Note: Repeat this process to define all your switches.

+
+
+

Topology Manager

+

The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents will configure the underlying network devices according to this topology dynamically and start watching against potential failures.

+

To define the links, right-click on the spine switch, then click create a link. Select the “from port,” then “to device” and “port.” See the example below.

+_images/create_link.png +

All links require definition in the topology manager. Topology links can also be described through a .yaml file when using Kubernetes CRD. (a GUI wizard is planned to be available in v2.10).

+_images/topology_manager.png +

Now when network units and links are defined, your network is automatically configured as long as physical connectivity is in place and Netris Agents can communicate with Netris Controller.

+
+
+

Hairpin (Cumulus only)

+

With Cumulus Linux only, we need to loop two ports on spine switches (hairpin cable) in the current release, usually two upstream (higher capacity) ports. We are planning to lift this requirement in the next Netris release (v2.10).

+

To define what ports will be used as a hairpin, navigate to Net→Switch Ports, or right-click on the spine switch, click Ports in Net–>Topology.

+

Example: Accessing Switch Ports from Net→Topology

+_images/switch_port.png +

For each spine switch, find the two ports that you are going to connect (loop/hairpin) and configure one port as a “hairpin l2” and another port as “hairpin l3”. The order doesn’t matter. The system needs to know which ports you have dedicated for the hairpin/loop on each spine switch. (do not do this for non-Cumulus switches) +| +| +Example: Editing Switch Port from Net→Switch Ports.

+_images/edit_switch_port.png +

Example: Setting port types to “hairpin l2” and “hairpin l3”.

+_images/hairpin.png +

Screenshot: Hairpin visualized in Net→Topology

+_images/hairpin_topology.png +
+
+

Adding SoftGate nodes to Topology

+

Every SoftGate node first needs to be defined in Netris Controller. +To add a SoftGate node, please go to Net→Topology and click +Add.

+
    +
  • Name - Descriptive name.

  • +
  • Owner Tenant - Tenant(typically Admin), who administers this node.

  • +
  • Description - Free text description.

  • +
  • Hardware Type - NFV node.

  • +
  • Site - The data center where the current SoftGate node belongs.

  • +
  • Inventory Profile - Profile describing the timezone, DNS, NTP, and Security features.

  • +
  • IP Address - IPv4 address for the loopback interface.

  • +
  • Management IP address - IPv4 address for the out of band management interface.

  • +
  • NFV Node Port - A physical port on a spine switch where the SoftGate node’s first SmartNIC port is connected. Typically each spine switch has one SoftGate node connected to it.

  • +
  • +NAT address - Public IP addresses to be used as global IP for SNAT/DNAT. (check Enabling NAT section of Network Policies chapter)

  • +
  • +NAT address pool - Public IP address subnets to be used as rolling global IP addresses for SNAT. (check Enabling NAT section of Network Policies chapter)

  • +
+

Example: Adding a SoftGate Node to Topology.

+_images/softgate_node.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/controller-installation.html b/en/2.9/controller-installation.html new file mode 100644 index 0000000000..61eac7af8e --- /dev/null +++ b/en/2.9/controller-installation.html @@ -0,0 +1,634 @@ + + + + + + + + + + + On-prem Netris Controller installation — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • On-prem Netris Controller installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

On-prem Netris Controller installation

+

Netris Controller can be hosted in Netris cloud, installed locally as a VM, or deployed as a Kubernetes application. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime.

+
+

KVM virtual machine

+
+
Minimal system requirements for the VM:
+
CPU - 8 Core
+
RAM - 16 Gb
+
Disk - 100Gb
+
Network - 1 virtual NIC
+
+
+
+

Installation steps for KVM hypervisor

+

If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04)

+
sudo apt-get install virt-manager
+
+
+
+
+

Netris Controller Installation steps

+
    +
  1. Download the Netris Controller image. (contact Netris support for repository access permissions).

  2. +
+
cd /var/lib/libvirt/images
+
+sudo wget http://img.netris.ai/netris-controller.qcow2
+
+
+
    +
  1. Download vm definition file.

  2. +
+
cd /etc/libvirt/qemu
+
+sudo wget http://img.netris.ai/netris-controller.xml
+
+
+
    +
  1. Define the KVM virtual machine

  2. +
+
sudo virsh define netris-controller.xml
+
+
+
+

Note

+

Netris controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below network interface configuration exam

+
+

Example: Network configuration on host (hypervisor) machine.

+
+

Note

+

replace <Physical NIC>, <host server management IP/prefix length> and <host server default gateway> +with the correct NIC and IP for your host machine.

+
+
sudo vim /etc/network/interfaces
+
+
+
#Physical NIC connected to the management network
+auto <Physical NIC>
+iface <Physical NIC> inet static
+                        address 0.0.0.0/0
+
+#bridge interface
+auto br-mgmt
+iface br-mgmt inet static
+                        address <host server management IP/prefix length>
+                        gateway <host server default gateway>
+                        bridge-ports <Physical NIC>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Set the virtual machine to autostart and start it.

  2. +
+
sudo virsh autostart netris-controller
+
+
+
sudo virsh start netris-controller
+
+
+
+
+

Accessing the Netris Controller

+

By default, Netris Controller will obtain an IP address from a DHCP server.

+

Below steps describe how to configure a static IP address for the Netris Controller.

+
    +
  1. Connecting to the VM console.

  2. +
+

default credentials. login: netris password: newNet0ps

+
sudo virsh console netris-controller
+
+
+
+

Note

+

Do not forget to change the default password (using passwd command).

+
+
    +
  1. Setting a static IP address.

  2. +
+

Edit network configuration file.

+
sudo vim /etc/network/interfaces
+
+
+

Example: IP configuration file.

+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <Netris Controller IP/prefix length>
+        gateway <Netris Controller default gateway>
+        dns-nameserver <a DNS server address>
+
+source /etc/network/interfaces.d/*
+
+
+

Reload the network config.

+
sudo ifreload -a
+
+
+
+

Note

+

Make sure Netris Controller has Internet access.

+
+
    +
  1. Reboot the controller

  2. +
+
sudo reboot
+
+
+

After reboot, the Netris Controller GUI should be accessible using a browser. Use netris/newNet0ps credentials.

+_images/credentials.png +

Don’t forget to change the default password by clicking your login name in the top right corner and then clicking “Change Password”.

+
+
+

Security hardening

+
+
Recommended for production use.
+

+
+
+

Changing the default GRPC authentication key.

+

Connect to the Netris Controller CLI (SSH or Console)

+

Tip: You can generate a random and secure key using sha256sum.

+
echo "<some random text here>" | sha256sum
+
+
+

example:

+
netris@iris:~$ echo "<some random text here>" | sha256sum
+6a284d55148f81728f932b28e9d020736c8f78e1950b3d576f6e679d90516df1  -
+
+
+

Set your newly generated secure key into Netris Controller.

+
sudo /opt/telescope/netris-set-auth.sh --key <your key>
+
+
+

Please store the auth key in a safe place as it will be required every time when installing Netris Agent for the switches and SoftGates.

+
+
+

Replacing the SSL certificate

+
    +
  1. Replace below file with your SSL certificate file.

  2. +
+
/etc/nginx/ssl/controller.cert.pem;
+
+
+
    +
  1. Replace below file with your SSL private key.

  2. +
+
/etc/nginx/ssl/controller.key.pem;
+
+
+
    +
  1. Restart Nginx service.

  2. +
+
systemctl restart nginx.service
+
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/controller-k8s-installation.html b/en/2.9/controller-k8s-installation.html new file mode 100644 index 0000000000..0988c5703b --- /dev/null +++ b/en/2.9/controller-k8s-installation.html @@ -0,0 +1,1168 @@ + + + + + + + + + + + Netris Controller Helm Chart — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris Controller Helm Chart
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Netris Controller Helm Chart

+ +
+

Prerequisites

+
    +
  • Kubernetes 1.12+

  • +
  • Helm 3.1+

  • +
  • PV provisioner support in the underlying infrastructure

  • +
+
+
+

Get Repo Info

+

Add the Netris Helm repository:

+
helm repo add netrisai https://netrisai.github.io/charts
+helm repo update
+
+
+
+
+

Installing the Chart

+

In order to install the Helm chart, you must follow these steps:

+

Create the namespace for netris-controller:

+
kubectl create namespace netris-controller
+
+
+

Generate strong auth key

+
export mystrongauthkey=$(date |base64 | md5sum | base64 | head -c 32)
+
+
+

Install helm chart with netris-controller

+
helm install netris-controller netrisai/netris-controller \
+  --namespace netris-controller \
+  --set app.ingress.hosts={my.domain.com} \
+  --set netris.authKey=$mystrongauthkey
+
+
+
+
+

Uninstalling the Chart

+

To uninstall/delete the netris-controller helm release:

+
helm uninstall netris-controller
+
+
+
+
+

Configuration

+

The following table lists the configurable parameters of the netris-controller chart and their default values.

+
+

Common parameters

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

nameOverride

String to partially override common.names.fullname template with a string (will prepend the release name)

nil

fullnameOverride

String to fully override common.names.fullname template with a string

nil

serviceAccount.create

Create a serviceAccount for the deployment

true

serviceAccount.name

Use the serviceAccount with the specified name

""

serviceAccount.annotations

Annotations to add to the service account

{}

podAnnotations

Pod annotations

{}

podSecurityContext

Pod Security Context

{}

securityContext

Containers security context

{}

resources

CPU/memory resource requests/limits

{}

nodeSelector

Node labels for pod assignment

{}

tolerations

Node tolerations for pod assignment

[]

affinity

Node affinity for pod assignment

{}

+
+
+

Netris-Controller common parameters

+ +++++ + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

netris.webLogin

Netris Controller GUI default login

netris

netris.webPassword

Netris Controller GUI default password

newNet0ps

netris.authKey

Netris Controller agents authentication key

mystrongkey

+
+
+

Netris-Controller app parameters

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

app.replicaCount

Number of replicas in app deployment

1

app.image.repository

Image repository

netrisai/xcaas-xcaas-web

app.image.tag

Image tag. Overrides the image tag whose default is the chart appVersion

""

app.image.pullPolicy

Image pull policy

IfNotPresent

app.imagePullSecrets

Reference to one or more secrets to be used when pulling images

[]

app.service.type

Kubernetes service type

ClusterIP

app.service.port

Kubernetes port where service is expose

80

app.service.portName

Name of the port on the service

http

app.ingress.enabled

Enables Ingress

true

app.ingress.annotations

Ingress annotations (values are templated)

{ kubernetes.io/ingress.class: nginx }

app.ingress.labels

Custom labels

{}

app.ingress.path

Ingress accepted path

/

app.ingress.pathType

Ingress type of path

Prefix

app.ingress.hosts

Ingress accepted hostnames

["chart-example.local"]

app.ingress.tls

Ingress TLS configuration

[]

app.autoscaling.enabled

Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics

false

app.autoscaling.minReplicas

Default min replicas for autoscaling

1

app.autoscaling.maxReplicas

Default max replicas for autoscaling

100

app.autoscaling.targetCPUUtilizationPercentage

The desired target CPU utilization for autoscaling

80

+
+
+

Netris-Controller grpc parameters

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

grpc.replicaCount

Number of replicas in grpc deployment

1

grpc.image.repository

Image repository

netrisai/xcaas-agent-api-server

grpc.image.tag

Image tag. Overrides the image tag whose default is the chart appVersion

""

grpc.image.pullPolicy

Image pull policy

IfNotPresent

grpc.imagePullSecrets

Reference to one or more secrets to be used when pulling images

[]

grpc.service.type

Kubernetes service type

ClusterIP

grpc.service.port

Kubernetes port where service is expose

443

grpc.service.portName

Name of the port on the service

grpc

grpc.autoscaling.enabled

Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics

false

grpc.autoscaling.minReplicas

Default min replicas for autoscaling

1

grpc.autoscaling.maxReplicas

Default max replicas for autoscaling

100

grpc.autoscaling.targetCPUUtilizationPercentage

The desired target CPU utilization for autoscaling

80

+
+
+

Netris-Controller telescope parameters

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

telescope.replicaCount

Number of replicas in telescope deployment

1

telescope.image.repository

Image repository

netrisai/xcaas-telescope-go

telescope.image.tag

Image tag. Overrides the image tag whose default is the chart appVersion

""

telescope.image.pullPolicy

Image pull policy

IfNotPresent

telescope.imagePullSecrets

Reference to one or more secrets to be used when pulling images

[]

telescope.service.type

Kubernetes service type

ClusterIP

telescope.service.port

Kubernetes port where service is expose

80

telescope.service.portName

Name of the port on the service

ws

telescope.service.securePort

Kubernetes secure port where service is expose

443

telescope.service.securePortName

Name of the secure port on the service

wss

telescope.autoscaling.enabled

Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics

false

telescope.autoscaling.minReplicas

Default min replicas for autoscaling

1

telescope.autoscaling.maxReplicas

Default max replicas for autoscaling

100

telescope.autoscaling.targetCPUUtilizationPercentage

The desired target CPU utilization for autoscaling

80

+
+
+

Netris-Controller k8s-watcher parameters

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

k8s-watcher.replicaCount

Number of replicas in k8s-watcher deployment

1

k8s-watcher.image.repository

Image repository

netrisai/xcaas-kuberis-k8-api-agent

k8s-watcher.image.tag

Image tag. Overrides the image tag whose default is the chart appVersion

""

k8s-watcher.image.pullPolicy

Image pull policy

IfNotPresent

k8s-watcher.imagePullSecrets

Reference to one or more secrets to be used when pulling images

[]

k8s-watcher.autoscaling.enabled

Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics

false

k8s-watcher.autoscaling.minReplicas

Default min replicas for autoscaling

1

k8s-watcher.autoscaling.maxReplicas

Default max replicas for autoscaling

100

k8s-watcher.autoscaling.targetCPUUtilizationPercentage

The desired target CPU utilization for autoscaling

80

+
+
+

Netris-Controller telescope-notifier parameters

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

telescope-notifier.replicaCount

Number of replicas in telescope-notifier deployment

1

telescope-notifier.image.repository

Image repository

netrisai/xcaas-xcaas-notifier

telescope-notifier.image.tag

Image tag. Overrides the image tag whose default is the chart appVersion

""

telescope-notifier.image.pullPolicy

Image pull policy

IfNotPresent

telescope-notifier.imagePullSecrets

Reference to one or more secrets to be used when pulling images

[]

telescope-notifier.autoscaling.enabled

Option to turn autoscaling on for app and specify params for HPA. Autoscaling needs metrics-server to access cpu metrics

false

telescope-notifier.autoscaling.minReplicas

Default min replicas for autoscaling

1

telescope-notifier.autoscaling.maxReplicas

Default max replicas for autoscaling

100

telescope-notifier.autoscaling.targetCPUUtilizationPercentage

The desired target CPU utilization for autoscaling

80

+
+
+

Mariadb parameters

+

Using default values from

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

mariadb.image.repository

MariaDB image name. We extended bitnami’s mariadb image with own plugin

netrisai/netris-mariadb

mariadb.image.tag

MariaDB image tag. (only 10.1 is supported)

10.1

mariadb.initdbScriptsConfigMap

ConfigMap with the initdb scripts.

netris-controller-initdb

mariadb.auth.database

Name for a database to create

netris

mariadb.auth.username

Name for a user to create

netris

mariadb.auth.password

Password for the new user

changeme

mariadb.auth.rootPassword

Password for the root user

changeme

+

Auth from existing secret not supported at the momment

+
+
+

Mongodb parameters

+

Using default values from

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Parameter

Description

Default

mongodb.useStatefulSet

Use StatefulSet instead of Deployment when deploying standalone

true

mongodb.initdbScriptsConfigMap

ConfigMap with the initdb scripts.

netris-controller-initdb-mongodb

mongodb.auth.database

Name for a database to create

netris

mongodb.auth.username

Name for a user to create

netris

mongodb.auth.password

Password for the new user

changeme

mongodb.auth.rootPassword

Password for the root user

changeme

+

Auth from existing secret not supported at the momment

+
+
+

Redis parameters

+

Using default values from

+ +++++ + + + + + + + + + + + + + + + + +

Parameter

Description

Default

redis.cluster.enabled

Use master-slave topology

false

redis.usePassword

Use password

false

+

Auth not supported at the momment

+
+
+

Smtp parameters

+

Using default values from

+ +++++ + + + + + + + + + + + + + + + + +

Parameter

Description

Default

smtp.config.DISABLE_IPV6

Disable IPv6

1

smtp.config.RELAY_NETWORKS

Relay networks. Change if your CNI use other subnets

:172.16.0.0/12:10.0.0.0/8:192.168.0.0/16

+
+
+

HAproxy parameters

+

Using default values from

+ +++++ + + + + + + + + + + + + + + + + +

Parameter

Description

Default

haproxy.enabled

Enable HAProxy. Used for exposing netris agents ports from single loadbalancer ip. Disable if you can’t have type:LoadBalancer service in cluster

true

haproxy.service.type

Kubernetes service type

LoadBalancer

+
+
+

Graphite parameters

+

Using default values from

+ +++++ + + + + + + + + + + + + + + + + +

Parameter

Description

Default

graphite.configMaps

Netris-Controller supported graphite’s config files

see in values.yaml

graphite.service.type

Kubernetes service type

ClusterIP

+
+
+
+

Usage

+

Specify each parameter using the –set key=value[,key=value] argument to helm install. For example,

+
helm install netris-controller netrisai/netris-controller \
+  --namespace netris-controller \
+  --set app.ingress.hosts={my.domain.com} \
+  --set netris.authKey=$mystrongauthkey \
+  --set mariadb.auth.rootPassword=my-root-password \
+  --set mariadb.auth.password=my-password \
+  --set mongodb.auth.rootPassword=my-root-password \
+  --set mongodb.auth.password=my-password
+
+
+

The above command sets netris-controller application ingress host to my.domain.com and sets generated netris.authKey. Additionally, it sets MariaDB and MongoDB root account password to my-root-password and user account password to my-password.

+

Alternatively, a YAML file that specifies the values for the parameters can be provided while installing the chart. For example,

+
helm install netris-controller netrisai/netris-controller --namespace netris-controller -f values.yaml
+
+
+

After installation use EXTERNAL-IP of haproxy service as --controller parameter in netris-setup

+
kubectl get svc -nnetris-controller |grep haproxy
+
+
+

and $mystrongauthkey as --auth parameter in netris-setup

+
echo $mystrongauthkey
+
+
+

Also you can see overrides values from helm get values

+
helm get values netris-controller
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/genindex.html b/en/2.9/genindex.html new file mode 100644 index 0000000000..4f7c23d411 --- /dev/null +++ b/en/2.9/genindex.html @@ -0,0 +1,436 @@ + + + + + + + + + + Index — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ + +

Index

+ +
+ +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/index.html b/en/2.9/index.html new file mode 100644 index 0000000000..9d4d82bba0 --- /dev/null +++ b/en/2.9/index.html @@ -0,0 +1,564 @@ + + + + + + + + + + Welcome to Netris documentation! — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris documentation!
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Welcome to Netris documentation!

+

Netris is the automatic NetOps platform that runs the physical network and provides cloud-like user experience for NetOps and DevOps engineers.

+

Contents:

+ + + + +
+

Updates

+ +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/network-policies.html b/en/2.9/network-policies.html new file mode 100644 index 0000000000..db8cecb1c6 --- /dev/null +++ b/en/2.9/network-policies.html @@ -0,0 +1,713 @@ + + + + + + + + + + + Basic BGP — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Basic BGP

+

BGP neighbors can be declared in the Net→E-BGP section. Netris will automatically generate and inject the right configuration to meet your requirements as declared. See below description of E-BGP neighbor declaration fields.

+
    +
  • Name - Name for BGP session.

  • +
  • Description - Free description.

  • +
  • Site - Selects the site (data center) where this BGP session should be terminated on.

  • +
  • NFV Node - Only if SoftGate nodes are in use, define on which node BGP session should be terminated on.

  • +
  • Neighbor AS - Autonomous System number of the remote side. (Local AS is defined at Net→Sites section)

  • +
  • Terminate on switch - Typically used for setups without SoftGate, for connecting with upstream routers. Instructs the system to terminate the BGP session directly on the switch.

  • +
  • Switch port - Switch Port for the physical cable to the BGP neighbor. (any port on the fabric). Optionally can bind to a V-NET service, typically used for peering with IXPs or systems like GGC (Google Global Cache).

  • +
  • VLAN ID - Optionally tag with a VLAN ID. (usually untagged)

  • +
  • IP Version - IPv4 / IPv6

  • +
  • Local IP - BGP peering IP address on Netris controlled side.

  • +
  • Remote IP - BGP peering IP address on the remote end.

  • +
  • State - Administrative state. (Enabled/Disabled)

  • +
  • Advanced - Advanced policy settings are described in the next section.

  • +
+

Example: Declare a basic BGP neighbor.

+_images/BGP_neighbor.png +
+
+

Advanced BGP

+

BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies.

+

Click Advanced to expand the BGP neighbor add/edit window.

+
    +
  • Neighbor address - IP address of the neighbor when peering with the loopback IP address instead of the interface IP address. (aka Multihop).

  • +
  • Update source - When Multihop BGP peering is used, it allows the operator to choose one of the loopback IP addresses of the SoftGate node as a BGP speaker source IP address.

  • +
  • BGP password - Password for the BGP session.

  • +
  • Allowas-in - Define the number of allowed occurrences of the self AS number in the received BGP NLRI to consider it valid. (normally 0)

  • +
  • Default Originate - Originate default route to the current neighbor.

  • +
  • Prefix Inbound Max - Drop the BGP session if the number of received prefixes exceeds this max limit. For switch termination maximum allowed is 1000 prefixes, while SoftGate termination can handle up to one million prefixes.

  • +
  • Inbound Route-Map - Apply BGP policies described in a route-map for inbound BGP updates.

  • +
  • Outbound Route-Map - Apply BGP policies described in a route-map for outbound BGP updates.

  • +
  • Local Preference - Set local preference for all inbound routes for the current neighbor.

  • +
  • Weight - Set weight for all inbound routes for the current neighbor.

  • +
  • Prepend Inbound(times) - How many times to prepend self AS number for inbound routes.

  • +
  • Prepend Outbound(times) - How many times to prepend self AS number for outbound routes.

  • +
  • Prefix List Inbound - List of IP addresses prefixes to permit or deny inbound.

  • +
  • Prefix List Outbound - List of IP addresses prefixes to permit or deny outbound.

  • +
  • Send BGP Community - List of BGP communities to send to the current neighbor.

  • +
+
+

BGP objects

+
+
Under Net→E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy.
+
Supported objects are:
+
+
    +
  • IPv4 Prefix

  • +
  • IPv6 Prefix

  • +
  • AS-PATH

  • +
  • Community

  • +
  • Extended Community

  • +
  • Large Community

  • +
+
+

IPv4 Prefix.

+
+
Rules defined one per line.
+
Each line in IPv4 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv4 prefix (mandatory).

  • +
  • Length - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv4 Prefix list.

+_images/IPv4_Prefix.png +
+
+

IPv6 Prefix.

+
+
Rules defined one per line.
+
Each line in IPv6 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv6 prefix (mandatory).

  • +
  • Keyword - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv6 Prefix list.

+_images/IPv6_Prefix.png +
+
+

Community.

+
+
Community field has two parts:
+
+
    +
  • Action - Possible values: permit or deny (mandatory).

  • +
  • Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive).

  • +
+

Example: Creating community.

+_images/community.png +
+
+
+

BGP route-maps

+
+
Under the Net→E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound.
+
Description of route-map fields:
+
+
    +
  • Sequence Number - Automatically assigned a sequence number. Drag and move sequences to organize the order.

  • +
  • Description - Free description.

  • +
  • Policy - Permit or deny the routes which match below all match clauses within the current sequence.

  • +
  • +Match - Rules for route matching.

    +
      +
    • Type - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag.

    • +
    • Object - Select an object from the list.

    • +
    +
  • +
  • Action - Action when all match clauses are met.

    +
      +
    • Action type - Define whether to manipulate a particular BGP attribute or go to another sequence.

    • +
    • Attribute - The attribute to be manipulated.

    • +
    • Value - New attribute value.

    • +
    +
  • +
+

Example: route-map

+_images/route-map.png +
+
+
+

Routes (static routing)

+

Located under Net→Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. +We recommend using the Routes only if BGP is not supported by the remote end.

+
+
Typical use cases for Routes
+
+
    +
  • To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported.

  • +
  • Temporary interconnection with the old network for a migration.

  • +
  • Routing a subnet behind a VM hypervisor machine for an internal VM network.

  • +
  • Specifically routing traffic destined to a particular prefix through an out-of-band management network.

  • +
+
+
Add new static route fields description:
+
+
    +
  • Prefix - Route destination to match.

  • +
  • Next-Hop - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network.

  • +
  • Description - Free description.

  • +
  • Site* - Site where Route belongs.

  • +
  • State - Administrative (enable/disable) state of the Route.

  • +
  • +Apply to - Limit the scope to particular units. It’s typically used for Null routes.

  • +
+

Example: Default route pointing to a Next-Hop that belongs to one of V-NETs.

+_images/defaultroute.png +

Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network.

+_images/static_route.png +

Screenshot: This Shows that my back route is actually applied on leaf1 and spine1.

+_images/leaf1_spine1.png +
+
+

NAT

+

Netris SoftGate nodes are required to support NAT (Network Address Translation).

+
+

Enabling NAT

+

To enable NAT for a given site, you first need to attach NAT IP addresses and/or NAT IP pool resources to SoftGate nodes. NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need, assuming it’s configured as an allocation under Net→Subnets section.

+
    +
  1. Allocate a public IP subnet for NAT under Net→Subnets.

  2. +
+

Example: Adding an IP allocation under Net→Subnets.

+_images/IP_allocation.png +
    +
  1. Attach NAT IP addresses and/or NAT IP Pools to just one SoftGate node. Other SoftGate Nodes on the same site will automatically add the same NAT IP/Pool resources for proper consistency and high availability.

  2. +
+

Example: Adding NAT IP addresses and NAT IP Address Pools to a SoftGate node.

+_images/NATIP_address.png +
+
+

Defining NAT rules

+

NAT rules are defined under Net→NAT.

+

NAT rule fields described:

+
    +
  • Name - Unique name.

  • +
  • Protocol

    +
      +
    • All - Match any IP protocol.

    • +
    • TCP - Match TCP traffic and ports.

    • +
    • UDP - Match UDP traffic and ports

    • +
    • ICMP - Match ICMP traffic.

    • +
    +
  • +
  • Action

    +
      +
    • SNAT - Replace the source IP address with specified NAT IP.

    • +
    • DNAT - Replace the destination IP address and/or destination port with specified NAT IP.

    • +
    • ACCEPT - Silently forward, typically used to add an exemption to broader SNAT or DNAT rule.

    • +
    +
  • +
  • Source

    +
      +
    • Address - Source IP address to match.

    • +
    • From port - Source ports to match starting with this value (TCP/UDP)

    • +
    • To port - Source ports to much up to this value (TCP/UDP)

    • +
    +
  • +
  • Destination

    +
      +
    • Address - Destination IP address to match. In the case of DNAT it should be one of the predefined NAT IP addresses.

    • +
    • Port - For DNAT only, to match a single destination port.

    • +
    • From port - For SNAT/ACCEPT only. Destination ports to match starting with this value (TCP/UDP)

    • +
    • To port - For SNAT/ACCEPT only. Destination ports to much up to this value (TCP/UDP)

    • +
    +
  • +
  • NAT IP - The global IP address for SNAT to be visible on Public Internet. The internal IP address for DNAT to replace the original destination address with.

  • +
  • Status - Administrative state (enable/disable).

  • +
  • Comment - Free optional comment.

  • +
+

Example: SNAT all hosts on 10.0.0.0/8 to the Internet using 198.51.100.65 as a global IP.

+_images/globalIP.png +

Example: Port forwarding. DNAT the traffic destined to 198.51.100.66:80 to be forwarded to the host 10.0.4.10 on port tcp/1080.

+_images/Port_Forwarding.png +

SiteMesh is a Netris service for site-to-site interconnects over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site.

+

Edit Net->Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below.

+
    +
  • Disabled - Do not participate in SiteMesh.

  • +
  • Hub - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites)

  • +
  • Spoke - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites)

  • +
  • Dynamic Spoke - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites)

  • +
+

Screenshot: Site Mesh parameter editing a Site under Net→Sites.

+_images/Site_Mesh.png +

You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge.

+_images/SiteMesh_modes.png +

Check the Net→Site Mesh section for the listing of tunnel statuses.

+

Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh)

+_images/SiteMesh_listing.png +
+
+
+

Looking Glass

+

The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Net→Looking Glass then selecting the device from the top-left dropdown menu.

+

Looking Glass controls described for IPv4/IPv6 protocol families.

+
    +
  • BGP Summary - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes.

  • +
  • BGP Route - Lookup the BGP table (RIB) for the given address.

  • +
  • Route - Lookup switch routing table for the given address.

  • +
  • Traceroute - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address.

  • +
  • Ping - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address.

  • +
+

Example: Spine1: listing BGP neighbors and number of received prefixes.

+_images/Spine1.png +

Example: BGP Route - looking up my leaf1 switch’s loopback address from spine1’s perspective. Spine1 is load balancing between two available paths.

+_images/BGP_route.png +

Example: Ping.

+_images/ping.png +
+
Looking Glass controls described for the EVPN family.
+
+
    +
  • BGP Summary - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received.

  • +
  • VNI - List VNIs learned.

  • +
  • BGP EVPN - List detailed EVPN routing information optionally for the given route distinguisher.

  • +
  • MAC table - List MAC address table for the given VNI.

  • +
+

Example: Listing of adjacent BGP neighbors and number of EVPN prefixes received.

+_images/BGP_neighbors_listing.png +

Example: Listing MAC addresses on VNI 2.

+_images/MAC_listing.png +

Example: EVPN routing information listing for a specified route distinguisher.

+_images/EVPN_routing.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/objects.inv b/en/2.9/objects.inv new file mode 100644 index 0000000000..599a1f4c62 Binary files /dev/null and b/en/2.9/objects.inv differ diff --git a/en/2.9/release-notes.html b/en/2.9/release-notes.html new file mode 100644 index 0000000000..f7e58747d9 --- /dev/null +++ b/en/2.9/release-notes.html @@ -0,0 +1,457 @@ + + + + + + + + + + + Release notes — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Release notes

+
    +
  • DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGateperformance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on.

  • +
  • L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, wenow support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates withKubernetes providing cloud-like load balancer service (type: load-balancer).

  • +
  • Kubenet​ - a network service purpose-built for Kubernetes cluster nodes.Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed tocomplement Kubernetes CNI networking with modern physical networking.

  • +
  • API logs​ - Comprehensive logging of all API calls sent to Netris Controller withthe ability to search by various attributes, sort by each column, and filter bymethod type.

  • +
  • SiteMesh​ - a Netris service for automatically configuring site-to-siteinterconnect over the public Internet. SiteMesh supports configuration forWireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a fewclicks, services in one site get connectivity to services in other sites over a meshof WireGuard tunnels.

  • +
  • Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loopcable. Removed the need for IP address reservation for V-NET, switchingentirely to the anycast default gateway.

  • +
  • Controller distributions​ - Netris controller, is now available in threedeployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application.3) Managed/Hosted in the cloud.

  • +
  • Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes.

  • +
  • Switch/SoftGate agents​ - New installer with easy initial config tool. Support forIP and FQDN as a controller address. Authentication key.

  • +
  • GUI​ - Improved Net→Topology section, becoming the main and required placefor defining the network topology. All sections got a column organizer, so everyuser can order and hide/show columns to their comfort.

  • +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox1/configurations.html b/en/2.9/sandbox/Sandbox1/configurations.html new file mode 100644 index 0000000000..87dd213570 --- /dev/null +++ b/en/2.9/sandbox/Sandbox1/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox1.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.24 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox1/creating-services.html b/en/2.9/sandbox/Sandbox1/creating-services.html new file mode 100644 index 0000000000..c7df4ac152 --- /dev/null +++ b/en/2.9/sandbox/Sandbox1/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox1.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox1.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1012.

    4. +
    5. In the Local IP field, type in 45.38.161.22

    6. +
    7. In the Remote IP field, type in 45.38.161.21.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.0/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox1.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 45.38.161.6/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox1.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox1/index.html b/en/2.9/sandbox/Sandbox1/index.html new file mode 100644 index 0000000000..b43287ff70 --- /dev/null +++ b/en/2.9/sandbox/Sandbox1/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox1 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox1/onprem-k8s.html b/en/2.9/sandbox/Sandbox1/onprem-k8s.html new file mode 100644 index 0000000000..8660379ceb --- /dev/null +++ b/en/2.9/sandbox/Sandbox1/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox1.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox1.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.10   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.10:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.10
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.10   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.10   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.10   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.10   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.11   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.11
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.11
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.11
+
+SRV05-NYC
+curl 45.38.161.11
+
+SRV05-NYC
+curl 45.38.161.11
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1012
+  localIP: 45.38.161.22/30
+  remoteIP: 45.38.161.21/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.0/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         45.38.161.22/30   45.38.161.21/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         45.38.161.22/30   45.38.161.21/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         45.38.161.22/30   45.38.161.21/30    7m59s
+sandbox1-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox1-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox1-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         45.38.161.22/30   45.38.161.21/30    8m41s
+sandbox1-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox1-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox1-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.10
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox1/sandbox-info.html b/en/2.9/sandbox/Sandbox1/sandbox-info.html new file mode 100644 index 0000000000..9bffff4290 --- /dev/null +++ b/en/2.9/sandbox/Sandbox1/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.24 -p 30061
+srv02: ssh demo@166.88.17.24 -p 30062
+srv03: ssh demo@166.88.17.24 -p 30063
+srv04: ssh demo@166.88.17.24 -p 30064
+srv05: ssh demo@166.88.17.24 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1011
+IP customer: 45.38.161.18/30
+IP Iris: 45.38.161.17/30
+
+Neighbor AS: 65007
+Vlan: 1012
+IP customer:  45.38.161.22/30
+IP Iris: 45.38.161.21/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     45.38.161.0/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox10/Configurations.html b/en/2.9/sandbox/Sandbox10/Configurations.html new file mode 100644 index 0000000000..1d4f819e5e --- /dev/null +++ b/en/2.9/sandbox/Sandbox10/Configurations.html @@ -0,0 +1,486 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions <http://creating_services/>

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox10.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “V-Net Example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net Example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select switch drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=33.4 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=27.1 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.702 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=1.37 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.609 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4008ms
+rtt min/avg/max/mdev = 0.609/12.660/33.422/14.545 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section below.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “ISP1 Example” configured as example with ISP1. This ensures communication of the inside network with the Internet. You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “ISP1 Example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” section below.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network with the public 1.1.1.1 IP address. You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.22 -p 23064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section below.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network with the public 1.1.1.1 IP address is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny. You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section below.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox10/Diagram.html b/en/2.9/sandbox/Sandbox10/Diagram.html new file mode 100644 index 0000000000..1d8b9407fa --- /dev/null +++ b/en/2.9/sandbox/Sandbox10/Diagram.html @@ -0,0 +1,435 @@ + + + + + + + + + + Diagram — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +
+

Diagram

+sandbox/images/sandbox_topology.png +

Management Subnet: 10.254.45.0/24 (X=45) Loopback Subnet: 10.254.46.0/24 (X=46)

+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox10/Sandbox-info.html b/en/2.9/sandbox/Sandbox10/Sandbox-info.html new file mode 100644 index 0000000000..819625c062 --- /dev/null +++ b/en/2.9/sandbox/Sandbox10/Sandbox-info.html @@ -0,0 +1,502 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions. https://netris.ai/slack

+

The credentials should be in the email response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGate: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Four Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, integrated with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology.png +
+ +
+

Linux servers

+
+
Example Netris services pre-configured:

srv01, srv02, srv03 - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH. +srv01, srv02 - are behing Anycast L3 load balancer. +srv04 - is consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.22 -p 23061
+srv02: ssh demo@166.88.17.22 -p 23062
+srv03: ssh demo@166.88.17.22 -p 23063
+srv04: ssh demo@166.88.17.22 -p 23064
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running Kubernetes cluster. You can deploy any application that needs to expose a TCP port, or you can deploy your favorite ingress controller that needs to expose it’s TCP port. Netris will process type:load-balancer automatically using it’s L4 Load Balancer service.

+

To access built-in Kubernetes cluster navigate to Services–>Kubenet in Netris GUI. You’ll find a pre-configured example Kubenet service. Kubenet is a network service purpose built for serving to Kubernetes nodes. Edit the service and copy “Kubeconfig” content from the example service into your “.kube/config” on your local machine. Now you should be able to kubectl the sandbox cluster.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP Iris.

+

ISP settings:

+
Customer subnet: 50.117.59.208/28
+
+(pre-configured example)
+Vlan: 801
+IP customer: 50.117.59.122/30
+IP Iris: 50.117.59.121/30
+
+Vlan: 802
+IP customer:  50.117.59.126/30
+IP Iris: 50.117.59.125/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+Public subnet:     50.117.59.208/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox10/creating-services.html b/en/2.9/sandbox/Sandbox10/creating-services.html new file mode 100644 index 0000000000..32d40ac422 --- /dev/null +++ b/en/2.9/sandbox/Sandbox10/creating-services.html @@ -0,0 +1,577 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv04-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv04-nyc server by typing ssh demo@166.88.17.22 -p 23064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth2 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting http://sandbox10.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv04-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can find the required port by typing “srv04” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw21-nyc-swp2 (srv04))@sw21-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with “ISP1”.

+

Optionally you can configure an E-BGP session to ISP 2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox10.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. ISP2 Customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 100.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. For the VLAN ID field, unselect Untag and type in 802.

    2. +
    3. In the Local IP field, type in 50.117.59.126

    4. +
    5. In the Remote IP field, type in 50.117.59.127.

    6. +
    7. Expand the Advanced section

    8. +
    9. In the Prefix List Outbound field, type in permit 50.117.59.208/28 le 32

    10. +
    11. And finally click Add

    12. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv04-nyc server to be able to communicate with 8.8.8.8.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.22 -p 23064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 8.8.8.8 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with public IP 8.8.8.8.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox10.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.213/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 8.8.8.8: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the 8.8.8.8 address is reachable.

+
+
+

ACL (Access Control List)

+

Now that srv02-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.22 -p 23064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 8.8.8.8 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox10.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 8.8.8.8 command, indicating that the 8.8.8.8 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv02-nyc server to communicate with 8.8.8.8.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL

    2. +
    3. Click +Add to define a new ACL

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+
+

+
+
    +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 8.8.8.8 command have resumed, indicating that the srv02-nyc server can communicate with 8.8.8.8 once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox10/index.html b/en/2.9/sandbox/Sandbox10/index.html new file mode 100644 index 0000000000..22bf7ea256 --- /dev/null +++ b/en/2.9/sandbox/Sandbox10/index.html @@ -0,0 +1,454 @@ + + + + + + + + + + Sandbox10 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+ +
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox2/configurations.html b/en/2.9/sandbox/Sandbox2/configurations.html new file mode 100644 index 0000000000..5b0c0f8cf5 --- /dev/null +++ b/en/2.9/sandbox/Sandbox2/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox2.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.190 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox2/creating-services.html b/en/2.9/sandbox/Sandbox2/creating-services.html new file mode 100644 index 0000000000..0b02178fea --- /dev/null +++ b/en/2.9/sandbox/Sandbox2/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox2.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox2.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1022.

    4. +
    5. In the Local IP field, type in 45.38.161.30

    6. +
    7. In the Remote IP field, type in 45.38.161.29.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.32/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox2.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 45.38.161.38/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox2.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox2/index.html b/en/2.9/sandbox/Sandbox2/index.html new file mode 100644 index 0000000000..29844b1cb5 --- /dev/null +++ b/en/2.9/sandbox/Sandbox2/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox2 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox2/onprem-k8s.html b/en/2.9/sandbox/Sandbox2/onprem-k8s.html new file mode 100644 index 0000000000..006f36b693 --- /dev/null +++ b/en/2.9/sandbox/Sandbox2/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox2.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox2.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.42   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.42:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.42
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.42   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.42   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.42   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.42   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.43   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.43
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.43
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.43
+
+SRV05-NYC
+curl 45.38.161.43
+
+SRV05-NYC
+curl 45.38.161.43
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1022
+  localIP: 45.38.161.30/30
+  remoteIP: 45.38.161.29/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.32/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         45.38.161.30/30   45.38.161.29/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         45.38.161.30/30   45.38.161.29/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         45.38.161.30/30   45.38.161.29/30    7m59s
+sandbox2-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox2-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox2-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         45.38.161.30/30   45.38.161.29/30    8m41s
+sandbox2-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox2-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox2-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.42
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox2/sandbox-info.html b/en/2.9/sandbox/Sandbox2/sandbox-info.html new file mode 100644 index 0000000000..117e8a9861 --- /dev/null +++ b/en/2.9/sandbox/Sandbox2/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.190 -p 30061
+srv02: ssh demo@166.88.17.190 -p 30062
+srv03: ssh demo@166.88.17.190 -p 30063
+srv04: ssh demo@166.88.17.190 -p 30064
+srv05: ssh demo@166.88.17.190 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1021
+IP customer: 45.38.161.26/30
+IP Iris: 45.38.161.25/30
+
+Neighbor AS: 65007
+Vlan: 1022
+IP customer:  45.38.161.30/30
+IP Iris: 45.38.161.29/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     45.38.161.32/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox3/configurations.html b/en/2.9/sandbox/Sandbox3/configurations.html new file mode 100644 index 0000000000..67c08c5969 --- /dev/null +++ b/en/2.9/sandbox/Sandbox3/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox3.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.189 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox3/creating-services.html b/en/2.9/sandbox/Sandbox3/creating-services.html new file mode 100644 index 0000000000..6cf442a2ab --- /dev/null +++ b/en/2.9/sandbox/Sandbox3/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox3.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox3.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1032.

    4. +
    5. In the Local IP field, type in 45.38.161.70

    6. +
    7. In the Remote IP field, type in 45.38.161.69.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.48/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox3.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 45.38.161.54/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox3.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox3/index.html b/en/2.9/sandbox/Sandbox3/index.html new file mode 100644 index 0000000000..8506273157 --- /dev/null +++ b/en/2.9/sandbox/Sandbox3/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox3 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox3/onprem-k8s.html b/en/2.9/sandbox/Sandbox3/onprem-k8s.html new file mode 100644 index 0000000000..a56ba9b3b0 --- /dev/null +++ b/en/2.9/sandbox/Sandbox3/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox3.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox3.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.58   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.58:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.58
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.58   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.58   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.58   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.58   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.59   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.59
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.59
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.59
+
+SRV05-NYC
+curl 45.38.161.59
+
+SRV05-NYC
+curl 45.38.161.59
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1032
+  localIP: 45.38.161.70/30
+  remoteIP: 45.38.161.69/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.48/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         45.38.161.70/30   45.38.161.69/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         45.38.161.70/30   45.38.161.69/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         45.38.161.70/30   45.38.161.69/30    7m59s
+sandbox3-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox3-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox3-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         45.38.161.70/30   45.38.161.69/30    8m41s
+sandbox3-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox3-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox3-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.58
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox3/sandbox-info.html b/en/2.9/sandbox/Sandbox3/sandbox-info.html new file mode 100644 index 0000000000..89e4be6a91 --- /dev/null +++ b/en/2.9/sandbox/Sandbox3/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.189 -p 30061
+srv02: ssh demo@166.88.17.189 -p 30062
+srv03: ssh demo@166.88.17.189 -p 30063
+srv04: ssh demo@166.88.17.189 -p 30064
+srv05: ssh demo@166.88.17.189 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1031
+IP customer: 45.38.161.66/30
+IP Iris: 45.38.161.65/30
+
+Neighbor AS: 65007
+Vlan: 1032
+IP customer:  45.38.161.70/30
+IP Iris: 45.38.161.69/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     45.38.161.48/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox4/configurations.html b/en/2.9/sandbox/Sandbox4/configurations.html new file mode 100644 index 0000000000..7eecb05b42 --- /dev/null +++ b/en/2.9/sandbox/Sandbox4/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox4.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.188 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox4/creating-services.html b/en/2.9/sandbox/Sandbox4/creating-services.html new file mode 100644 index 0000000000..3a08c1a64d --- /dev/null +++ b/en/2.9/sandbox/Sandbox4/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox4.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox4.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1042.

    4. +
    5. In the Local IP field, type in 45.38.161.78

    6. +
    7. In the Remote IP field, type in 45.38.161.77.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.80/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox4.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 45.38.161.86/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox4.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox4/index.html b/en/2.9/sandbox/Sandbox4/index.html new file mode 100644 index 0000000000..0c2e401451 --- /dev/null +++ b/en/2.9/sandbox/Sandbox4/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox4 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox4/onprem-k8s.html b/en/2.9/sandbox/Sandbox4/onprem-k8s.html new file mode 100644 index 0000000000..7aa58010f5 --- /dev/null +++ b/en/2.9/sandbox/Sandbox4/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox4.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox4.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.90   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.90:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.90
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.90   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.90   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.90   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.90   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.91   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.91
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.91
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.91
+
+SRV05-NYC
+curl 45.38.161.91
+
+SRV05-NYC
+curl 45.38.161.91
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1042
+  localIP: 45.38.161.78/30
+  remoteIP: 45.38.161.77/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.80/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         45.38.161.78/30   45.38.161.77/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         45.38.161.78/30   45.38.161.77/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         45.38.161.78/30   45.38.161.77/30    7m59s
+sandbox4-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox4-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox4-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         45.38.161.78/30   45.38.161.77/30    8m41s
+sandbox4-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox4-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox4-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.90
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox4/sandbox-info.html b/en/2.9/sandbox/Sandbox4/sandbox-info.html new file mode 100644 index 0000000000..77b23e4d3d --- /dev/null +++ b/en/2.9/sandbox/Sandbox4/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.188 -p 30061
+srv02: ssh demo@166.88.17.188 -p 30062
+srv03: ssh demo@166.88.17.188 -p 30063
+srv04: ssh demo@166.88.17.188 -p 30064
+srv05: ssh demo@166.88.17.188 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1041
+IP customer: 45.38.161.74/30
+IP Iris: 45.38.161.73/30
+
+Neighbor AS: 65007
+Vlan: 1042
+IP customer:  45.38.161.78/30
+IP Iris: 45.38.161.77/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     45.38.161.80/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox5/configurations.html b/en/2.9/sandbox/Sandbox5/configurations.html new file mode 100644 index 0000000000..a64c7adce7 --- /dev/null +++ b/en/2.9/sandbox/Sandbox5/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox5.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.187 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox5/creating-services.html b/en/2.9/sandbox/Sandbox5/creating-services.html new file mode 100644 index 0000000000..a00d3c0b1a --- /dev/null +++ b/en/2.9/sandbox/Sandbox5/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox5.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox5.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1052.

    4. +
    5. In the Local IP field, type in 50.117.59.86

    6. +
    7. In the Remote IP field, type in 50.117.59.85.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.128/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox5.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.134/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox5.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox5/index.html b/en/2.9/sandbox/Sandbox5/index.html new file mode 100644 index 0000000000..62cc61bfe1 --- /dev/null +++ b/en/2.9/sandbox/Sandbox5/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox5 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox5/onprem-k8s.html b/en/2.9/sandbox/Sandbox5/onprem-k8s.html new file mode 100644 index 0000000000..7015932d1b --- /dev/null +++ b/en/2.9/sandbox/Sandbox5/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox5.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox5.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.138   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.138:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.138
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.138   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.138   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.138   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.138   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.139   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.139
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.139
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.139
+
+SRV05-NYC
+curl 50.117.59.139
+
+SRV05-NYC
+curl 50.117.59.139
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1052
+  localIP: 50.117.59.86/30
+  remoteIP: 50.117.59.85/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.128/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.86/30   50.117.59.85/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.86/30   50.117.59.85/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.86/30   50.117.59.85/30    7m59s
+sandbox5-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox5-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox5-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.86/30   50.117.59.85/30    8m41s
+sandbox5-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox5-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox5-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.138
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox5/sandbox-info.html b/en/2.9/sandbox/Sandbox5/sandbox-info.html new file mode 100644 index 0000000000..fc7c70eeb5 --- /dev/null +++ b/en/2.9/sandbox/Sandbox5/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.187 -p 30061
+srv02: ssh demo@166.88.17.187 -p 30062
+srv03: ssh demo@166.88.17.187 -p 30063
+srv04: ssh demo@166.88.17.187 -p 30064
+srv05: ssh demo@166.88.17.187 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1051
+IP customer: 50.117.59.82/30
+IP Iris: 50.117.59.81/30
+
+Neighbor AS: 65007
+Vlan: 1052
+IP customer:  50.117.59.86/30
+IP Iris: 50.117.59.85/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     50.117.59.128/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox6/configurations.html b/en/2.9/sandbox/Sandbox6/configurations.html new file mode 100644 index 0000000000..0620acec8a --- /dev/null +++ b/en/2.9/sandbox/Sandbox6/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox6.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.186 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox6/creating-services.html b/en/2.9/sandbox/Sandbox6/creating-services.html new file mode 100644 index 0000000000..94c2a2a8ce --- /dev/null +++ b/en/2.9/sandbox/Sandbox6/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1062.

    4. +
    5. In the Local IP field, type in 50.117.59.94

    6. +
    7. In the Remote IP field, type in 50.117.59.93.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.144/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.150/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox6/index.html b/en/2.9/sandbox/Sandbox6/index.html new file mode 100644 index 0000000000..c959d325f3 --- /dev/null +++ b/en/2.9/sandbox/Sandbox6/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox6 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox6/onprem-k8s.html b/en/2.9/sandbox/Sandbox6/onprem-k8s.html new file mode 100644 index 0000000000..e255cbbaa8 --- /dev/null +++ b/en/2.9/sandbox/Sandbox6/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox6.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox6.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.154   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.154:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.154
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.155   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.155
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.155
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1062
+  localIP: 50.117.59.94/30
+  remoteIP: 50.117.59.93/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.144/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.94/30   50.117.59.93/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.94/30   50.117.59.93/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.94/30   50.117.59.93/30    7m59s
+sandbox6-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox6-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox6-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.94/30   50.117.59.93/30    8m41s
+sandbox6-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox6-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox6-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.154
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox6/sandbox-info.html b/en/2.9/sandbox/Sandbox6/sandbox-info.html new file mode 100644 index 0000000000..e4bddbaadd --- /dev/null +++ b/en/2.9/sandbox/Sandbox6/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.186 -p 30061
+srv02: ssh demo@166.88.17.186 -p 30062
+srv03: ssh demo@166.88.17.186 -p 30063
+srv04: ssh demo@166.88.17.186 -p 30064
+srv05: ssh demo@166.88.17.186 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1061
+IP customer: 50.117.59.90/30
+IP Iris: 50.117.59.89/30
+
+Neighbor AS: 65007
+Vlan: 1062
+IP customer:  50.117.59.94/30
+IP Iris: 50.117.59.93/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     50.117.59.144/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox7/configurations.html b/en/2.9/sandbox/Sandbox7/configurations.html new file mode 100644 index 0000000000..2f6027c544 --- /dev/null +++ b/en/2.9/sandbox/Sandbox7/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox7.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.185 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox7/creating-services.html b/en/2.9/sandbox/Sandbox7/creating-services.html new file mode 100644 index 0000000000..c3b79b21eb --- /dev/null +++ b/en/2.9/sandbox/Sandbox7/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox7.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox7.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1072.

    4. +
    5. In the Local IP field, type in 50.117.59.102

    6. +
    7. In the Remote IP field, type in 50.117.59.101.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.160/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox7.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.166/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox7.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox7/index.html b/en/2.9/sandbox/Sandbox7/index.html new file mode 100644 index 0000000000..f1b39e6637 --- /dev/null +++ b/en/2.9/sandbox/Sandbox7/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox7 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox7/onprem-k8s.html b/en/2.9/sandbox/Sandbox7/onprem-k8s.html new file mode 100644 index 0000000000..510c087403 --- /dev/null +++ b/en/2.9/sandbox/Sandbox7/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox7.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox7.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.170   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.170:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.170
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.170   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.170   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.170   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.170   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.171   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.171
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.171
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.171
+
+SRV05-NYC
+curl 50.117.59.171
+
+SRV05-NYC
+curl 50.117.59.171
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1072
+  localIP: 50.117.59.102/30
+  remoteIP: 50.117.59.101/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.160/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.102/30   50.117.59.101/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.102/30   50.117.59.101/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.102/30   50.117.59.101/30    7m59s
+sandbox7-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox7-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox7-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.102/30   50.117.59.101/30    8m41s
+sandbox7-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox7-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox7-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.170
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox7/sandbox-info.html b/en/2.9/sandbox/Sandbox7/sandbox-info.html new file mode 100644 index 0000000000..fad2f38624 --- /dev/null +++ b/en/2.9/sandbox/Sandbox7/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.185 -p 30061
+srv02: ssh demo@166.88.17.185 -p 30062
+srv03: ssh demo@166.88.17.185 -p 30063
+srv04: ssh demo@166.88.17.185 -p 30064
+srv05: ssh demo@166.88.17.185 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1071
+IP customer: 50.117.59.98/30
+IP Iris: 50.117.59.97/30
+
+Neighbor AS: 65007
+Vlan: 1072
+IP customer:  50.117.59.102/30
+IP Iris: 50.117.59.101/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     50.117.59.160/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox8/configurations.html b/en/2.9/sandbox/Sandbox8/configurations.html new file mode 100644 index 0000000000..61b5e567ba --- /dev/null +++ b/en/2.9/sandbox/Sandbox8/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox8.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.42.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.41.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.41.64
+PING 192.168.41.64 (192.168.41.64) 56(84) bytes of data.
+64 bytes from 192.168.41.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.41.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.41.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.41.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.41.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.41.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.41.0/24 network with the public 1.1.1.1 IP address.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.29 -p 22864.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.41.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox8/creating-services.html b/en/2.9/sandbox/Sandbox8/creating-services.html new file mode 100644 index 0000000000..812ef21778 --- /dev/null +++ b/en/2.9/sandbox/Sandbox8/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.29 -p 22865.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.42.1 is configured as the default gateway, indicated by the “default via 192.168.42.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.42.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.42.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox8.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.42.0/24(CUSTOMER) and IP 192.168.42.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.42.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox8.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1082.

    4. +
    5. In the Local IP field, type in 50.117.59.110

    6. +
    7. In the Remote IP field, type in 50.117.59.109.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.176/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.29 -p 22865.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.42.0/24 can communicate with public IP 1.1.1.1.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox8.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.42.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.182/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.29 -p 22865.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox8.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.42.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox8/index.html b/en/2.9/sandbox/Sandbox8/index.html new file mode 100644 index 0000000000..0afd351827 --- /dev/null +++ b/en/2.9/sandbox/Sandbox8/index.html @@ -0,0 +1,461 @@ + + + + + + + + + + Sandbox8 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox8/onprem-k8s.html b/en/2.9/sandbox/Sandbox8/onprem-k8s.html new file mode 100644 index 0000000000..01401e15b2 --- /dev/null +++ b/en/2.9/sandbox/Sandbox8/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox8.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox8.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.186   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.186:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.186
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.186   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.186   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.41.64:80
+   - 192.168.42.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.186   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.186   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.187   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.187
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.42.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.42.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.187
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.187
+
+SRV05-NYC
+curl 50.117.59.187
+
+SRV05-NYC
+curl 50.117.59.187
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.42.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1082
+  localIP: 50.117.59.110/30
+  remoteIP: 50.117.59.109/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.176/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.110/30   50.117.59.109/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.110/30   50.117.59.109/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.110/30   50.117.59.109/30    7m59s
+sandbox8-srv06-nyc-192.168.108.66   enabled                                                               4200070000    192.168.108.1/24   192.168.108.66/24   26s
+sandbox8-srv07-nyc-192.168.108.67   enabled                                                               4200070001    192.168.108.1/24   192.168.108.67/24   26s
+sandbox8-srv08-nyc-192.168.108.68   enabled                                                               4200070002    192.168.108.1/24   192.168.108.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.110/30   50.117.59.109/30    8m41s
+sandbox8-srv06-nyc-192.168.108.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.108.1/24   192.168.108.66/24   68s
+sandbox8-srv07-nyc-192.168.108.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.108.1/24   192.168.108.67/24   68s
+sandbox8-srv08-nyc-192.168.108.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.108.1/24   192.168.108.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.186
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox8/sandbox-info.html b/en/2.9/sandbox/Sandbox8/sandbox-info.html new file mode 100644 index 0000000000..8dc56e0c83 --- /dev/null +++ b/en/2.9/sandbox/Sandbox8/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.29 -p 22861
+srv02: ssh demo@166.88.17.29 -p 22862
+srv03: ssh demo@166.88.17.29 -p 22863
+srv04: ssh demo@166.88.17.29 -p 22864
+srv05: ssh demo@166.88.17.29 -p 22865
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1081
+IP customer: 50.117.59.106/30
+IP Iris: 50.117.59.105/30
+
+Neighbor AS: 65007
+Vlan: 1082
+IP customer:  50.117.59.110/30
+IP Iris: 50.117.59.109/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.41.0/24
+Loopback subnet:   10.254.42.0/24
+Example subnet:    192.168.41.0/24
+Customer subnet:   192.168.42.0/24
+K8s subnet:        192.168.108.0/24
+Public subnet:     50.117.59.176/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox9/configurations.html b/en/2.9/sandbox/Sandbox9/configurations.html new file mode 100644 index 0000000000..5408bc9f8f --- /dev/null +++ b/en/2.9/sandbox/Sandbox9/configurations.html @@ -0,0 +1,490 @@ + + + + + + + + + + Provided Example Configurations — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox9.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.19 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox9/creating-services.html b/en/2.9/sandbox/Sandbox9/creating-services.html new file mode 100644 index 0000000000..798ee89d62 --- /dev/null +++ b/en/2.9/sandbox/Sandbox9/creating-services.html @@ -0,0 +1,573 @@ + + + + + + + + + + Learn by Creating Services — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox9.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox9.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1092.

    4. +
    5. In the Local IP field, type in 50.117.59.118

    6. +
    7. In the Remote IP field, type in 50.117.59.117.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.192/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox9.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.198/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox9.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox9/index.html b/en/2.9/sandbox/Sandbox9/index.html new file mode 100644 index 0000000000..28d0b05d96 --- /dev/null +++ b/en/2.9/sandbox/Sandbox9/index.html @@ -0,0 +1,469 @@ + + + + + + + + + + Sandbox9 — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox9/onprem-k8s.html b/en/2.9/sandbox/Sandbox9/onprem-k8s.html new file mode 100644 index 0000000000..9058de2a24 --- /dev/null +++ b/en/2.9/sandbox/Sandbox9/onprem-k8s.html @@ -0,0 +1,920 @@ + + + + + + + + + + Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox9.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox9.netris.ai/' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.202   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.202:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.202
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.202   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.202   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.203   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.203
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.203
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.203
+
+SRV05-NYC
+curl 50.117.59.203
+
+SRV05-NYC
+curl 50.117.59.203
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1092
+  localIP: 50.117.59.118/30
+  remoteIP: 50.117.59.117/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.192/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.118/30   50.117.59.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.118/30   50.117.59.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.118/30   50.117.59.117/30    7m59s
+sandbox9-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox9-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox9-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.118/30   50.117.59.117/30    8m41s
+sandbox9-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox9-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox9-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.202
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/sandbox/Sandbox9/sandbox-info.html b/en/2.9/sandbox/Sandbox9/sandbox-info.html new file mode 100644 index 0000000000..d2cb2ee2df --- /dev/null +++ b/en/2.9/sandbox/Sandbox9/sandbox-info.html @@ -0,0 +1,506 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.19 -p 30061
+srv02: ssh demo@166.88.17.19 -p 30062
+srv03: ssh demo@166.88.17.19 -p 30063
+srv04: ssh demo@166.88.17.19 -p 30064
+srv05: ssh demo@166.88.17.19 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1091
+IP customer: 50.117.59.114/30
+IP Iris: 50.117.59.113/30
+
+Neighbor AS: 65007
+Vlan: 1092
+IP customer:  50.117.59.118/30
+IP Iris: 50.117.59.117/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     50.117.59.192/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/search.html b/en/2.9/search.html new file mode 100644 index 0000000000..1511fb2802 --- /dev/null +++ b/en/2.9/search.html @@ -0,0 +1,445 @@ + + + + + + + + + + Search — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Search
  • + + +
  • + +
  • + +
+ + +
+
+
+
+ + + + +
+ +
+ +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/searchindex.js b/en/2.9/searchindex.js new file mode 100644 index 0000000000..b02012e767 --- /dev/null +++ b/en/2.9/searchindex.js @@ -0,0 +1 @@ +Search.setIndex({"docnames": ["SoftGate-agent-installation", "accounts", "concepts", "controller-initial-configuration", "controller-installation", "controller-k8s-installation", "index", "network-policies", "release-notes", "sandbox/Sandbox1/configurations", "sandbox/Sandbox1/creating-services", "sandbox/Sandbox1/index", "sandbox/Sandbox1/onprem-k8s", "sandbox/Sandbox1/sandbox-info", "sandbox/Sandbox10/Configurations", "sandbox/Sandbox10/Diagram", "sandbox/Sandbox10/Sandbox-info", "sandbox/Sandbox10/creating-services", "sandbox/Sandbox10/index", "sandbox/Sandbox2/configurations", "sandbox/Sandbox2/creating-services", "sandbox/Sandbox2/index", "sandbox/Sandbox2/onprem-k8s", "sandbox/Sandbox2/sandbox-info", "sandbox/Sandbox3/configurations", "sandbox/Sandbox3/creating-services", "sandbox/Sandbox3/index", "sandbox/Sandbox3/onprem-k8s", "sandbox/Sandbox3/sandbox-info", "sandbox/Sandbox4/configurations", "sandbox/Sandbox4/creating-services", "sandbox/Sandbox4/index", "sandbox/Sandbox4/onprem-k8s", "sandbox/Sandbox4/sandbox-info", "sandbox/Sandbox5/configurations", "sandbox/Sandbox5/creating-services", "sandbox/Sandbox5/index", "sandbox/Sandbox5/onprem-k8s", "sandbox/Sandbox5/sandbox-info", "sandbox/Sandbox6/configurations", "sandbox/Sandbox6/creating-services", "sandbox/Sandbox6/index", "sandbox/Sandbox6/onprem-k8s", "sandbox/Sandbox6/sandbox-info", "sandbox/Sandbox7/configurations", "sandbox/Sandbox7/creating-services", "sandbox/Sandbox7/index", "sandbox/Sandbox7/onprem-k8s", "sandbox/Sandbox7/sandbox-info", "sandbox/Sandbox8/configurations", "sandbox/Sandbox8/creating-services", "sandbox/Sandbox8/index", "sandbox/Sandbox8/onprem-k8s", "sandbox/Sandbox8/sandbox-info", "sandbox/Sandbox9/configurations", "sandbox/Sandbox9/creating-services", "sandbox/Sandbox9/index", "sandbox/Sandbox9/onprem-k8s", "sandbox/Sandbox9/sandbox-info", "services", "switch-agent-installation", "visibility"], "filenames": ["SoftGate-agent-installation.rst", "accounts.rst", "concepts.rst", "controller-initial-configuration.rst", "controller-installation.rst", "controller-k8s-installation.rst", "index.rst", "network-policies.rst", "release-notes.rst", "sandbox/Sandbox1/configurations.rst", "sandbox/Sandbox1/creating-services.rst", "sandbox/Sandbox1/index.rst", "sandbox/Sandbox1/onprem-k8s.rst", "sandbox/Sandbox1/sandbox-info.rst", "sandbox/Sandbox10/Configurations.rst", "sandbox/Sandbox10/Diagram.rst", "sandbox/Sandbox10/Sandbox-info.rst", "sandbox/Sandbox10/creating-services.rst", "sandbox/Sandbox10/index.rst", "sandbox/Sandbox2/configurations.rst", "sandbox/Sandbox2/creating-services.rst", "sandbox/Sandbox2/index.rst", "sandbox/Sandbox2/onprem-k8s.rst", "sandbox/Sandbox2/sandbox-info.rst", "sandbox/Sandbox3/configurations.rst", "sandbox/Sandbox3/creating-services.rst", "sandbox/Sandbox3/index.rst", "sandbox/Sandbox3/onprem-k8s.rst", "sandbox/Sandbox3/sandbox-info.rst", "sandbox/Sandbox4/configurations.rst", "sandbox/Sandbox4/creating-services.rst", "sandbox/Sandbox4/index.rst", "sandbox/Sandbox4/onprem-k8s.rst", "sandbox/Sandbox4/sandbox-info.rst", "sandbox/Sandbox5/configurations.rst", "sandbox/Sandbox5/creating-services.rst", "sandbox/Sandbox5/index.rst", "sandbox/Sandbox5/onprem-k8s.rst", "sandbox/Sandbox5/sandbox-info.rst", "sandbox/Sandbox6/configurations.rst", "sandbox/Sandbox6/creating-services.rst", "sandbox/Sandbox6/index.rst", "sandbox/Sandbox6/onprem-k8s.rst", "sandbox/Sandbox6/sandbox-info.rst", "sandbox/Sandbox7/configurations.rst", "sandbox/Sandbox7/creating-services.rst", "sandbox/Sandbox7/index.rst", "sandbox/Sandbox7/onprem-k8s.rst", "sandbox/Sandbox7/sandbox-info.rst", "sandbox/Sandbox8/configurations.rst", "sandbox/Sandbox8/creating-services.rst", "sandbox/Sandbox8/index.rst", "sandbox/Sandbox8/onprem-k8s.rst", "sandbox/Sandbox8/sandbox-info.rst", "sandbox/Sandbox9/configurations.rst", "sandbox/Sandbox9/creating-services.rst", "sandbox/Sandbox9/index.rst", "sandbox/Sandbox9/onprem-k8s.rst", "sandbox/Sandbox9/sandbox-info.rst", "services.rst", "switch-agent-installation.rst", "visibility.rst"], "titles": ["Netris SoftGate agent installation", "Accounts", "Concepts", "Controller initial configuration", "On-prem Netris Controller installation", "Netris Controller Helm Chart", "Welcome to Netris documentation!", "Basic BGP", "Release notes", "Provided Example Configurations", "Learn by Creating Services", "Sandbox1", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Diagram", "Welcome to Netris Sandbox", "Learn by Creating Services", "Sandbox10", "Provided Example Configurations", "Learn by Creating Services", "Sandbox2", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox3", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox4", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox5", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox6", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox7", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox8", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox9", "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps", "Welcome to Netris Sandbox", "V-NET", "Netris switch agent installation", "Visibility (Telescope)"], "terms": {"A": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61], "variabl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61], "2": [0, 2, 3, 7, 8, 9, 14, 17, 19, 24, 29, 34, 39, 44, 49, 54, 59], "x": [0, 2, 15, 60], "intel": [0, 2], "silver": 0, "cpu": [0, 2, 4, 5, 61], "96": [0, 3], "gb": [0, 4], "ram": [0, 4, 61], "300": 0, "hdd": 0, "nvidia": [0, 3], "mellanox": [0, 3, 60], "connect": [0, 3, 4, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60], "5": [0, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59], "smartnic": [0, 2, 3], "card": [0, 2], "The": [0, 1, 2, 3, 4, 5, 6, 7, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 60], "follow": [0, 5, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "ar": [0, 1, 5, 7, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59], "some": [0, 4, 9, 12, 13, 14, 16, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59], "recommend": [0, 2, 4, 7], "set": [0, 1, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 61], "differ": [0, 1, 3, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "vendor": 0, "have": [0, 1, 2, 3, 5, 7, 9, 10, 13, 14, 16, 17, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58, 59, 60], "name": [0, 1, 3, 4, 5, 7, 9, 10, 12, 13, 14, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 61], "so": [0, 2, 7, 8, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 61], "i": [0, 1, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 61], "mainli": 0, "refer": [0, 2, 3, 5, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "befor": [0, 12, 22, 27, 32, 37, 42, 47, 52, 57], "start": [0, 3, 4, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 61], "consid": [0, 7], "reset": 0, "all": [0, 1, 2, 3, 4, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 61], "default": [0, 1, 3, 5, 6, 7, 8, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 60], "disabl": [0, 5, 7, 59], "power": [0, 61], "save": [0, 3, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 61], "option": [0, 4, 5, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 61], "perform": [0, 2, 59], "tune": [0, 7], "p": [0, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58], "state": [0, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 61], "c3": 0, "report": [0, 2], "c6": 0, "select": [0, 3, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 61], "polici": [0, 1, 2, 3, 5, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "turbo": 0, "boost": 0, "ensur": [0, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "scale": [0, 7], "increas": 0, "number": [0, 1, 3, 5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "core": [0, 2, 4], "memori": [0, 5], "frequenc": 0, "highest": 0, "avail": [0, 3, 7, 8, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59, 60, 61], "NOT": 0, "auto": [0, 4, 59, 60], "virtual": [0, 6, 8, 59], "when": [0, 1, 3, 4, 5, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "you": [0, 2, 3, 4, 5, 7, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 61], "test": [0, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "physic": [0, 2, 3, 4, 6, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "function": [0, 2, 4, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 61], "nic": [0, 2, 4], "turn": [0, 5, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "off": [0, 2], "vt": 0, "d": [0, 4, 60], "hyper": 0, "thread": 0, "freshli": [0, 12, 22, 27, 32, 37, 42, 47, 52, 57], "ubuntu": [0, 3, 4, 6, 8], "linux": [0, 2, 3, 4, 6, 11, 12, 18, 21, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 59], "18": [0, 4, 12, 13, 22, 27, 32, 37, 42, 47, 52, 57, 60], "04": [0, 4, 60], "network": [0, 1, 2, 3, 4, 5, 7, 8, 11, 12, 18, 21, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 59, 60, 61], "your": [0, 2, 4, 5, 7, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59, 60], "control": [0, 1, 6, 7, 8, 11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 60, 61], "over": [0, 1, 2, 3, 7, 8, 59], "out": [0, 3, 7, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 60], "band": [0, 3, 7, 60], "manag": [0, 1, 2, 4, 6, 7, 8, 12, 13, 15, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59, 61], "environ": [0, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "us": [0, 1, 2, 4, 5, 7, 10, 11, 17, 18, 20, 21, 25, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 59, 60], "proxi": [0, 60], "export": [0, 5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "http_proxi": [0, 60], "http": [0, 4, 5, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60], "address": [0, 1, 2, 4, 7, 8, 11, 12, 13, 16, 18, 21, 22, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 59, 61], "3128": [0, 60], "https_proxi": 0, "echo": [0, 4, 5, 60], "e": [0, 1, 3, 7, 11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 60, 61], "acquir": [0, 60], "nacquir": [0, 60], "sudo": [0, 4, 60], "tee": [0, 60], "etc": [0, 4, 60, 61], "apt": [0, 4, 60], "conf": [0, 60], "config": [0, 4, 5, 8, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59], "repositori": [0, 4, 5, 60], "wget": [0, 4, 60], "qo": [0, 60], "www": 0, "com": [0, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "download": [0, 4, 12, 22, 27, 32, 37, 42, 47, 52, 57], "ofed": 0, "rpm": 0, "gpg": 0, "kei": [0, 5, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "add": [0, 3, 5, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 60, 61], "public": [0, 3, 7, 8, 9, 13, 14, 16, 17, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 50, 53, 54, 58, 60, 61], "repo": [0, 6, 60], "mlnx_ofe": 0, "0": [0, 3, 4, 5, 7, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60], "1": [0, 4, 5, 8, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 60], "8": [0, 2, 3, 4, 5, 7, 12, 17, 22, 27, 32, 37, 42, 47, 52, 57, 60], "ubuntu18": 0, "mellanox_mlnx_of": 0, "list": [0, 1, 3, 5, 6, 7, 11, 12, 18, 21, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 60, 61], "o": [0, 2, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "tmp": 0, "mv": 0, "sourc": [0, 4, 7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59, 60, 61], "ai": [0, 4, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 60], "deb": [0, 60], "bionic": [0, 60], "main": [0, 2, 8, 60], "driver": 0, "get": [0, 3, 4, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "updat": [0, 5, 7, 8, 60], "mlnx": 0, "dpdk": [0, 2, 8], "packag": 0, "depend": [0, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "includ": [0, 7, 10, 13, 16, 17, 20, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 59], "specif": [0, 1, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "kernel": [0, 2, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "version": [0, 3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "ip": [0, 1, 4, 5, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59], "In": [0, 3, 5, 7, 8, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 60], "case": [0, 7, 59, 60], "same": [0, 4, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "oob": 0, "rout": [0, 2, 6, 8, 10, 13, 16, 17, 20, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60], "No": [0, 60], "other": [0, 1, 2, 3, 5, 7, 8, 59, 60], "should": [0, 3, 4, 7, 9, 10, 12, 14, 16, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 60], "vim": [0, 4, 60], "interfac": [0, 2, 3, 4, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "loopback": [0, 3, 4, 7, 13, 15, 16, 23, 28, 33, 38, 43, 48, 53, 58, 59, 60], "lo": [0, 4, 60], "ifac": [0, 4, 60], "inet": [0, 4, 60], "primari": [0, 4, 60], "eth0": [0, 4, 60], "static": [0, 4, 6, 60], "prefix": [0, 4, 5, 6, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 60], "length": [0, 4, 7, 60], "up": [0, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 61], "ro": [0, 60], "via": [0, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60], "gatewai": [0, 4, 8, 11, 12, 13, 16, 18, 21, 22, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 59, 60], "delet": [0, 5, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 60], "thi": [0, 1, 3, 7, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60], "line": [0, 3, 7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59, 60], "locat": [0, 7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 60], "node": [0, 5, 6, 7, 8, 10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 59], "ifreload": [0, 4, 60], "initi": [0, 6, 8, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 60], "setup": [0, 3, 5, 7, 60], "paramet": [0, 3, 7, 59, 60], "describ": [0, 3, 4, 7, 59], "below": [0, 3, 4, 7, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59], "auth": [0, 4, 5, 60], "authent": [0, 5, 8, 60], "6878c6dd88224981967f67ee2a73f092": [0, 60], "valu": [0, 5, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "we": [0, 2, 3, 5, 7, 10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "strongli": 0, "chang": [0, 2, 5, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "string": [0, 5, 7, 60], "section": [0, 1, 3, 7, 8, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "domain": [0, 3, 5, 60], "hostnam": [0, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "specifi": [0, 5, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "current": [0, 3, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 60, 61], "switch": [0, 1, 4, 6, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 61], "match": [0, 7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59, 60], "defin": [0, 1, 2, 3, 4, 6, 8, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59, 60], "particular": [0, 1, 2, 7, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 61], "prio": 0, "brief": [0, 7], "explan": [0, 61], "prioriti": 0, "goe": 0, "here": [0, 4, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57], "run": [0, 2, 6, 7, 8, 10, 11, 13, 16, 17, 20, 21, 23, 25, 26, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 53, 55, 56, 58, 59], "opt": [0, 4, 60], "bin": [0, 60], "fqdn": [0, 8, 60], "exampl": [0, 1, 4, 5, 7, 10, 11, 12, 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 59, 61], "10": [0, 3, 5, 7, 9, 12, 13, 14, 15, 16, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 60], "254": [0, 3, 7, 9, 13, 14, 15, 16, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58], "97": [0, 3, 48], "33": [0, 12, 14, 22, 27, 32, 37, 42, 47, 52, 57], "softgate1": [0, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "6a284d55148f81728f932b28e9d020736c8f78e1950b3d576f6e679d90516df1": [0, 4], "host": [0, 4, 5, 6, 7, 8, 10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "keepaliv": 0, "collectd": 0, "frr": [0, 2, 7, 8, 59], "bgp": [0, 3, 6, 11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 59, 61], "daemon": [0, 59], "router": [0, 2, 7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "systemd": 0, "unit": [0, 3, 7], "grub": [0, 60], "attent": 0, "must": [0, 5], "reboot": [0, 4, 60], "complet": [0, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "server": [0, 2, 3, 4, 5, 9, 10, 11, 12, 14, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59], "boot": [0, 60], "see": [0, 1, 3, 4, 5, 7, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 61], "": [0, 1, 2, 3, 5, 7, 10, 12, 16, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "heartbeat": [0, 60], "statu": [0, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "net": [0, 3, 6, 7, 8, 11, 12, 13, 16, 18, 21, 22, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 60, 61], "inventori": [0, 6, 8, 59, 60], "access": [1, 2, 3, 5, 6, 7, 8, 11, 12, 13, 16, 18, 21, 22, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61], "descript": [1, 5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60, 61], "field": [1, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 61], "usernam": [1, 3, 5, 60], "uniqu": [1, 7, 59], "full": [1, 2, 7, 59], "mail": [1, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "email": [1, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59], "also": [1, 2, 3, 5, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 61], "system": [1, 2, 3, 4, 7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59, 61], "notif": [1, 59], "password": [1, 3, 4, 5, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 60], "retriev": 1, "cc": 1, "send": [1, 2, 7], "copi": [1, 2, 16, 60], "phone": 1, "compani": [1, 59], "work": [1, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "usual": [1, 3, 7], "multi": [1, 59], "where": [1, 3, 5, 7, 9, 13, 14, 16, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59], "provid": [1, 3, 4, 5, 6, 8, 10, 11, 12, 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 59, 61], "netri": [1, 3, 7, 8, 9, 10, 11, 14, 17, 18, 19, 20, 21, 24, 25, 26, 29, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 61], "custom": [1, 2, 5, 10, 11, 13, 16, 17, 20, 21, 23, 25, 26, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 53, 55, 56, 58, 59, 60, 61], "posit": [1, 3], "within": [1, 7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "object": [1, 3, 6, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "rbac": [1, 59], "base": [1, 2, 3, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "deactiv": 1, "view": [1, 9, 13, 14, 16, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 61], "edit": [1, 3, 4, 7, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59], "part": [1, 7, 59, 61], "servic": [1, 2, 3, 4, 5, 7, 8, 9, 11, 12, 13, 14, 16, 18, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 60, 61], "port": [1, 3, 4, 5, 7, 9, 10, 12, 14, 16, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 60, 61], "resourc": [1, 3, 5, 7, 11, 21, 26, 31, 36, 41, 46, 56, 59], "assign": [1, 3, 5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "variou": [1, 7, 8, 61], "creat": [1, 3, 5, 7, 8, 9, 11, 12, 13, 14, 16, 18, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61], "read": [1, 59], "onli": [1, 2, 5, 6, 7, 59, 60, 61], "ani": [1, 3, 7, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59], "admin": [1, 3, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "To": [1, 3, 5, 7, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "form": [1, 2, 7, 8, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "go": [1, 2, 3, 5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "click": [1, 3, 4, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 61], "menu": [1, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 60, 61], "right": [1, 3, 4, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 61], "side": [1, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 61], "can": [1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 61], "under": [1, 3, 7, 13, 23, 28, 33, 38, 43, 48, 53, 58, 59, 61], "own": [1, 3, 5], "concept": [1, 6], "share": [1, 3, 7, 59], "deleg": [1, 3], "typic": [1, 3, 7], "team": [1, 3, 59], "grant": [1, 3, 59], "request": [1, 3, 5, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59], "self": [1, 3, 7, 60], "portal": [1, 3], "programmat": [1, 3], "kubernet": [1, 2, 3, 4, 5, 8, 11, 18, 21, 26, 31, 36, 41, 46, 51, 56, 59], "crd": [1, 3, 12, 22, 27, 32, 37, 42, 47, 52, 57], "devop": [1, 3, 6], "netop": [1, 2, 3, 5, 6, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58], "pipelin": [1, 3], "ha": [1, 2, 3, 4, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "just": [1, 2, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "two": [1, 3, 7, 9, 12, 13, 14, 16, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 61], "ad": [1, 6, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "per": [1, 3, 7, 59, 61], "basi": [1, 3, 59], "attach": [1, 3, 7, 59], "individu": [1, 3, 7, 59], "everi": [1, 3, 4, 7, 59], "attribut": [1, 3, 7, 8, 61], "either": [1, 2, 7, 59], "link": [1, 3, 59, 61], "directli": [1, 2, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "an": [1, 2, 3, 4, 7, 8, 9, 10, 11, 13, 14, 16, 17, 19, 20, 21, 23, 24, 25, 26, 28, 29, 30, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 53, 54, 55, 56, 58, 59, 60], "which": [1, 2, 3, 7, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 61], "automat": [2, 3, 5, 6, 7, 8, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58, 59, 60, 61], "softwar": [2, 6, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "oper": [2, 3, 7, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58], "like": [2, 6, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "cloud": [2, 4, 6, 8, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59], "configur": [2, 4, 6, 7, 8, 10, 11, 12, 13, 16, 17, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 59, 61], "load": [2, 6, 7, 8, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58, 61], "balanc": [2, 6, 7, 8, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58, 61], "secur": [2, 3, 5, 6, 8, 59], "user": [2, 3, 5, 6, 12, 13, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59, 61], "continu": 2, "monitor": [2, 3, 61], "health": [2, 59, 60, 61], "appli": [2, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "remedi": [2, 12, 22, 27, 32, 37, 42, 47, 52, 57], "inform": [2, 7], "necessari": 2, "action": [2, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "human": 2, "intervent": 2, "requir": [2, 3, 4, 6, 7, 8, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 60], "abstract": 2, "awai": [2, 59], "complex": 2, "detail": [2, 7, 13, 23, 28, 33, 38, 43, 48, 53, 58, 61], "let": [2, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "effici": 2, "top": [2, 4, 7, 61], "down": [2, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 61], "approach": 2, "instead": [2, 5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "legaci": [2, 59], "box": [2, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "center": [2, 3, 7, 59], "engin": [2, 6], "gui": [2, 3, 4, 5, 7, 8, 9, 10, 11, 14, 17, 18, 19, 20, 21, 24, 25, 26, 29, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59], "restapi": [2, 3], "devic": [2, 3, 7, 9, 10, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "store": [2, 4], "data": [2, 3, 7, 8, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59, 61], "repres": [2, 3, 59], "statist": [2, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 61], "analyt": 2, "receiv": [2, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "from": [2, 3, 4, 5, 7, 9, 10, 14, 16, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 61], "integr": [2, 8, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58, 59], "modul": 2, "extern": [2, 3, 5, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 61], "vm": [2, 4, 7, 59], "contain": [2, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "prem": [2, 6, 8, 11, 13, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58], "diagram": [2, 11, 18, 21, 26, 31, 36, 41, 46, 51, 56], "high": [2, 7, 59], "level": [2, 12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "architectur": [2, 7, 59], "highli": 2, "more": [2, 5, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "than": 2, "one": [2, 3, 5, 7, 8, 10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 59, 61], "databas": [2, 5], "replic": 2, "multipl": [2, 59, 61], "site": [2, 3, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59], "design": [2, 8, 59], "singl": [2, 5, 7, 59, 61], "unreach": [2, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "toler": [2, 5, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "stat": [2, 59], "collect": 2, "unavail": 2, "dure": 2, "window": [2, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "howev": 2, "affect": 2, "space": [2, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "NOS": [2, 3, 60], "respons": [2, 10, 13, 16, 17, 20, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 59], "gener": [2, 4, 5, 7, 8, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59], "accord": [2, 3], "encrypt": [2, 7, 8], "grpc": 2, "protocol": [2, 3, 7, 11, 12, 18, 21, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 59], "commun": [2, 3, 6, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "through": [2, 3, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "local": [2, 3, 4, 5, 7, 10, 12, 16, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "internet": [2, 4, 7, 8, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 60], "enabl": [2, 3, 5, 6, 12, 22, 27, 32, 37, 42, 47, 52, 57], "border": [2, 11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58], "layer": [2, 59], "4": [2, 7, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 60], "translat": [2, 7, 11, 18, 21, 26, 31, 36, 41, 46, 51, 56], "nat": [2, 3, 6, 8, 11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58], "vpn": [2, 3, 7, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "regular": [2, 12, 22, 27, 32, 37, 42, 47, 52, 57], "x86": 2, "support": [2, 4, 5, 7, 8, 59, 60, 61], "plane": [2, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "It": [2, 3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "packet": [2, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59, 61], "enter": [2, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "bypass": 2, "applic": [2, 4, 5, 8, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58, 59], "traffic": [2, 7, 59, 61], "travel": 2, "pcie": 2, "bu": 2, "closest": 2, "last": 2, "cach": [2, 7], "reserv": [2, 8], "process": [2, 3, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 60], "make": [2, 4, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "header": 2, "rewrit": 2, "mac": [2, 3, 7, 61], "vlan": [2, 7, 11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 59], "id": [2, 7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "return": [2, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "further": [2, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "each": [2, 3, 5, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "One": [2, 3], "100": [2, 5, 7, 17], "usag": [2, 6, 59], "anoth": [2, 3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "standard": 2, "util": [2, 5, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59, 61], "wireguard": [2, 7, 8], "mesh": [2, 3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "tunnel": [2, 7, 8], "between": [2, 7, 8, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59], "dynam": [2, 3, 7, 8, 59], "account": [3, 5, 6], "newnet0p": [3, 4, 5, 60], "tenant": [3, 6, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "group": [3, 6, 59], "permiss": [3, 4, 6], "role": [3, 6, 60], "separ": 3, "deploy": [3, 5], "come": [3, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "preconfigur": 3, "entri": [3, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "global": [3, 7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "AS": [3, 7, 10, 12, 13, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "acl": [3, 6, 11, 12, 18, 21, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57], "type": [3, 5, 7, 8, 9, 10, 12, 14, 16, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 60, 61], "ipv4": [3, 6, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "ipv6": [3, 5, 6, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "status": [3, 7, 59, 61], "take": [3, 12, 22, 27, 32, 37, 42, 47, 52, 57], "basic": [3, 6], "agent": [3, 4, 5, 6, 8, 59], "peer": [3, 7, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57], "ibgp": 3, "ebgp": 3, "least": [3, 59], "identif": 3, "There": [3, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "need": [3, 5, 7, 8, 10, 12, 16, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 60, 61], "new": [3, 5, 7, 8, 10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 59, 61], "alloc": [3, 7, 59], "larg": [3, 7], "block": 3, "organ": [3, 7, 8, 59], "smaller": 3, "insid": [3, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "yet": 3, "mani": [3, 7, 59], "24": [3, 9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59], "netmanag": 3, "netloopback": 3, "screenshot": [3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60, 61], "after": [3, 4, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "timezon": [3, 8], "dn": [3, 4, 8], "ntp": [3, 8], "free": [3, 7, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "text": [3, 4], "allow": [3, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "ssh": [3, 4, 9, 10, 13, 14, 16, 17, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58, 59], "adjust": 3, "time": [3, 4, 7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 61], "common": [3, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "owner": [3, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "who": [3, 59], "administ": 3, "hardwar": [3, 6, 59, 61], "For": [3, 5, 6, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "spine": [3, 9, 13, 14, 16, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 60], "leaf": [3, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59, 60], "switchdev": [3, 6, 8], "sonic": 3, "product": [3, 4], "belong": [3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "featur": [3, 7, 8, 59], "zero": [3, 59], "touch": 3, "provis": [3, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "instal": [3, 6, 8, 11, 13, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58, 59], "experiment": 3, "WIll": 3, "sync": [3, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "real": [3, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "establish": [3, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "veri": [3, 12, 22, 27, 32, 37, 42, 47, 52, 57], "first": [3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "w": [3, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "tip": [3, 4, 59], "drag": [3, 7], "move": [3, 4, 7], "desir": [3, 5], "note": [3, 6, 7], "repeat": 3, "underli": [3, 5], "watch": [3, 59], "against": 3, "potenti": 3, "failur": [3, 59], "yaml": [3, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "file": [3, 4, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "wizard": 3, "v2": 3, "now": [3, 8, 10, 12, 16, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 61], "long": 3, "place": [3, 4], "With": 3, "loop": 3, "cabl": [3, 7], "releas": [3, 5, 6, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "upstream": [3, 7, 11, 18, 21, 26, 31, 36, 41, 46, 51, 56], "higher": [3, 8, 59], "capac": 3, "lift": 3, "next": [3, 7, 59, 60], "what": [3, 6, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "navig": [3, 7, 9, 10, 12, 14, 16, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 61], "find": [3, 9, 10, 14, 16, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "l2": 3, "l3": [3, 6, 8, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "order": [3, 5, 6, 7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57], "doesn": [3, 59], "t": [3, 4, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "matter": 3, "know": 3, "dedic": [3, 12, 22, 27, 32, 37, 42, 47, 52, 57], "do": [3, 4, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "non": [3, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "visual": [3, 61], "nfv": [3, 7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "snat": [3, 7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "dnat": [3, 7], "check": [3, 7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 61], "chapter": 3, "pool": [3, 7, 59], "roll": [3, 7], "deploi": [4, 5, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58], "three": [4, 7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "anytim": 4, "minim": [4, 6, 59], "16": [4, 5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "disk": [4, 61], "100gb": 4, "If": [4, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 60], "alreadi": [4, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "qemu": 4, "virt": 4, "imag": [4, 5, 60], "contact": 4, "cd": 4, "var": 4, "lib": 4, "libvirt": 4, "img": 4, "qcow2": 4, "definit": [4, 6], "xml": 4, "virsh": 4, "bind": [4, 7], "br": 4, "mgmt": 4, "exam": 4, "correct": 4, "bridg": [4, 59], "autostart": 4, "By": [4, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 61], "obtain": 4, "dhcp": [4, 60], "how": [4, 7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "consol": [4, 60], "credenti": [4, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "login": [4, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "forget": 4, "passwd": 4, "command": [4, 5, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "nameserv": [4, 60], "reload": 4, "sure": [4, 12, 22, 27, 32, 37, 42, 47, 52, 57], "browser": [4, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "don": [4, 59, 60], "corner": [4, 61], "cli": 4, "random": 4, "sha256sum": 4, "iri": [4, 9, 10, 13, 16, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58], "newli": [4, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "telescop": [4, 6, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 60], "sh": 4, "pleas": [4, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "safe": 4, "softgat": [4, 6, 7, 8, 10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 59], "nginx": [4, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "cert": 4, "pem": 4, "privat": [4, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "restart": [4, 12, 22, 27, 32, 37, 42, 47, 52, 57], "systemctl": 4, "platform": [5, 6], "12": [5, 14, 60], "3": [5, 8, 9, 13, 14, 16, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 60], "pv": 5, "provision": 5, "infrastructur": 5, "netrisai": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "github": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "io": 5, "step": [5, 6, 9, 12, 13, 14, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58], "namespac": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "kubectl": [5, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57], "strong": 5, "mystrongauthkei": 5, "date": [5, 59], "base64": 5, "md5sum": 5, "head": 5, "c": [5, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "32": [5, 10, 12, 17, 20, 22, 23, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "ingress": [5, 16, 59], "my": [5, 7, 61], "authkei": 5, "tabl": [5, 7, 59], "nameoverrid": 5, "partial": 5, "overrid": 5, "fullnam": 5, "templat": 5, "prepend": [5, 7], "nil": 5, "fullnameoverrid": 5, "fulli": [5, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "serviceaccount": 5, "true": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "annot": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "podannot": 5, "pod": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "podsecuritycontext": 5, "context": 5, "securitycontext": 5, "limit": [5, 7, 59], "nodeselector": 5, "label": [5, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "affin": 5, "weblogin": 5, "webpassword": 5, "mystrongkei": 5, "replicacount": 5, "replica": 5, "xcaa": 5, "web": [5, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "tag": [5, 7, 59], "whose": 5, "appvers": 5, "pullpolici": 5, "pull": 5, "ifnotpres": 5, "imagepullsecret": 5, "secret": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "clusterip": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "expos": [5, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59], "80": [5, 7, 12, 22, 27, 30, 32, 33, 37, 42, 47, 52, 57], "portnam": 5, "class": 5, "path": [5, 7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "accept": [5, 7], "pathtyp": 5, "tl": 5, "autosc": 5, "param": 5, "hpa": 5, "metric": 5, "fals": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "minreplica": 5, "min": [5, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "maxreplica": 5, "max": [5, 7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "targetcpuutilizationpercentag": 5, "target": 5, "api": [5, 6, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "443": 5, "secureport": 5, "secureportnam": 5, "wss": 5, "kuberi": 5, "extend": [5, 7], "bitnami": 5, "plugin": 5, "initdbscriptsconfigmap": 5, "configmap": 5, "initdb": 5, "script": 5, "changem": 5, "rootpassword": 5, "root": [5, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "exist": [5, 59], "momment": 5, "usestatefulset": 5, "statefulset": 5, "standalon": 5, "cluster": [5, 8, 11, 18, 21, 26, 31, 36, 41, 46, 51, 56, 59], "master": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "slave": 5, "topologi": [5, 6, 7, 8, 11, 12, 18, 21, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 60], "usepassword": 5, "disable_ipv6": 5, "relay_network": 5, "relai": 5, "cni": [5, 8, 11, 21, 26, 31, 36, 41, 46, 56, 59], "subnet": [5, 6, 7, 10, 13, 15, 16, 17, 20, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 59], "172": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "192": [5, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 60], "168": [5, 9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 60], "loadbalanc": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "argument": 5, "abov": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "addition": 5, "altern": [5, 7, 59], "while": [5, 7, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59], "f": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "svc": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "nnetri": [5, 12, 22, 27, 32, 37, 42, 47, 52, 57], "grep": 5, "experi": [6, 59], "content": [6, 11, 16, 18, 21, 26, 31, 36, 41, 46, 51, 56], "introduct": 6, "On": [6, 8, 11, 13, 16, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 53, 56, 58], "kvm": [6, 8], "machin": [6, 7, 8, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57], "hypervisor": [6, 7, 59], "harden": 6, "helm": [6, 12, 22, 27, 32, 37, 42, 47, 52, 57], "chart": [6, 12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "prerequisit": 6, "info": [6, 12, 22, 27, 32, 37, 42, 47, 52, 57], "uninstal": [6, 60], "profil": [6, 8, 59], "hairpin": [6, 8], "cumulu": 6, "bio": 6, "advanc": [6, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "map": 6, "rule": [6, 8, 10, 20, 25, 30, 35, 40, 45, 50, 55], "look": [6, 9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57], "glass": [6, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "v": [6, 7, 8, 11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 60, 61], "kubenet": [6, 8, 16], "roh": [6, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "anycast": [6, 8, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "lb": [6, 12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "l4": [6, 8, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "l4lb": [6, 8], "consum": [6, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "approv": [6, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "workflow": 6, "sequenc": [6, 7], "visibl": [6, 7], "graph": 6, "board": 6, "log": [6, 8, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "dashboard": [6, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 60], "neighbor": [7, 10, 12, 13, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "declar": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "inject": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "meet": 7, "session": [7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 61], "termin": [7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "autonom": 7, "remot": [7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "without": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "instruct": [7, 9, 12, 13, 14, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58], "fabric": [7, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58, 59], "ixp": 7, "ggc": 7, "googl": 7, "untag": [7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "end": [7, 59], "administr": [7, 59], "fine": 7, "expand": [7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "aka": 7, "multihop": 7, "choos": 7, "speaker": 7, "allowa": 7, "occurr": 7, "nlri": 7, "valid": [7, 60], "normal": [7, 61], "origin": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "inbound": [7, 59], "drop": [7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "exce": 7, "maximum": 7, "1000": 7, "handl": 7, "million": 7, "outbound": [7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "prefer": 7, "weight": 7, "permit": [7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 60], "deni": [7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "referenc": 7, "consist": 7, "possibl": [7, 59], "mandatori": 7, "le": [7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "len": 7, "ge": 7, "keyword": 7, "format": [7, 59], "aa": 7, "nn": 7, "65535": 7, "rang": [7, 59, 60], "well": [7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "known": [7, 59], "advertis": [7, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58, 59], "addit": [7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "associ": 7, "claus": 7, "hop": 7, "med": 7, "met": 7, "whether": 7, "manipul": 7, "method": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "appropri": [7, 59], "isp": [7, 11, 17, 18, 21, 26, 31, 36, 41, 46, 51, 56, 61], "situat": 7, "dual": [7, 59], "home": [7, 59], "temporari": 7, "interconnect": 7, "old": [7, 59], "migrat": 7, "behind": [7, 13, 23, 28, 33, 38, 43, 48, 53, 58, 59], "intern": [7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "destin": [7, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "toward": [7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "scope": 7, "null": 7, "point": 7, "back": [7, 10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "show": [7, 8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "actual": [7, 59, 60, 61], "leaf1": 7, "spine1": 7, "given": [7, 59], "larger": 7, "similar": [7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "carrier": 7, "grade": 7, "alwai": 7, "overload": 7, "few": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "assum": [7, 59], "proper": [7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "tcp": [7, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59], "udp": [7, 59], "icmp": [7, 59], "replac": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "silent": 7, "forward": [7, 59], "exempt": 7, "broader": [7, 59], "much": 7, "predefin": 7, "comment": [7, 59], "198": [7, 55], "51": [7, 12, 22, 27, 32, 37, 42, 47, 52, 57], "65": [7, 12, 22, 27, 28, 32, 37, 42, 47, 52, 57], "66": [7, 10, 12, 17, 20, 22, 25, 27, 28, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "1080": 7, "sitemesh": [7, 8], "particip": [7, 8, 59], "henc": 7, "learn": [7, 9, 11, 12, 14, 18, 19, 21, 22, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57], "reach": [7, 10, 13, 16, 17, 20, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58], "hub": 7, "carri": 7, "transit": 7, "major": 7, "spoke": 7, "small": 7, "offic": 7, "maintain": 7, "measur": 7, "underneath": 7, "mathemat": 7, "model": 7, "mode": [7, 59], "randomli": 7, "converg": 7, "tool": [7, 8], "perspect": 7, "left": [7, 61], "dropdown": [7, 61], "famili": 7, "summari": [7, 61], "adjac": [7, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "queri": 7, "lookup": 7, "rib": 7, "tracerout": 7, "conduct": 7, "determin": 7, "ping": [7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "execut": [7, 59], "evpn": [7, 59], "vni": 7, "distinguish": 7, "softgateperform": 8, "27mpp": 8, "100gbp": 8, "12mpp": 8, "wenow": 8, "withkubernet": 8, "purpos": [8, 10, 16, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "built": [8, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "kube": [8, 12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59], "demand": [8, 11, 21, 26, 31, 36, 41, 46, 56, 59], "andoth": 8, "tocompl": 8, "modern": 8, "comprehens": [8, 61], "call": [8, 61], "sent": [8, 61], "withth": 8, "abil": [8, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 61], "search": [8, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 61], "sort": [8, 61], "column": [8, 61], "filter": [8, 61], "bymethod": 8, "siteinterconnect": 8, "forwireguard": 8, "andautomat": 8, "fewclick": 8, "meshof": 8, "remov": [8, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "loopcabl": 8, "switchingentir": 8, "distribut": 8, "threedeploy": 8, "construct": 8, "easi": 8, "forip": 8, "improv": 8, "becom": [8, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "placefor": 8, "got": 8, "everyus": 8, "hide": 8, "comfort": 8, "onc": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60], "certain": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "been": [9, 12, 13, 14, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58], "pre": [9, 13, 14, 16, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58], "explor": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "interact": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "yourself": [9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57], "document": [9, 12, 13, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58], "visit": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "sandbox1": [9, 10, 12, 13], "vnet": [9, 10, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "examin": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "indic": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "vertic": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "dot": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "mai": [9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59], "verifi": [9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57], "properli": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "remain": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "unchang": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "retain": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "sw01": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "nyc": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "46": [9, 10, 12, 13, 14, 15, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 54, 55, 57, 58], "radio": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "button": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "row": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "choic": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "45": [9, 10, 12, 13, 14, 15, 16, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 54, 57, 58], "64": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "submit": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "result": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "output": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "srv04": [9, 12, 13, 14, 16, 17, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58], "thank": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "56": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "84": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "byte": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "icmp_seq": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "ttl": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "543": [9, 19, 24, 29, 34, 39, 44, 49, 54], "m": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "436": [9, 19, 24, 29, 34, 39, 44, 49, 54], "445": [9, 19, 24, 29, 34, 39, 44, 49, 54], "354": [9, 19, 24, 29, 34, 39, 44, 49, 54], "345": [9, 19, 24, 29, 34, 39, 44, 49, 54], "transmit": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "loss": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "4074m": [9, 19, 24, 29, 34, 39, 44, 49, 54], "rtt": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "avg": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "mdev": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "424": [9, 19, 24, 29, 34, 39, 44, 49, 54], "075": [9, 19, 24, 29, 34, 39, 44, 49, 54], "interest": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "asid": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "softgate2": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "respect": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "sw02": [9, 10, 12, 14, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "isp1": [9, 10, 13, 14, 17, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58], "bottom": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "screen": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "abl": [9, 10, 12, 14, 16, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "fault": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "isp2": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "observ": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "pubblic": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "demo": [9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58], "166": [9, 10, 13, 14, 16, 17, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58], "88": [9, 10, 13, 14, 16, 17, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58], "17": [9, 10, 13, 14, 16, 17, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58], "30064": [9, 13, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 54, 58], "introductori": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "repli": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "62": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "wan": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "even": [9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 61], "scenario": [9, 12, 14, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57], "u": [9, 10, 12, 13, 14, 16, 17, 19, 20, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58], "our": [9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "sandbox": [9, 10, 11, 12, 14, 17, 18, 19, 20, 21, 22, 24, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57], "impos": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "granular": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "implement": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "would": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 59], "disallow": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "give": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "srv05": [10, 12, 13, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "its": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60], "30065": [10, 13, 20, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 55, 58], "l": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "dev": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 60], "eth1": [10, 20, 25, 30, 35, 40, 45, 50, 55], "proto": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "onlink": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "keep": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "until": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59], "reachabl": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "g": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "exercis": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "easili": [10, 20, 25, 30, 35, 40, 45, 50, 55], "swp2": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "sw22": [10, 12, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "previous": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "outsid": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 59, 60], "world": [10, 13, 16, 17, 20, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58], "65007": [10, 12, 13, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "swp14": [10, 12, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "unselect": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "1012": [10, 12, 13], "38": [10, 12, 13, 20, 22, 23, 25, 27, 28, 30, 32, 33], "161": [10, 12, 13, 20, 22, 23, 25, 27, 28, 30, 32, 33], "22": [10, 12, 13, 14, 16, 17], "21": [10, 12, 13, 22, 27, 32, 37, 42, 47, 52, 57], "28": [10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "And": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "final": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "minut": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "both": [10, 12, 13, 16, 17, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 59], "page": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "green": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "successfulli": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "face": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "aim": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "6": [10, 12, 22, 27, 32, 37, 42, 47, 52, 57], "soon": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "23": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "durat": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "uc": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "notic": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "longer": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "pop": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "again": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "finish": [10, 12, 17, 20, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57], "member": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 61], "resum": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "welcom": [11, 18, 21, 26, 31, 36, 41, 46, 51, 56], "ethernet": [11, 18, 21, 26, 31, 36, 41, 46, 51, 56, 59], "vxlan": [11, 13, 16, 18, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 59], "exterior": [11, 18, 21, 26, 31, 36, 41, 46, 51, 56], "intro": [11, 21, 26, 31, 36, 41, 46, 56], "calico": [11, 21, 26, 31, 36, 41, 46, 56], "kubesprai": [12, 22, 27, 32, 37, 42, 47, 52, 57], "k8": [12, 13, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "apiserv": [12, 22, 27, 32, 37, 42, 47, 52, 57], "put": [12, 22, 27, 32, 37, 42, 47, 52, 57], "kubeconfig": [12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59], "try": [12, 22, 27, 32, 37, 42, 47, 52, 57], "mean": [12, 22, 27, 32, 37, 42, 47, 52, 57], "ve": [12, 22, 27, 32, 37, 42, 47, 52, 57], "6443": [12, 22, 27, 32, 37, 42, 47, 52, 57], "debug": [12, 22, 27, 32, 37, 42, 47, 52, 57], "diagnos": [12, 22, 27, 32, 37, 42, 47, 52, 57], "problem": [12, 22, 27, 32, 37, 42, 47, 52, 57], "dump": [12, 22, 27, 32, 37, 42, 47, 52, 57], "accomplish": [12, 22, 27, 32, 37, 42, 47, 52, 57], "manifest": [12, 22, 27, 32, 37, 42, 47, 52, 57], "latest": [12, 22, 27, 32, 37, 42, 47, 52, 57], "v0": [12, 22, 27, 32, 37, 42, 47, 52, 57], "7": [12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "cred": [12, 22, 27, 32, 37, 42, 47, 52, 57], "liter": [12, 22, 27, 32, 37, 42, 47, 52, 57], "pass": [12, 22, 27, 32, 37, 42, 47, 52, 57], "inspect": [12, 22, 27, 32, 37, 42, 47, 52, 57], "demonstr": [12, 22, 27, 32, 37, 42, 47, 52, 57], "success": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "1629994653": [12, 22, 27, 32, 37, 42, 47, 52, 57], "6441543": [12, 22, 27, 32, 37, 42, 47, 52, 57], "logger": [12, 22, 27, 32, 37, 42, 47, 52, 57], "msg": [12, 22, 27, 32, 37, 42, 47, 52, 57], "worker": [12, 22, 27, 32, 37, 42, 47, 52, 57], "reconcilergroup": [12, 22, 27, 32, 37, 42, 47, 52, 57], "reconcilerkind": [12, 22, 27, 32, 37, 42, 47, 52, 57], "count": [12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "simpl": [12, 22, 27, 32, 37, 42, 47, 52, 57], "podinfo": [12, 22, 27, 32, 37, 42, 47, 52, 57], "k": [12, 22, 27, 32, 37, 42, 47, 52, 57], "stefanprodan": [12, 22, 27, 32, 37, 42, 47, 52, 57], "kustom": [12, 22, 27, 32, 37, 42, 47, 52, 57], "po": [12, 22, 27, 32, 37, 42, 47, 52, 57], "As": [12, 22, 27, 32, 37, 42, 47, 52, 57], "readi": [12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "ag": [12, 22, 27, 32, 37, 42, 47, 52, 57], "576d5bf6bd": [12, 22, 27, 32, 37, 42, 47, 52, 57], "7z9jl": [12, 22, 27, 32, 37, 42, 47, 52, 57], "49": [12, 22, 27, 32, 37, 42, 47, 52, 57], "nhlmh": [12, 22, 27, 32, 37, 42, 47, 52, 57], "106": [12, 22, 27, 32, 37, 42, 47, 52, 53, 57], "none": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "9898": [12, 22, 27, 32, 37, 42, 47, 52, 57], "9999": [12, 22, 27, 32, 37, 42, 47, 52, 57], "50": [12, 16, 17, 22, 27, 32, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "patch": [12, 22, 27, 32, 37, 42, 47, 52, 57], "spec": [12, 22, 27, 32, 37, 42, 47, 52, 57], "pend": [12, 22, 27, 32, 37, 42, 47, 52, 57], "32584": [12, 22, 27, 32, 37, 42, 47, 52, 57], "30365": [12, 22, 27, 32, 37, 42, 47, 52, 57], "8m57": [12, 22, 27, 32, 37, 42, 47, 52, 57], "becaus": [12, 22, 27, 32, 37, 42, 47, 52, 57], "xxxxxxxx": [12, 22, 27, 32, 37, 42, 47, 52, 57], "9m17": [12, 22, 27, 32, 37, 42, 47, 52, 57], "curl": [12, 22, 27, 32, 37, 42, 47, 52, 57], "rememb": [12, 22, 27, 32, 37, 42, 47, 52, 57], "previou": [12, 22, 27, 32, 37, 42, 47, 52, 57], "revis": [12, 22, 27, 32, 37, 42, 47, 52, 57], "color": [12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "34577c": [12, 22, 27, 32, 37, 42, 47, 52, 57], "logo": [12, 22, 27, 32, 37, 42, 47, 52, 57], "raw": [12, 22, 27, 32, 37, 42, 47, 52, 57], "githubusercont": [12, 22, 27, 32, 37, 42, 47, 52, 57], "gh": [12, 22, 27, 32, 37, 42, 47, 52, 57], "cuddle_clap": [12, 22, 27, 32, 37, 42, 47, 52, 57], "gif": [12, 22, 27, 32, 37, 42, 47, 52, 57], "messag": [12, 22, 27, 32, 37, 42, 47, 52, 57], "greet": [12, 22, 27, 32, 37, 42, 47, 52, 57], "v6": [12, 22, 27, 32, 37, 42, 47, 52, 57], "goo": [12, 22, 27, 32, 37, 42, 47, 52, 57], "goarch": [12, 22, 27, 32, 37, 42, 47, 52, 57], "amd64": [12, 22, 27, 32, 37, 42, 47, 52, 57], "runtim": [12, 22, 27, 32, 37, 42, 47, 52, 57], "go1": [12, 22, 27, 32, 37, 42, 47, 52, 57], "num_goroutin": [12, 22, 27, 32, 37, 42, 47, 52, 57], "num_cpu": [12, 22, 27, 32, 37, 42, 47, 52, 57], "seen": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "develop": [12, 22, 27, 32, 37, 42, 47, 52, 57], "decid": [12, 22, 27, 32, 37, 42, 47, 52, 57], "json": [12, 22, 27, 32, 37, 42, 47, 52, 57], "op": [12, 22, 27, 32, 37, 42, 47, 52, 57], "wait": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "second": [12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "congratul": [12, 22, 27, 32, 37, 42, 47, 52, 57], "ic": [12, 22, 27, 32, 37, 42, 47, 52, 57], "coffe": [12, 22, 27, 32, 37, 42, 47, 52, 57], "creation": [12, 22, 27, 32, 37, 42, 47, 52, 57], "detect": [12, 22, 27, 32, 37, 42, 47, 52, 57], "them": [12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "frontend": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "66d44feb": [12, 22, 27, 32, 37, 42, 47, 52, 57], "0278": [12, 22, 27, 32, 37, 42, 47, 52, 57], "412a": [12, 22, 27, 32, 37, 42, 47, 52, 57], "a32d": [12, 22, 27, 32, 37, 42, 47, 52, 57], "73afe011f2c6": [12, 22, 27, 32, 37, 42, 47, 52, 57], "activ": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "ok": [12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "33m": [12, 22, 27, 32, 37, 42, 47, 52, 57], "32m": [12, 22, 27, 32, 37, 42, 47, 52, 57], "recreat": [12, 22, 27, 32, 37, 42, 47, 52, 57], "wa": [12, 22, 27, 32, 37, 42, 47, 52, 57], "being": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "nativ": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "schema": [12, 22, 27, 32, 37, 42, 47, 52, 57], "backend": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 61], "These": [12, 22, 27, 32, 37, 42, 47, 52, 57], "present": [12, 22, 27, 32, 37, 42, 47, 52, 57], "index": [12, 22, 27, 32, 37, 42, 47, 52, 57], "html": [12, 22, 27, 32, 37, 42, 47, 52, 57], "cat": [12, 22, 27, 32, 37, 42, 47, 52, 57], "eof": [12, 22, 27, 32, 37, 42, 47, 52, 57], "apivers": [12, 22, 27, 32, 37, 42, 47, 52, 57], "v1alpha1": [12, 22, 27, 32, 37, 42, 47, 52, 57], "kind": [12, 22, 27, 32, 37, 42, 47, 52, 57], "metadata": [12, 22, 27, 32, 37, 42, 47, 52, 57], "ownerten": [12, 22, 27, 32, 37, 42, 47, 52, 57], "timeout": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "3000": [12, 22, 27, 32, 37, 42, 47, 52, 57], "d07acd0f": [12, 22, 27, 32, 37, 42, 47, 52, 57], "51ea": [12, 22, 27, 32, 37, 42, 47, 52, 57], "429a": [12, 22, 27, 32, 37, 42, 47, 52, 57], "89dd": [12, 22, 27, 32, 37, 42, 47, 52, 57], "8e4c1d6d0a86": [12, 22, 27, 32, 37, 42, 47, 52, 57], "2m17": [12, 22, 27, 32, 37, 42, 47, 52, 57], "3m47": [12, 22, 27, 32, 37, 42, 47, 52, 57], "11": 12, "displai": [12, 22, 27, 32, 37, 42, 47, 52, 57], "shown": [12, 22, 27, 32, 37, 42, 47, 52, 57], "didn": [12, 22, 27, 32, 37, 42, 47, 52, 57], "manual": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59, 60], "guestten": [12, 22, 27, 32, 37, 42, 47, 52, 57], "switchport": [12, 22, 27, 32, 37, 42, 47, 52, 57], "incom": [12, 22, 27, 32, 37, 42, 47, 52, 57], "sever": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "behavior": [12, 22, 27, 32, 37, 42, 47, 52, 57], "round": [12, 22, 27, 32, 37, 42, 47, 52, 57], "robin": [12, 22, 27, 32, 37, 42, 47, 52, 57], "intermitt": [12, 22, 27, 32, 37, 42, 47, 52, 57], "went": [12, 22, 27, 32, 37, 42, 47, 52, 57], "review": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "btw": [12, 22, 27, 32, 37, 42, 47, 52, 57], "appear": [12, 22, 27, 32, 37, 42, 47, 52, 57], "neighbora": [12, 22, 27, 32, 37, 42, 47, 52, 57], "transport": [12, 22, 27, 32, 37, 42, 47, 52, 57], "vlanid": [12, 22, 27, 32, 37, 42, 47, 52, 57], "localip": [12, 22, 27, 32, 37, 42, 47, 52, 57], "30": [12, 13, 16, 20, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 59], "remoteip": [12, 22, 27, 32, 37, 42, 47, 52, 57], "prefixlistoutbound": [12, 22, 27, 32, 37, 42, 47, 52, 57], "15": [12, 22, 27, 32, 37, 42, 47, 52, 57], "Then": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "00": [12, 22, 27, 32, 37, 42, 47, 52, 57], "2m3": [12, 22, 27, 32, 37, 42, 47, 52, 57], "feel": [12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "ui": [12, 22, 27, 32, 37, 42, 47, 52, 57], "about": [12, 22, 27, 32, 37, 42, 47, 52, 57], "otherwis": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "complain": [12, 22, 27, 32, 37, 42, 47, 52, 57], "thei": [12, 22, 27, 32, 37, 42, 47, 52, 57], "won": [12, 22, 27, 32, 37, 42, 47, 52, 57], "everyth": [12, 22, 27, 32, 37, 42, 47, 52, 57], "suppos": [12, 22, 27, 32, 37, 42, 47, 52, 57], "want": [12, 22, 27, 32, 37, 42, 47, 52, 57, 61], "prevent": [12, 22, 27, 32, 37, 42, 47, 52, 57], "reclaimpolici": [12, 22, 27, 32, 37, 42, 47, 52, 57], "Or": [12, 22, 27, 32, 37, 42, 47, 52, 57], "ll": [12, 16, 22, 27, 32, 37, 42, 47, 52, 57, 59], "tor": [12, 22, 27, 32, 37, 42, 47, 52, 57], "clean": [12, 22, 27, 32, 37, 42, 47, 52, 57], "understand": [12, 22, 27, 32, 37, 42, 47, 52, 57], "why": [12, 22, 27, 32, 37, 42, 47, 52, 57], "doc": [12, 22, 27, 32, 37, 42, 47, 52, 57], "bgpconfigur": [12, 22, 27, 32, 37, 42, 47, 52, 57], "nodetonodemeshen": [12, 22, 27, 32, 37, 42, 47, 52, 57], "asnumb": [12, 22, 27, 32, 37, 42, 47, 52, 57], "64512": [12, 22, 27, 32, 37, 42, 47, 52, 57], "projectcalico": [12, 22, 27, 32, 37, 42, 47, 52, 57], "org": [12, 22, 27, 32, 37, 42, 47, 52, 57], "v1": [12, 22, 27, 32, 37, 42, 47, 52, 57], "logseverityscreen": [12, 22, 27, 32, 37, 42, 47, 52, 57], "06": [12, 22, 27, 32, 37, 42, 47, 52, 57], "7m59": [12, 22, 27, 32, 37, 42, 47, 52, 57], "srv06": [12, 22, 27, 32, 37, 42, 47, 52, 57], "110": [12, 13, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 50, 52, 53, 57, 58], "4200070000": [12, 22, 27, 32, 37, 42, 47, 52, 57], "26": [12, 22, 23, 27, 32, 37, 42, 47, 52, 57], "srv07": [12, 22, 27, 32, 37, 42, 47, 52, 57], "67": [12, 22, 27, 32, 37, 42, 47, 52, 57], "4200070001": [12, 22, 27, 32, 37, 42, 47, 52, 57], "srv08": [12, 22, 27, 32, 37, 42, 47, 52, 57], "68": [12, 22, 27, 32, 37, 42, 47, 52, 57], "4200070002": [12, 22, 27, 32, 37, 42, 47, 52, 57], "might": [12, 22, 27, 32, 37, 42, 47, 52, 57], "07": [12, 22, 27, 32, 37, 42, 47, 52, 57], "48": [12, 22, 25, 27, 28, 32, 37, 42, 47, 52, 57], "8m41": [12, 22, 27, 32, 37, 42, 47, 52, 57], "44": [12, 22, 27, 32, 37, 42, 47, 52, 57], "n": [12, 22, 27, 32, 37, 42, 47, 52, 57], "19": [12, 22, 27, 32, 37, 42, 47, 52, 54, 55, 57, 58], "still": [12, 22, 27, 32, 37, 42, 47, 52, 57], "earlier": [12, 22, 27, 32, 37, 42, 47, 52, 57], "ye": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "mfpdt": [12, 22, 27, 32, 37, 42, 47, 52, 57], "beer": [12, 22, 27, 32, 37, 42, 47, 52, 57], "found": [13, 23, 28, 33, 38, 43, 48, 53, 58], "question": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "slack": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "four": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "five": [13, 23, 28, 33, 38, 43, 48, 53, 58], "integrat": [13, 23, 28, 33, 38, 43, 48, 53, 58], "routabl": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "srv01": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "srv02": [13, 16, 17, 23, 28, 33, 38, 43, 48, 53, 58], "srv03": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "30061": [13, 23, 28, 33, 38, 43, 48, 58], "30062": [13, 23, 28, 33, 38, 43, 48, 58], "30063": [13, 23, 28, 33, 38, 43, 48, 58], "1011": 13, "creating_servic": 14, "sandbox10": [14, 16, 17], "27": 14, "702": 14, "37": 14, "609": 14, "4008m": 14, "660": 14, "422": 14, "14": 14, "545": 14, "23064": [14, 16, 17], "behe": 16, "23061": 16, "23062": 16, "23063": 16, "favorit": 16, "serv": 16, "117": [16, 17, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "59": [16, 17, 27, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58], "208": [16, 17], "801": 16, "122": 16, "121": 16, "802": [16, 17], "126": [16, 17], "125": 16, "eth2": 17, "sw21": 17, "127": 17, "213": 17, "sandbox2": [19, 20, 22, 23], "190": [19, 20, 23], "1022": [20, 22, 23], "29": [20, 22, 23, 49, 50, 53], "42": [22, 49, 50, 52, 53], "43": 22, "1021": 23, "25": 23, "sandbox3": [24, 25, 27, 28], "189": [24, 25, 28], "1032": [25, 27, 28], "70": [25, 27, 28], "69": [25, 27, 28], "54": 25, "58": 27, "1031": 28, "sandbox4": [29, 30, 32, 33], "188": [29, 30, 33], "1042": [30, 32, 33], "78": [30, 32, 33], "77": [30, 32, 33], "86": [30, 35, 37, 38], "90": [32, 43], "91": 32, "1041": 33, "74": 33, "73": 33, "sandbox5": [34, 35, 37, 38], "187": [34, 35, 38, 52], "1052": [35, 37, 38], "85": [35, 37, 38], "128": [35, 37, 38], "134": 35, "138": 37, "139": 37, "1051": 38, "82": 38, "81": 38, "sandbox6": [39, 40, 42, 43], "186": [39, 40, 43, 52], "1062": [40, 42, 43], "94": [40, 42, 43], "93": [40, 42, 43], "144": [40, 42, 43], "150": 40, "154": 42, "155": 42, "1061": 43, "89": 43, "sandbox7": [44, 45, 47, 48], "185": [44, 45, 48], "1072": [45, 47, 48], "102": [45, 47, 48], "101": [45, 47, 48], "160": [45, 47, 48], "170": 47, "171": 47, "1071": 48, "98": 48, "sandbox8": [49, 50, 52, 53], "41": [49, 52, 53], "22864": [49, 53], "22865": [50, 53], "1082": [50, 52, 53], "109": [50, 52, 53], "176": [50, 52, 53], "182": 50, "108": [52, 53], "22861": 53, "22862": 53, "22863": 53, "1081": 53, "105": 53, "sandbox9": [54, 55, 57, 58], "1092": [55, 57, 58], "118": [55, 57, 58], "202": 57, "203": 57, "1091": 58, "114": 58, "113": 58, "unrout": 59, "segment": 59, "involv": 59, "anywher": 59, "collabor": 59, "unnumb": 59, "underlai": 59, "scene": 59, "entir": [59, 61], "awar": 59, "rare": 59, "guest": 59, "circuit": 59, "backbon": 59, "intend": 59, "span": 59, "alia": 59, "swp": 59, "signific": 59, "egress": 59, "unless": 59, "lag": 59, "standbi": 59, "futur": 59, "svi": 59, "nose": 59, "autodetect": 59, "1gbp": 59, "10gpb": 59, "speed": 59, "bulk": 59, "complement": 59, "fact": 59, "leverag": 59, "therefor": 59, "event": 59, "watcher": 59, "adapt": 59, "reflect": [59, 60], "resili": 59, "ecosystem": 59, "demon": 59, "bare": 59, "metal": 59, "most": 59, "commonli": 59, "thu": 59, "throughout": 59, "ecmp": 59, "capabl": 59, "instanc": 59, "interim": 59, "proxmox": 59, "Will": 59, "aggreg": 59, "inherit": 59, "sometim": 59, "b": 59, "255": 59, "expect": 59, "hash": 59, "deliv": 59, "rate": 59, "besid": [59, 61], "unicast": 59, "pair": 59, "rerout": 59, "extra": 59, "guid": 59, "requestor": 59, "probe": 59, "uncondition": 59, "millisecond": 59, "acl2": 59, "orient": 59, "wai": 59, "enforc": 59, "import": 59, "mind": 59, "size": 59, "tcam": 59, "optim": 59, "algorithm": 59, "achiev": 59, "sitedefault": 59, "code": 59, "icmpv6": 59, "revers": 59, "except": 59, "syn": 59, "flag": 59, "swap": 59, "qa_ten": 59, "stai": 59, "reject": 59, "push": 59, "rest": 59, "fresh": 60, "cl": 60, "past": 60, "press": 60, "ctrl": 60, "rfc1918": 60, "iptabl": 60, "accordingli": 60, "jessi": 60, "sw": 60, "critic": [60, 61], "fist": 60, "oni": 60, "stop": 60, "discoveri": 60, "addr": 60, "resolv": 60, "x86_64": 60, "mlnx_x86": 60, "r0": 60, "siwtchdev": 60, "nos": 60, "sum": 61, "open": 61, "later": 61, "mark": 61, "across": 61, "titl": 61, "bp": 61, "bit": 61, "pp": 61, "error": 61, "optic": 61, "signal": 61, "histori": 61, "iris1": 61, "iris2": 61, "summar": 61, "icon": 61, "pie": 61, "suppli": 61, "fan": 61, "temperatur": 61, "sensor": 61, "synchron": 61, "vip": 61, "alarm": 61}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"netri": [0, 2, 4, 5, 6, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 60], "softgat": [0, 2, 3], "agent": [0, 2, 60], "instal": [0, 4, 5, 12, 22, 27, 32, 37, 42, 47, 52, 57, 60], "minim": 0, "hardwar": 0, "requir": 0, "bio": 0, "configur": [0, 3, 5, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54, 60], "softwar": 0, "account": 1, "user": 1, "tenant": 1, "permiss": 1, "group": 1, "role": 1, "concept": 2, "introduct": [2, 12, 22, 27, 32, 37, 42, 47, 52, 57], "what": 2, "i": 2, "control": [2, 3, 4, 5, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59], "switch": [2, 3, 60], "initi": 3, "definit": 3, "subnet": 3, "exampl": [3, 9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "ip": [3, 60], "address": [3, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 60], "us": [3, 12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "ar": 3, "just": 3, "pleas": 3, "replac": [3, 4], "them": 3, "follow": 3, "your": 3, "plan": 3, "inventori": 3, "profil": 3, "field": 3, "descript": 3, "ad": 3, "topologi": [3, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "manag": [3, 60], "hairpin": 3, "cumulu": [3, 60], "onli": 3, "node": 3, "On": [4, 12, 22, 27, 32, 37, 42, 47, 52, 57], "prem": [4, 12, 22, 27, 32, 37, 42, 47, 52, 57], "kvm": 4, "virtual": 4, "machin": 4, "step": 4, "hypervisor": 4, "access": [4, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "secur": 4, "harden": 4, "chang": 4, "default": [4, 59], "grpc": [4, 5], "authent": 4, "kei": 4, "ssl": 4, "certif": 4, "helm": 5, "chart": 5, "prerequisit": 5, "get": [5, 6], "repo": 5, "info": 5, "uninstal": 5, "common": 5, "paramet": 5, "app": 5, "telescop": [5, 61], "k8": 5, "watcher": 5, "notifi": 5, "mariadb": 5, "mongodb": 5, "redi": 5, "smtp": 5, "haproxi": 5, "graphit": 5, "usag": 5, "welcom": [6, 13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "document": 6, "start": 6, "network": [6, 9, 10, 13, 14, 16, 17, 19, 20, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58], "polici": [6, 12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "servic": [6, 10, 17, 20, 25, 30, 35, 40, 45, 50, 55, 59], "oper": [6, 12, 22, 27, 32, 37, 42, 47, 52, 57], "updat": 6, "basic": 7, "bgp": [7, 9, 10, 12, 14, 17, 19, 20, 22, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57], "advanc": 7, "object": 7, "ipv4": 7, "prefix": 7, "ipv6": 7, "commun": 7, "rout": [7, 59], "map": 7, "static": 7, "nat": [7, 9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "enabl": [7, 59], "defin": 7, "rule": [7, 59], "look": 7, "glass": 7, "releas": 8, "note": 8, "provid": [9, 14, 19, 24, 29, 34, 39, 44, 49, 54], "v": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "net": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "ethernet": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "vlan": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "vxlan": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "e": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "exterior": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "border": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "gatewai": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "protocol": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "translat": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55], "acl": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "list": [9, 10, 14, 17, 19, 20, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59], "learn": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "creat": [10, 17, 20, 25, 30, 35, 40, 45, 50, 55], "sandbox1": 11, "run": [12, 22, 27, 32, 37, 42, 47, 52, 57], "an": [12, 22, 27, 32, 37, 42, 47, 52, 57], "kubernet": [12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "cluster": [12, 13, 16, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58], "automat": [12, 22, 27, 32, 37, 42, 47, 52, 57], "netop": [12, 22, 27, 32, 37, 42, 47, 52, 57], "intro": [12, 22, 27, 32, 37, 42, 47, 52, 57], "deploi": [12, 22, 27, 32, 37, 42, 47, 52, 57], "applic": [12, 22, 27, 32, 37, 42, 47, 52, 57], "demand": [12, 22, 27, 32, 37, 42, 47, 52, 57], "load": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "balanc": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "mileston": [12, 22, 27, 32, 37, 42, 47, 52, 57], "1": [12, 22, 27, 32, 37, 42, 47, 52, 57], "custom": [12, 22, 27, 32, 37, 42, 47, 52, 57], "resourc": [12, 22, 27, 32, 37, 42, 47, 52, 57], "l4lb": [12, 22, 27, 32, 37, 42, 47, 52, 57, 59], "vnet": [12, 22, 27, 32, 37, 42, 47, 52, 57], "import": [12, 22, 27, 32, 37, 42, 47, 52, 57], "exist": [12, 22, 27, 32, 37, 42, 47, 52, 57], "from": [12, 22, 27, 32, 37, 42, 47, 52, 57], "reclaim": [12, 22, 27, 32, 37, 42, 47, 52, 57], "calico": [12, 22, 27, 32, 37, 42, 47, 52, 57], "cni": [12, 22, 27, 32, 37, 42, 47, 52, 57], "integr": [12, 22, 27, 32, 37, 42, 47, 52, 57], "disabl": [12, 22, 27, 32, 37, 42, 47, 52, 57], "2": [12, 22, 27, 32, 37, 42, 47, 52, 57], "sandbox": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "diagram": [13, 15, 16, 23, 28, 33, 38, 43, 48, 53, 58], "gui": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "linux": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58, 60], "server": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "upstream": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "isp": [13, 16, 23, 28, 33, 38, 43, 48, 53, 58], "sandbox10": 18, "sandbox2": 21, "sandbox3": 26, "sandbox4": 31, "sandbox5": 36, "sandbox6": 41, "sandbox7": 46, "sandbox8": 51, "sandbox9": 56, "kubenet": 59, "roh": 59, "host": 59, "l3": 59, "anycast": 59, "lb": 59, "l4": 59, "consum": 59, "approv": 59, "workflow": 59, "The": 59, "sequenc": 59, "order": 59, "For": 60, "oob": 60, "licens": 60, "ubuntu": 60, "switchdev": 60, "visibl": 61, "graph": 61, "board": 61, "api": 61, "log": 61, "dashboard": 61}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.todo": 2, "sphinx.ext.viewcode": 1, "sphinx": 57}, "alltitles": {"Netris SoftGate agent installation": [[0, "netris-softgate-agent-installation"]], "Minimal hardware requirements": [[0, "minimal-hardware-requirements"]], "BIOS configuration": [[0, "bios-configuration"]], "Software installation": [[0, "software-installation"]], "Accounts": [[1, "accounts"]], "Users": [[1, "users"]], "Tenants": [[1, "tenants"]], "Permission Groups": [[1, "permission-groups"]], "User Roles": [[1, "user-roles"]], "Concepts": [[2, "concepts"]], "Introduction to Netris": [[2, "introduction-to-netris"]], "What is Netris Controller": [[2, "what-is-netris-controller"]], "Netris Switch Agent": [[2, "netris-switch-agent"]], "Netris SoftGate": [[2, "netris-softgate"]], "Controller initial configuration": [[3, "controller-initial-configuration"]], "Definitions": [[3, "definitions"]], "Subnets": [[3, "subnets"]], "Example: (IP addresses used are just examples, please replace them following your IP planning.)": [[3, "example-ip-addresses-used-are-just-examples-please-replace-them-following-your-ip-planning"]], "Inventory Profiles": [[3, "inventory-profiles"]], "Fields descriptions:": [[3, "fields-descriptions"]], "Adding Switches to Topology": [[3, "adding-switches-to-topology"]], "Topology Manager": [[3, "topology-manager"]], "Hairpin (Cumulus only)": [[3, "hairpin-cumulus-only"]], "Adding SoftGate nodes to Topology": [[3, "adding-softgate-nodes-to-topology"]], "On-prem Netris Controller installation": [[4, "on-prem-netris-controller-installation"]], "KVM virtual machine": [[4, "kvm-virtual-machine"]], "Installation steps for KVM hypervisor": [[4, "installation-steps-for-kvm-hypervisor"]], "Netris Controller Installation steps": [[4, "netris-controller-installation-steps"]], "Accessing the Netris Controller": [[4, "accessing-the-netris-controller"]], "Security hardening": [[4, "security-hardening"]], "Changing the default GRPC authentication key.": [[4, "changing-the-default-grpc-authentication-key"]], "Replacing the SSL certificate": [[4, "replacing-the-ssl-certificate"]], "Netris Controller Helm Chart": [[5, "netris-controller-helm-chart"]], "Prerequisites": [[5, "prerequisites"]], "Get Repo Info": [[5, "get-repo-info"]], "Installing the Chart": [[5, "installing-the-chart"]], "Uninstalling the Chart": [[5, "uninstalling-the-chart"]], "Configuration": [[5, "configuration"]], "Common parameters": [[5, "common-parameters"]], "Netris-Controller common parameters": [[5, "netris-controller-common-parameters"]], "Netris-Controller app parameters": [[5, "netris-controller-app-parameters"]], "Netris-Controller grpc parameters": [[5, "netris-controller-grpc-parameters"]], "Netris-Controller telescope parameters": [[5, "netris-controller-telescope-parameters"]], "Netris-Controller k8s-watcher parameters": [[5, "netris-controller-k8s-watcher-parameters"]], "Netris-Controller telescope-notifier parameters": [[5, "netris-controller-telescope-notifier-parameters"]], "Mariadb parameters": [[5, "mariadb-parameters"]], "Mongodb parameters": [[5, "mongodb-parameters"]], "Redis parameters": [[5, "redis-parameters"]], "Smtp parameters": [[5, "smtp-parameters"]], "HAproxy parameters": [[5, "haproxy-parameters"]], "Graphite parameters": [[5, "graphite-parameters"]], "Usage": [[5, "usage"]], "Welcome to Netris documentation!": [[6, "welcome-to-netris-documentation"]], "Getting Started": [[6, null]], "Network Policies": [[6, null]], "Network Services": [[6, null]], "Operations": [[6, null]], "Updates": [[6, null]], "Basic BGP": [[7, "basic-bgp"]], "Advanced BGP": [[7, "advanced-bgp"]], "BGP objects": [[7, "bgp-objects"]], "IPv4 Prefix.": [[7, "ipv4-prefix"]], "IPv6 Prefix.": [[7, "ipv6-prefix"]], "Community.": [[7, "community"]], "BGP route-maps": [[7, "bgp-route-maps"]], "Routes (static routing)": [[7, "routes-static-routing"]], "NAT": [[7, "nat"]], "Enabling NAT": [[7, "enabling-nat"]], "Defining NAT rules": [[7, "defining-nat-rules"]], "Looking Glass": [[7, "looking-glass"]], "Release notes": [[8, "release-notes"]], "Provided Example Configurations": [[9, "provided-example-configurations"], [14, "provided-example-configurations"], [19, "provided-example-configurations"], [24, "provided-example-configurations"], [29, "provided-example-configurations"], [34, "provided-example-configurations"], [39, "provided-example-configurations"], [44, "provided-example-configurations"], [49, "provided-example-configurations"], [54, "provided-example-configurations"]], "V-Net (Ethernet/Vlan/VXlan) Example": [[9, "v-net-ethernet-vlan-vxlan-example"], [14, "v-net-ethernet-vlan-vxlan-example"], [19, "v-net-ethernet-vlan-vxlan-example"], [24, "v-net-ethernet-vlan-vxlan-example"], [29, "v-net-ethernet-vlan-vxlan-example"], [34, "v-net-ethernet-vlan-vxlan-example"], [39, "v-net-ethernet-vlan-vxlan-example"], [44, "v-net-ethernet-vlan-vxlan-example"], [49, "v-net-ethernet-vlan-vxlan-example"], [54, "v-net-ethernet-vlan-vxlan-example"]], "E-BGP (Exterior Border Gateway Protocol) Example": [[9, "e-bgp-exterior-border-gateway-protocol-example"], [14, "e-bgp-exterior-border-gateway-protocol-example"], [19, "e-bgp-exterior-border-gateway-protocol-example"], [24, "e-bgp-exterior-border-gateway-protocol-example"], [29, "e-bgp-exterior-border-gateway-protocol-example"], [34, "e-bgp-exterior-border-gateway-protocol-example"], [39, "e-bgp-exterior-border-gateway-protocol-example"], [44, "e-bgp-exterior-border-gateway-protocol-example"], [49, "e-bgp-exterior-border-gateway-protocol-example"], [54, "e-bgp-exterior-border-gateway-protocol-example"]], "NAT (Network Address Translation) Example": [[9, "nat-network-address-translation-example"], [14, "nat-network-address-translation-example"], [19, "nat-network-address-translation-example"], [24, "nat-network-address-translation-example"], [29, "nat-network-address-translation-example"], [34, "nat-network-address-translation-example"], [39, "nat-network-address-translation-example"], [44, "nat-network-address-translation-example"], [49, "nat-network-address-translation-example"], [54, "nat-network-address-translation-example"]], "ACL (Access Control List) Example": [[9, "acl-access-control-list-example"], [14, "acl-access-control-list-example"], [19, "acl-access-control-list-example"], [24, "acl-access-control-list-example"], [29, "acl-access-control-list-example"], [34, "acl-access-control-list-example"], [39, "acl-access-control-list-example"], [44, "acl-access-control-list-example"], [49, "acl-access-control-list-example"], [54, "acl-access-control-list-example"]], "Learn by Creating Services": [[10, "learn-by-creating-services"], [17, "learn-by-creating-services"], [20, "learn-by-creating-services"], [25, "learn-by-creating-services"], [30, "learn-by-creating-services"], [35, "learn-by-creating-services"], [40, "learn-by-creating-services"], [45, "learn-by-creating-services"], [50, "learn-by-creating-services"], [55, "learn-by-creating-services"]], "V-Net (Ethernet/Vlan/VXlan)": [[10, "v-net-ethernet-vlan-vxlan"], [17, "v-net-ethernet-vlan-vxlan"], [20, "v-net-ethernet-vlan-vxlan"], [25, "v-net-ethernet-vlan-vxlan"], [30, "v-net-ethernet-vlan-vxlan"], [35, "v-net-ethernet-vlan-vxlan"], [40, "v-net-ethernet-vlan-vxlan"], [45, "v-net-ethernet-vlan-vxlan"], [50, "v-net-ethernet-vlan-vxlan"], [55, "v-net-ethernet-vlan-vxlan"]], "E-BGP (Exterior Border Gateway Protocol)": [[10, "e-bgp-exterior-border-gateway-protocol"], [17, "e-bgp-exterior-border-gateway-protocol"], [20, "e-bgp-exterior-border-gateway-protocol"], [25, "e-bgp-exterior-border-gateway-protocol"], [30, "e-bgp-exterior-border-gateway-protocol"], [35, "e-bgp-exterior-border-gateway-protocol"], [40, "e-bgp-exterior-border-gateway-protocol"], [45, "e-bgp-exterior-border-gateway-protocol"], [50, "e-bgp-exterior-border-gateway-protocol"], [55, "e-bgp-exterior-border-gateway-protocol"]], "NAT (Network Address Translation)": [[10, "nat-network-address-translation"], [17, "nat-network-address-translation"], [20, "nat-network-address-translation"], [25, "nat-network-address-translation"], [30, "nat-network-address-translation"], [35, "nat-network-address-translation"], [40, "nat-network-address-translation"], [45, "nat-network-address-translation"], [50, "nat-network-address-translation"], [55, "nat-network-address-translation"]], "ACL (Access Control List)": [[10, "acl-access-control-list"], [17, "acl-access-control-list"], [20, "acl-access-control-list"], [25, "acl-access-control-list"], [30, "acl-access-control-list"], [35, "acl-access-control-list"], [40, "acl-access-control-list"], [45, "acl-access-control-list"], [50, "acl-access-control-list"], [55, "acl-access-control-list"]], "Sandbox1": [[11, "sandbox1"]], "Run an On-Prem Kubernetes Cluster with Netris Automatic NetOps": [[12, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [22, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [27, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [32, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [37, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [42, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [47, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [52, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"], [57, "run-an-on-prem-kubernetes-cluster-with-netris-automatic-netops"]], "Intro": [[12, "intro"], [22, "intro"], [27, "intro"], [32, "intro"], [37, "intro"], [42, "intro"], [47, "intro"], [52, "intro"], [57, "intro"]], "Install Netris Operator": [[12, "install-netris-operator"], [22, "install-netris-operator"], [27, "install-netris-operator"], [32, "install-netris-operator"], [37, "install-netris-operator"], [42, "install-netris-operator"], [47, "install-netris-operator"], [52, "install-netris-operator"], [57, "install-netris-operator"]], "Deploy an Application with an On-Demand Netris Load Balancer": [[12, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [22, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [27, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [32, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [37, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [42, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [47, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [52, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [57, "deploy-an-application-with-an-on-demand-netris-load-balancer"]], "Milestone 1": [[12, null], [22, null], [27, null], [32, null], [37, null], [42, null], [47, null], [52, null], [57, null]], "Using Netris Custom Resources": [[12, "using-netris-custom-resources"], [22, "using-netris-custom-resources"], [27, "using-netris-custom-resources"], [32, "using-netris-custom-resources"], [37, "using-netris-custom-resources"], [42, "using-netris-custom-resources"], [47, "using-netris-custom-resources"], [52, "using-netris-custom-resources"], [57, "using-netris-custom-resources"]], "Introduction to Netris Custom Resources": [[12, "introduction-to-netris-custom-resources"], [22, "introduction-to-netris-custom-resources"], [27, "introduction-to-netris-custom-resources"], [32, "introduction-to-netris-custom-resources"], [37, "introduction-to-netris-custom-resources"], [42, "introduction-to-netris-custom-resources"], [47, "introduction-to-netris-custom-resources"], [52, "introduction-to-netris-custom-resources"], [57, "introduction-to-netris-custom-resources"]], "L4LB Custom Resource": [[12, "l4lb-custom-resource"], [22, "l4lb-custom-resource"], [27, "l4lb-custom-resource"], [32, "l4lb-custom-resource"], [37, "l4lb-custom-resource"], [42, "l4lb-custom-resource"], [47, "l4lb-custom-resource"], [52, "l4lb-custom-resource"], [57, "l4lb-custom-resource"]], "VNet Custom Resource": [[12, "vnet-custom-resource"], [22, "vnet-custom-resource"], [27, "vnet-custom-resource"], [32, "vnet-custom-resource"], [37, "vnet-custom-resource"], [42, "vnet-custom-resource"], [47, "vnet-custom-resource"], [52, "vnet-custom-resource"], [57, "vnet-custom-resource"]], "BGP Custom Resource": [[12, "bgp-custom-resource"], [22, "bgp-custom-resource"], [27, "bgp-custom-resource"], [32, "bgp-custom-resource"], [37, "bgp-custom-resource"], [42, "bgp-custom-resource"], [47, "bgp-custom-resource"], [52, "bgp-custom-resource"], [57, "bgp-custom-resource"]], "Importing existing resources from Netris Controller to Kubernetes": [[12, "importing-existing-resources-from-netris-controller-to-kubernetes"], [22, "importing-existing-resources-from-netris-controller-to-kubernetes"], [27, "importing-existing-resources-from-netris-controller-to-kubernetes"], [32, "importing-existing-resources-from-netris-controller-to-kubernetes"], [37, "importing-existing-resources-from-netris-controller-to-kubernetes"], [42, "importing-existing-resources-from-netris-controller-to-kubernetes"], [47, "importing-existing-resources-from-netris-controller-to-kubernetes"], [52, "importing-existing-resources-from-netris-controller-to-kubernetes"], [57, "importing-existing-resources-from-netris-controller-to-kubernetes"]], "Reclaim Policy": [[12, "reclaim-policy"], [22, "reclaim-policy"], [27, "reclaim-policy"], [32, "reclaim-policy"], [37, "reclaim-policy"], [42, "reclaim-policy"], [47, "reclaim-policy"], [52, "reclaim-policy"], [57, "reclaim-policy"]], "Netris Calico CNI Integration": [[12, "netris-calico-cni-integration"], [22, "netris-calico-cni-integration"], [27, "netris-calico-cni-integration"], [32, "netris-calico-cni-integration"], [37, "netris-calico-cni-integration"], [42, "netris-calico-cni-integration"], [47, "netris-calico-cni-integration"], [52, "netris-calico-cni-integration"], [57, "netris-calico-cni-integration"]], "Disabling Netris-Calico Integration": [[12, "disabling-netris-calico-integration"], [22, "disabling-netris-calico-integration"], [27, "disabling-netris-calico-integration"], [32, "disabling-netris-calico-integration"], [37, "disabling-netris-calico-integration"], [42, "disabling-netris-calico-integration"], [47, "disabling-netris-calico-integration"], [52, "disabling-netris-calico-integration"], [57, "disabling-netris-calico-integration"]], "Milestone 2": [[12, null], [22, null], [27, null], [32, null], [37, null], [42, null], [47, null], [52, null], [57, null]], "Welcome to Netris Sandbox": [[13, "welcome-to-netris-sandbox"], [16, "welcome-to-netris-sandbox"], [23, "welcome-to-netris-sandbox"], [28, "welcome-to-netris-sandbox"], [33, "welcome-to-netris-sandbox"], [38, "welcome-to-netris-sandbox"], [43, "welcome-to-netris-sandbox"], [48, "welcome-to-netris-sandbox"], [53, "welcome-to-netris-sandbox"], [58, "welcome-to-netris-sandbox"]], "Topology diagram": [[13, "topology-diagram"], [16, "topology-diagram"], [23, "topology-diagram"], [28, "topology-diagram"], [33, "topology-diagram"], [38, "topology-diagram"], [43, "topology-diagram"], [48, "topology-diagram"], [53, "topology-diagram"], [58, "topology-diagram"]], "Netris GUI": [[13, "netris-gui"], [16, "netris-gui"], [23, "netris-gui"], [28, "netris-gui"], [33, "netris-gui"], [38, "netris-gui"], [43, "netris-gui"], [48, "netris-gui"], [53, "netris-gui"], [58, "netris-gui"]], "Linux servers": [[13, "linux-servers"], [16, "linux-servers"], [23, "linux-servers"], [28, "linux-servers"], [33, "linux-servers"], [38, "linux-servers"], [43, "linux-servers"], [48, "linux-servers"], [53, "linux-servers"], [58, "linux-servers"]], "Kubernetes cluster": [[13, "kubernetes-cluster"], [16, "kubernetes-cluster"], [23, "kubernetes-cluster"], [28, "kubernetes-cluster"], [33, "kubernetes-cluster"], [38, "kubernetes-cluster"], [43, "kubernetes-cluster"], [48, "kubernetes-cluster"], [53, "kubernetes-cluster"], [58, "kubernetes-cluster"]], "Upstream ISP": [[13, "upstream-isp"], [16, "upstream-isp"], [23, "upstream-isp"], [28, "upstream-isp"], [33, "upstream-isp"], [38, "upstream-isp"], [43, "upstream-isp"], [48, "upstream-isp"], [53, "upstream-isp"], [58, "upstream-isp"]], "Networks used": [[13, "networks-used"], [16, "networks-used"], [23, "networks-used"], [28, "networks-used"], [33, "networks-used"], [38, "networks-used"], [43, "networks-used"], [48, "networks-used"], [53, "networks-used"], [58, "networks-used"]], "Diagram": [[15, "diagram"]], "Sandbox10": [[18, "sandbox10"]], "Sandbox2": [[21, "sandbox2"]], "Sandbox3": [[26, "sandbox3"]], "Sandbox4": [[31, "sandbox4"]], "Sandbox5": [[36, "sandbox5"]], "Sandbox6": [[41, "sandbox6"]], "Sandbox7": [[46, "sandbox7"]], "Sandbox8": [[51, "sandbox8"]], "Sandbox9": [[56, "sandbox9"]], "V-NET": [[59, "v-net"]], "Kubenet": [[59, "kubenet"]], "ROH (Routing on the Host)": [[59, "roh-routing-on-the-host"]], "L3 Load Balancer (Anycast LB)": [[59, "l3-load-balancer-anycast-lb"]], "L4 Load Balancer (L4LB)": [[59, "l4-load-balancer-l4lb"]], "Enabling L4LB service": [[59, "enabling-l4lb-service"]], "Consuming L4LB service": [[59, "consuming-l4lb-service"]], "Access Control Lists (ACL)": [[59, "access-control-lists-acl"]], "ACL Default Policy.": [[59, "acl-default-policy"]], "ACL rules": [[59, "acl-rules"]], "ACL approval workflow": [[59, "acl-approval-workflow"]], "The sequence order of ACL rules": [[59, "the-sequence-order-of-acl-rules"]], "Netris switch agent installation": [[60, "netris-switch-agent-installation"]], "For Cumulus Linux": [[60, "for-cumulus-linux"]], "Configure the OOB Management IP address": [[60, "configure-the-oob-management-ip-address"], [60, "id1"]], "Configure Cumulus Linux license": [[60, "configure-cumulus-linux-license"]], "Install the Netris Agent": [[60, "install-the-netris-agent"], [60, "id2"]], "For Ubuntu SwitchDev": [[60, "for-ubuntu-switchdev"]], "Visibility (Telescope)": [[61, "visibility-telescope"]], "Graph Boards": [[61, "graph-boards"]], "API Logs": [[61, "api-logs"]], "Dashboard": [[61, "dashboard"]]}, "indexentries": {}}) \ No newline at end of file diff --git a/en/2.9/services.html b/en/2.9/services.html new file mode 100644 index 0000000000..e8cc51ff8a --- /dev/null +++ b/en/2.9/services.html @@ -0,0 +1,685 @@ + + + + + + + + + + + V-NET — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

V-NET

+

V-NET is a virtual networking service. V-NETs can be used for Layer-2 (unrouted) or Layer-3 (routed) virtual network segments involving switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization).

+
+

Automatically, Netris will configure a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes.

+
+

V-Net fields.

+
    +
  • Name - Unique name for the V-NET.

  • +
  • Owner - Tenant, who can make any changes to current V-NET.

  • +
  • V-Net state - Active/Disable state for entire V-NET.

  • +
  • VLAN aware - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible.

  • +
  • Guest tenants - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters.

  • +
  • Sites - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites).

  • +
  • IPv4 Gateway - IPv4 address to be used as a default gateway in this V-NET. Should be configured under Net→Subnets as an assignment, assigned to the owner tenant, and available in the site where V-NET is intended to span.

  • +
  • IPv6 Gateway - IPv6 address to be used as a default gateway in this V-NET. Should be configured under Net→Subnets as an assignment, assigned to the owner tenant, and available in the site or sites where V-NET is intended to span.

  • +
  • Port - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports.

    +
      +
    • Enabled - Enable or disable individual Switch Port under current V-NET

    • +
    • Port Name - Switch Port format: <alias>(swp<number>)@<switch name>

    • +
    • VLAN ID / Untag - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port’s ingress/egress unless VLAN aware mode is used.

    • +
    • LAG Mode - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes).

    • +
    +
  • +
+

Tip: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+

Example: Adding a new V-NET.

+_images/new_VNET.png +

Example: Listing of V-NETs.

+_images/VNETs.png +

Example: Expanded view of a V-NET listing.

+_images/VNET_listing.png +
+
+

Kubenet

+

Kubenet is a network service purpose-built for Kubernetes cluster nodes. Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters.

+

The Gateway and Switch Port part of Kubenet is similar to the V-NET. In fact, it is leveraging a V-NET. Kubeconfig is for granting Netris Controller access to your Kube API. Kubenet therefore, dynamically leverages Netris L4LB and other services based on events that Netris kube-watcher (Kube API integration adapter) watches in your Kube API.

+

Description of Kubenet fields.

+
    +
  • Name - Unique name for the Kubenet.

  • +
  • Tenant - Tenant, who can make any changes to current Kubenet.

  • +
  • Site - Site where Kubernetes cluster belongs.

  • +
  • State - Active/Disable state for particular Kubenet service.

  • +
  • IPv4 Gateway - IPv4 address to be used as a default gateway for current Kubenet.

  • +
  • Port - Physical Switch Port anywhere on the switch fabric. Switch Port should be assigned to the owner tenant under Net→Switch Ports.

    +
      +
    • Enabled - Enable or disable individual Switch Port under current Kubenet.

    • +
    • Port Name - Switch Port format: <alias>(swp<number>)@<switch name>

    • +
    • VLAN ID / Untag - Specify a VLAN ID for tagging traffic on a per-port basis or set to Untag not to use tagging on a particular port.

    • +
    +
  • +
  • Kubeconfig - After installing the Kubernetes cluster, add your Kube config for granting Netris at least read-only access to the Kube API.

  • +
+

Tip: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+

Example: Adding a new Kubenet service.

+_images/new_kubenet.png +

Once Netris Controller establishes a connection with Kube API, status will reflect on the listing.

+

Screenshot: Listing of Kubenet services. Kube API connection is successful.

+_images/listing_kubenet.png +

Screenshot: Physical Switch Port statuses.

+_images/switch_port_statuses.png +

Screenshot: Statuses of on-demand load balancers (type: load-balancer)

+_images/load-balancer.png +
+
+

ROH (Routing on the Host)

+

To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly to their servers. Known as ROH (Routing on the Host).

+

In ROH architectures, servers use a routing demon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon is FRR.

+

Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. Thus, leveraging the Layer-3 network throughout the entire network down the servers.

+

ROH architecture with Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). +For each instance of ROH, you’ll need to create an ROH entry in Netris Controller.

+

Description of ROH instance fields:

+
    +
  • Name - Unique name for the ROH instance.

  • +
  • Site - Site where the current ROH instance belongs.

  • +
  • Type - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor.

  • +
  • ROH Routing Profile - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances.

    +
      +
    • Default route only (a most common choice) - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch.

    • +
    • Default + Aggregate - Will add prefixes of defined assignments + “Default” profile.

    • +
    • Full table - Will advertise all prefixes available in the routing table of the connected switch.

    • +
    • Inherit - will inherit policy from site objects defined under Net→Sites.

    • +
    +
  • +
  • Legacy Mode - Switch from default zero-config mode to using /30 IP addresses. Use for MS Windows Servers or other OS that doesn’t support FRR.

  • +
  • +Port - Physical Switch Ports anywhere on the network.

  • +
  • +IPv4 - IPv4 addresses for the loopback interface.

  • +
  • +Inbound Prefix List - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks.

  • +
+

Tip: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+

Example: Adding an ROH instance. (Yes, you can use A.B.C.0/32 and A.B.C.255/32)

+_images/ROH_instance.png +

Screenshot: Expanded view of ROH listing. BGP sessions are up, and the expected IP is in fact received from the actual ROH server. Traffic stats are available per port.

+_images/ROH_listing.png +
+
+

L3 Load Balancer (Anycast LB)

+

L3 (Anycast) load balancer is leveraging ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks.

+

ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server.

+

End-user traffic should be destined to the anycast IP address. Switch fabric will ECMP load balance the traffic towards every server, as well as will hash based on IP/Protocol/Port such that TCP sessions will keep complete between given end-user and server pair. Optionally health checks are available to reroute the traffic away in the event of application failure.

+

To configure L3 (Anycast) load balancing, edit an existing ROH instance entry and add an extra IPv4 address, and select Anycast. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances.

+

Example: Adding an Anycast IPv4 address

+_images/anycast_IPv4_address.png +

Example: Under Services→Load Balancer, you can find the listing of L3 (Anycast) Load Balancers, service statuses, and you can add/remove more ROH instances and/or health checks.

+_images/listing_L3.png +

Screenshot: L3 (Anycast) Load Balancer listing.

+_images/loadbalancer_listing.png +
+
+

L4 Load Balancer (L4LB)

+

Netris L4 Load Balancer (L4LB) is leveraging SoftGate(Linux router) nodes for providing Layer-4 load balancing service, including on-demand cloud load balancer with native integration with Kubernetes.

+
+

Enabling L4LB service

+

L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer).

+

The IP address pool for L4LB can be defined in the Net→Subnets section by adding an Allocation and setting the purpose field to ‘load-balancer.’ You can define multiple IP pools for L4LB at any given Site. See the below example.

+

Example: Adding a load-balancer IP pool assignment.

+_images/IP_pool_assignment.png +

Screenshot: Listing of Net→Subnets after adding a load-balancer assignment

+_images/NetSubnets_listing.png +
+
+

Consuming L4LB service

+

This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section.

+

Click +add under Services→L4 Load Balancer to request an L4LB service.

+

Add new L4 Load Balancer fields are described below:

+

General fields

+
    +
  • Name* - Unique name.

  • +
  • Protocol* - TCP or UDP.

  • +
  • Tenant* - Requestor Tenant should have access to the backend IP space.

  • +
  • Site* - Site where L4LB service is being requested for. Backends should belong on this site.

  • +
  • State* - Administrative state.

  • +
+

Frontend

+
    +
  • Address* - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses.

  • +
  • Port* - TCP or UDP port to be exposed.

  • +
+

Health-check

+
    +
  • Type* - Probe backends on service availability.

    +
      +
    • None - load balance unconditionally.

    • +
    • TCP - probe backend service availability through TCP connect checks.

    • +
    • HTTP - probe backend service availability through http GET checks.

    • +
    +
  • +
  • Timeout(ms)* - Probe timeout in milliseconds.

  • +
  • Request path* - Http request path.

  • +
+

Backend

+
    +
  • +Add - add a backend host.

  • +
  • Address - IP address of the backend host.

  • +
  • Port - Service port on the backend host.

  • +
  • Enabled - Administrative state of particular backend.

  • +
+

Example: Requesting an L4 Load Balancer service.

+_images/request_L4.png +

Example: Listing of L4 Load Balancer services

+_images/listing_L4.png +
+
+
+

Access Control Lists (ACL)

+

Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access.

+

Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches.

+

Screenshot: TCAM utilization can be seen under Net→Inventory

+_images/TCAM.png +

Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements.

+
+

ACL Default Policy.

+

The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules.

+

Example: Changing “ACL Default Policy” for the site “siteDefault”.

+_images/siteDefault.png +
+
+

ACL rules

+

ACL rules can be created, listed, edited, approved under Services→ACL.

+

Description of ACL fields. +General

+
    +
  • Name* - Unique name for the ACL entry.

  • +
  • Protocol* - IP protocol to match.

    +
      +
    • All - Any IP protocols.

    • +
    • IP - Specific IP protocol number.

    • +
    • TCP - TCP.

    • +
    • UDP - UDP.

    • +
    • ICMP ALL - Any IPv4 ICMP protocol.

    • +
    • ICMP Custom - Custom IPv4 ICMP code.

    • +
    • ICMPv6 ALL - Any IPv6 ICMP protocol.

    • +
    • ICMPv6 Custom - Custom IPv6 ICMP code.

    • +
    +
  • +
  • Active Until - Disable this rule at the defined date/time.

  • +
  • Action - Permit or Deny forwarding of matched packets.

  • +
  • Established/Reverse - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination.

  • +
+

Source/Destination - Source and destination addresses and ports to match.

+
    +
  • Source* IPv4/IPv6 - IPv4/IPv6 address.

  • +
  • Ports Type*

    +
      +
    • Port Range - Match on the port or a port range defined in this window.

    • +
    • Port Group - Match on a group of ports defined under Services→ ACL Port Group.

    • +
    +
  • +
  • From Port* - Port range starting from.

  • +
  • To Port* - Port range ending with.

  • +
  • Comment - Descriptive comment, commonly used for approval workflows.

  • +
  • Check button - Check if Another ACL on the system already permits the described network access.

  • +
+

Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established).

+_images/action_permit.png +

Example: “Check” shows that requested access is already provided by a broader ACL rule.

+_images/ACL_rule.png +
+
+

ACL approval workflow

+

When one Tenant (one team) needs to get network access to resources under the responsibility of another Tenant (another team), an ACL can be created but will activate only after approval of the Tenant responsible for the destination address resources. See the below example.

+

Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant.

+_images/ACL_approval.png +

Screenshot: ACL stays in “waiting for approval” state until approved.

+_images/waiting_approval.png +

Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it.

+_images/approve_reject.png +

Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric.

+_images/ACL_active.png +
+
+

The sequence order of ACL rules

+
    +
  1. User-defined Deny Rules

  2. +
  3. User-defined Permit Rules

  4. +
  5. Deny the rest

  6. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/switch-agent-installation.html b/en/2.9/switch-agent-installation.html new file mode 100644 index 0000000000..fbd78813ba --- /dev/null +++ b/en/2.9/switch-agent-installation.html @@ -0,0 +1,664 @@ + + + + + + + + + + + Netris switch agent installation — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris switch agent installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Netris switch agent installation

+
+

For Cumulus Linux

+

Requirements: +* Fresh install of Cumulus Linux v. 3.7.(x) - Cumulus 4.X is in the process of validation and will be supported in the next Netris release.

+
+

Configure the OOB Management IP address

+

Configure out of band management IP address, and in case Netris Controller is not in the same OOB network then configure a route to Netris Controller. No default route or other IP addresses should be configured.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <Management IP address/prefix length>
+        up ip ro add <Controller address> via <Management network gateway> #delete this line if Netris Controller is located in the same network with the switch.
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
+
+

Configure Cumulus Linux license

+
sudo cl-license -i
+
+
+

Copy/paste the Cumulus Linux license string then press ctrl-d.

+
+
+

Install the Netris Agent

+
    +
  1. Add netris repository using Netris Controller as an http proxy. Replace <Your Netris Controller address> with your actual Netris Controller address.

  2. +
+
+

Note

+

Netris Controller built-in proxy, by default, permits RFC1918 IP addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16). +If your management network is using IP addresses outside these ranges you will need to configure iptables on the Netris Controller accordingly.

+
+
export http_proxy=http://<Your Netris Controller address>:3128
+
+wget -qO - http://repo.netris.ai/repo/public.key | sudo apt-key add -
+
+echo "deb http://repo.netris.ai/repo/ jessie main" | sudo tee /etc/apt/sources.list.d/netris.list
+
+
+
    +
  1. Update the apt

  2. +
+
echo -e 'Acquire::http::Proxy "http://<Your Netris Controller address>:3128";\nAcquire::https::Proxy "http://<Your Netris Controller address>:3128";' | sudo tee -a /etc/apt/apt.conf.d/netris-proxy
+
+sudo apt update
+
+
+
    +
  1. Install Netris Agent and dependencies

  2. +
+
sudo apt install netris-sw
+
+
+
    +
  1. Initialize the switch using netris-setup

  2. +
+

Description of netris-setup parameters

+
--auth - Authentication key, "6878C6DD88224981967F67EE2A73F092" is the default key.
+--controller - IP address or domain name of Netris Controller.
+--hostname - The hostname for the current switch, this hostname should match the name defined in the Controller.
+--lo - IP address for the loopback interface, as it is defined in the controller.
+--type - Role of the switch in your topology: spine/leaf
+
+
+
sudo /opt/netris/bin/netris-setup --auth=<authentication key> --controller=<IP or FQDN> --hostname=<name> --lo=<loopback IP address> --type=<spine/leaf>
+
+
+
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+

Once the switch boots up you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and switch color will reflect its health in Net→Topology

+

Screenshot: Net→Inventory

+_images/inventory_heartbeat.png +
+
+
+

For Ubuntu SwitchDev

+
+

Note

+

Further installation requires a Console and Internet connectivity via management port!

+
+
    +
  1. NOS Uninstall

  2. +
+

Fist of all uninstall current NOS using Uninstall OS from grub menu:

+_images/uninstallOS.png +

Once the uninstallation is completed, the switch will reboot automatically.

+
    +
  1. Update ONIE

  2. +
+

Select Update ONIE from grub menu:

+_images/updateONIE.png +

In case you don’t have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually:

+
onie-discovery-stop
+ip addr add <management IP address/prefix> dev eth0
+ip route add default via <gateway of management network>
+echo "nameserver 1.1.1.1" > /etc/resolv.conf
+
+
+

Update ONIE to the supported version.

+
+

Note

+

ONIE image available for Mellanox switches only!

+
+
onie-self-update http://repo.netris.ai/repo/onie-updater-x86_64-mlnx_x86-r0
+
+
+
    +
  1. NOS Install

  2. +
+

Select Install OS from grub menu:

+_images/installOS.png +

In case you don’t have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually:

+
onie-discovery-stop
+ip addr add <management IP address/prefix> dev eth0
+ip route add default via <gateway of management network>
+echo "nameserver 1.1.1.1" > /etc/resolv.conf
+
+
+

Install Ubuntu-SiwtchDev from the Netris custom image:

+
onie-nos-install http://repo.netris.ai/repo/netris-ubuntu-18.04.1.bin
+
+
+

Default username/password

+

netris/newNet0ps

+
+

Configure the OOB Management IP address

+

Configure out of band management IP address, and in case Netris Controller is not in the same OOB network then configure a route to Netris Controller. No default route or other IP addresses should be configured.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+    address <Management IP address/prefix length>
+    up ip ro add <Controller address> via <Management network gateway> #delete this line if Netris Controller is located in the same network with the switch.
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
+
+

Install the Netris Agent

+
    +
  1. Add netris repository using Netris Controller as an http proxy. Replace <Your Netris Controller address> with your actual Netris Controller address.

  2. +
+
+

Note

+

Netris Controller built-in proxy, by default, permits RFC1918 IP addresses (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16). If your management network is using IP addresses outside these ranges you will need to configure iptables on the Netris Controller accordingly.

+
+
export http_proxy=http://<Your Netris Controller address>:3128
+
+wget -qO - http://repo.netris.ai/repo/public.key | sudo apt-key add -
+
+echo "deb http://repo.netris.ai/repo/ bionic main" | sudo tee /etc/apt/sources.list.d/netris.list
+
+
+
    +
  1. Update the apt

  2. +
+
echo -e 'Acquire::http::Proxy "http://<Your Netris Controller address>:3128";\nAcquire::https::Proxy "http://<Your Netris Controller address>:3128";' | sudo tee -a /etc/apt/apt.conf.d/netris-proxy
+
+sudo apt update
+
+
+
    +
  1. Install Netris Agent and dependencies

  2. +
+
sudo apt-get update && sudo apt-get install netris-sw
+
+
+
    +
  1. Initialize the switch using netris-setup

  2. +
+

Description of netris-setup parameters

+
--auth - Authentication key, "6878C6DD88224981967F67EE2A73F092" is the default key.
+--controller - IP address or domain name of Netris Controller.
+--hostname - The hostname for the current switch, this hostname should match the name defined in the Controller.
+--lo - IP address for the loopback interface, as it is defined in the controller.
+--type - Role of the switch in your topology: spine/leaf
+
+
+
sudo /opt/netris/bin/netris-setup --auth=<authentication key> --controller=<IP or FQDN> --hostname=<name> --lo=<loopback IP address> --type=<spine/leaf>
+
+
+
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/2.9/visibility.html b/en/2.9/visibility.html new file mode 100644 index 0000000000..3ef77ae6bb --- /dev/null +++ b/en/2.9/visibility.html @@ -0,0 +1,506 @@ + + + + + + + + + + + Visibility (Telescope) — Netris 2.9 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Visibility (Telescope)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Visibility (Telescope)

+
+

Graph Boards

+

You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view.

+

To start with Graph Boards, first, you need to add a new Graph Board.

+
    +
  1. Navigate to Telescope→Graph Boards, open the dropdown menu in the top left corner, then click +Add board.

  2. +
+_images/telescope.png +
    +
  1. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants.

  2. +
+_images/createboard.png +

Now you can add graphs by clicking +Add graph.

+

Description of +Add graph fields:

+
    +
  • Title - Title for the new graph.

  • +
  • Type - Type of data source.

    +
      +
    • Bps - Traffic bits per second.

    • +
    • Pps - Traffic packets per second.

    • +
    • Errors - Errors per second.

    • +
    • Optical - Optical signal statistics/history.

    • +
    • MAC Count - History of the number of MAC addresses on the port.

    • +
    +
  • +
  • Function - Currently, only summing is supported.

  • +
  • +Member - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port.

  • +
+

Example: Sum of traffic on two ISP(Iris1 + Iris2) links.

+_images/ISP_Iris.png +

Example: Sum of the traffic on all ports under the service called “my V-NET”

+_images/V_NET.png +

Screenshot: Listing of a Graph Board with the explanation of the controls.

+_images/graphboard.png +
+
+

API Logs

+

Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type.

+
+
+

Dashboard

+

Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems.

+

Telescope→Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner.

+

Description of the pie charts.

+
    +
  • Hardware Health - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions.

  • +
  • E-BGP - Statuses of external BGP sessions.

  • +
  • LB VIP - Statuses of Load Balancer frontend / VIP availability.

  • +
  • LB Members - Statuses of Load Balancer backend members.

  • +
+

By clicking on each title you can see the details of the checks on the right side.

+

Screenshot: Dashboard showing details of “Hardware Health.”

+_images/hardware_health.png +

Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state.

+

Screenshot: “Save as normal” on selected ports.

+_images/saveasnormal.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 2.9 + + +
+ + +
+
Versions
+ + +
2.9
+
+ + +
3.0
+ + + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/.buildinfo b/en/3.0/.buildinfo new file mode 100644 index 0000000000..2fe2f3e53c --- /dev/null +++ b/en/3.0/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: f667f46892e3cd54e916173467cbccd6 +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/en/3.0/.doctrees/SoftGate-PRO-installation.doctree b/en/3.0/.doctrees/SoftGate-PRO-installation.doctree new file mode 100644 index 0000000000..86b8ebb1e9 Binary files /dev/null and b/en/3.0/.doctrees/SoftGate-PRO-installation.doctree differ diff --git a/en/3.0/.doctrees/SoftGate-installation.doctree b/en/3.0/.doctrees/SoftGate-installation.doctree new file mode 100644 index 0000000000..5c3b97feb6 Binary files /dev/null and b/en/3.0/.doctrees/SoftGate-installation.doctree differ diff --git a/en/3.0/.doctrees/accounts.doctree b/en/3.0/.doctrees/accounts.doctree new file mode 100644 index 0000000000..b1110b9109 Binary files /dev/null and b/en/3.0/.doctrees/accounts.doctree differ diff --git a/en/3.0/.doctrees/acls.doctree b/en/3.0/.doctrees/acls.doctree new file mode 100644 index 0000000000..a542b65b01 Binary files /dev/null and b/en/3.0/.doctrees/acls.doctree differ diff --git a/en/3.0/.doctrees/controller-k3s-installation.doctree b/en/3.0/.doctrees/controller-k3s-installation.doctree new file mode 100644 index 0000000000..93596d8a93 Binary files /dev/null and b/en/3.0/.doctrees/controller-k3s-installation.doctree differ diff --git a/en/3.0/.doctrees/controller-k8s-installation.doctree b/en/3.0/.doctrees/controller-k8s-installation.doctree new file mode 100644 index 0000000000..08688a1da2 Binary files /dev/null and b/en/3.0/.doctrees/controller-k8s-installation.doctree differ diff --git a/en/3.0/.doctrees/controller-k8s-quickstart.doctree b/en/3.0/.doctrees/controller-k8s-quickstart.doctree new file mode 100644 index 0000000000..6a88b84f8b Binary files /dev/null and b/en/3.0/.doctrees/controller-k8s-quickstart.doctree differ diff --git a/en/3.0/.doctrees/controller-vm-installation.doctree b/en/3.0/.doctrees/controller-vm-installation.doctree new file mode 100644 index 0000000000..3f72680226 Binary files /dev/null and b/en/3.0/.doctrees/controller-vm-installation.doctree differ diff --git a/en/3.0/.doctrees/definitions.doctree b/en/3.0/.doctrees/definitions.doctree new file mode 100644 index 0000000000..ad9a04c62e Binary files /dev/null and b/en/3.0/.doctrees/definitions.doctree differ diff --git a/en/3.0/.doctrees/environment.pickle b/en/3.0/.doctrees/environment.pickle new file mode 100644 index 0000000000..a990389451 Binary files /dev/null and b/en/3.0/.doctrees/environment.pickle differ diff --git a/en/3.0/.doctrees/index.doctree b/en/3.0/.doctrees/index.doctree new file mode 100644 index 0000000000..ad886d1a00 Binary files /dev/null and b/en/3.0/.doctrees/index.doctree differ diff --git a/en/3.0/.doctrees/installation.doctree b/en/3.0/.doctrees/installation.doctree new file mode 100644 index 0000000000..d736b47b21 Binary files /dev/null and b/en/3.0/.doctrees/installation.doctree differ diff --git a/en/3.0/.doctrees/installing-netris-controller.doctree b/en/3.0/.doctrees/installing-netris-controller.doctree new file mode 100644 index 0000000000..88a487d883 Binary files /dev/null and b/en/3.0/.doctrees/installing-netris-controller.doctree differ diff --git a/en/3.0/.doctrees/introduction.doctree b/en/3.0/.doctrees/introduction.doctree new file mode 100644 index 0000000000..42f08ca00e Binary files /dev/null and b/en/3.0/.doctrees/introduction.doctree differ diff --git a/en/3.0/.doctrees/inventory-profiles.doctree b/en/3.0/.doctrees/inventory-profiles.doctree new file mode 100644 index 0000000000..75abb37c6f Binary files /dev/null and b/en/3.0/.doctrees/inventory-profiles.doctree differ diff --git a/en/3.0/.doctrees/ipam.doctree b/en/3.0/.doctrees/ipam.doctree new file mode 100644 index 0000000000..c18591500e Binary files /dev/null and b/en/3.0/.doctrees/ipam.doctree differ diff --git a/en/3.0/.doctrees/kubernetes-integration.doctree b/en/3.0/.doctrees/kubernetes-integration.doctree new file mode 100644 index 0000000000..86a4026287 Binary files /dev/null and b/en/3.0/.doctrees/kubernetes-integration.doctree differ diff --git a/en/3.0/.doctrees/l3-load-balancer.doctree b/en/3.0/.doctrees/l3-load-balancer.doctree new file mode 100644 index 0000000000..e308bb8e97 Binary files /dev/null and b/en/3.0/.doctrees/l3-load-balancer.doctree differ diff --git a/en/3.0/.doctrees/l4-load-balancer.doctree b/en/3.0/.doctrees/l4-load-balancer.doctree new file mode 100644 index 0000000000..0b12dbc612 Binary files /dev/null and b/en/3.0/.doctrees/l4-load-balancer.doctree differ diff --git a/en/3.0/.doctrees/netris-architecture.doctree b/en/3.0/.doctrees/netris-architecture.doctree new file mode 100644 index 0000000000..8fcb0a95c3 Binary files /dev/null and b/en/3.0/.doctrees/netris-architecture.doctree differ diff --git a/en/3.0/.doctrees/network-policies.doctree b/en/3.0/.doctrees/network-policies.doctree new file mode 100644 index 0000000000..dc34ffc122 Binary files /dev/null and b/en/3.0/.doctrees/network-policies.doctree differ diff --git a/en/3.0/.doctrees/reference-designs.doctree b/en/3.0/.doctrees/reference-designs.doctree new file mode 100644 index 0000000000..7242a6ba41 Binary files /dev/null and b/en/3.0/.doctrees/reference-designs.doctree differ diff --git a/en/3.0/.doctrees/release-notes.doctree b/en/3.0/.doctrees/release-notes.doctree new file mode 100644 index 0000000000..0111fa4795 Binary files /dev/null and b/en/3.0/.doctrees/release-notes.doctree differ diff --git a/en/3.0/.doctrees/roh.doctree b/en/3.0/.doctrees/roh.doctree new file mode 100644 index 0000000000..5dcb8a3e2a Binary files /dev/null and b/en/3.0/.doctrees/roh.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox1/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox1/configurations.doctree new file mode 100644 index 0000000000..a6f3ca6431 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox1/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox1/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox1/creating-services.doctree new file mode 100644 index 0000000000..7185abe512 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox1/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox1/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox1/index.doctree new file mode 100644 index 0000000000..bfbfa4ef91 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox1/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree new file mode 100644 index 0000000000..c59e710e39 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox1/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox1/sandbox-info.doctree new file mode 100644 index 0000000000..466a3251f8 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox1/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox10/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox10/configurations.doctree new file mode 100644 index 0000000000..b28bf42cc9 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox10/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox10/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox10/creating-services.doctree new file mode 100644 index 0000000000..95f881a665 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox10/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox10/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox10/index.doctree new file mode 100644 index 0000000000..3959348855 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox10/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree new file mode 100644 index 0000000000..ec6b7ffa99 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox10/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox10/sandbox-info.doctree new file mode 100644 index 0000000000..e45fe596ca Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox10/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox11/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox11/configurations.doctree new file mode 100644 index 0000000000..e082136fbf Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox11/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox11/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox11/creating-services.doctree new file mode 100644 index 0000000000..2df631858c Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox11/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox11/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox11/index.doctree new file mode 100644 index 0000000000..1f9aba3b30 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox11/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree new file mode 100644 index 0000000000..5c7f46bdce Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox11/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox11/sandbox-info.doctree new file mode 100644 index 0000000000..4cb751ca99 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox11/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox12/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox12/configurations.doctree new file mode 100644 index 0000000000..5752dee1bd Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox12/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox12/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox12/creating-services.doctree new file mode 100644 index 0000000000..2845b5c948 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox12/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox12/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox12/index.doctree new file mode 100644 index 0000000000..4c3f4ad04b Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox12/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree new file mode 100644 index 0000000000..f2d721dbf2 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox12/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox12/sandbox-info.doctree new file mode 100644 index 0000000000..86f000987c Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox12/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox13/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox13/configurations.doctree new file mode 100644 index 0000000000..057091cb43 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox13/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox13/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox13/creating-services.doctree new file mode 100644 index 0000000000..ac31c64553 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox13/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox13/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox13/index.doctree new file mode 100644 index 0000000000..f5affbdf90 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox13/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree new file mode 100644 index 0000000000..75c07e0b76 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox13/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox13/sandbox-info.doctree new file mode 100644 index 0000000000..6ef194a3d9 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox13/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox14/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox14/configurations.doctree new file mode 100644 index 0000000000..43ce2e52d4 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox14/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox14/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox14/creating-services.doctree new file mode 100644 index 0000000000..6d07054554 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox14/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox14/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox14/index.doctree new file mode 100644 index 0000000000..f2897077c7 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox14/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree new file mode 100644 index 0000000000..b4dbf5cf8c Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox14/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox14/sandbox-info.doctree new file mode 100644 index 0000000000..9a64c386e1 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox14/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox15/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox15/configurations.doctree new file mode 100644 index 0000000000..8760729500 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox15/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox15/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox15/creating-services.doctree new file mode 100644 index 0000000000..2065b4fc6a Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox15/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox15/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox15/index.doctree new file mode 100644 index 0000000000..edb1910acf Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox15/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree new file mode 100644 index 0000000000..1ecea3cb05 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox15/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox15/sandbox-info.doctree new file mode 100644 index 0000000000..2692d58aa0 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox15/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox2/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox2/configurations.doctree new file mode 100644 index 0000000000..e549f5d607 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox2/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox2/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox2/creating-services.doctree new file mode 100644 index 0000000000..f549b55622 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox2/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox2/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox2/index.doctree new file mode 100644 index 0000000000..838c291c04 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox2/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree new file mode 100644 index 0000000000..e3cdab697f Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox2/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox2/sandbox-info.doctree new file mode 100644 index 0000000000..86686a9139 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox2/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox3/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox3/configurations.doctree new file mode 100644 index 0000000000..dc75c2e019 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox3/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox3/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox3/creating-services.doctree new file mode 100644 index 0000000000..aa22e3a86d Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox3/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox3/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox3/index.doctree new file mode 100644 index 0000000000..aadce7ac4d Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox3/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree new file mode 100644 index 0000000000..636ec56cbf Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox3/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox3/sandbox-info.doctree new file mode 100644 index 0000000000..0cd51dc327 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox3/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox4/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox4/configurations.doctree new file mode 100644 index 0000000000..5f4da37cc9 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox4/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox4/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox4/creating-services.doctree new file mode 100644 index 0000000000..daeb40c835 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox4/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox4/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox4/index.doctree new file mode 100644 index 0000000000..14c3139c1e Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox4/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree new file mode 100644 index 0000000000..e6856850e3 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox4/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox4/sandbox-info.doctree new file mode 100644 index 0000000000..049ef1e7e5 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox4/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox5/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox5/configurations.doctree new file mode 100644 index 0000000000..74c74c3651 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox5/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox5/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox5/creating-services.doctree new file mode 100644 index 0000000000..eb7b2aaada Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox5/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox5/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox5/index.doctree new file mode 100644 index 0000000000..77802000ae Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox5/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree new file mode 100644 index 0000000000..1f6865a416 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox5/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox5/sandbox-info.doctree new file mode 100644 index 0000000000..61875274cb Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox5/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox6/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox6/configurations.doctree new file mode 100644 index 0000000000..62cbf799cb Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox6/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox6/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox6/creating-services.doctree new file mode 100644 index 0000000000..240cf70be4 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox6/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox6/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox6/index.doctree new file mode 100644 index 0000000000..06406a4b94 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox6/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree new file mode 100644 index 0000000000..bd0635ae83 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox6/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox6/sandbox-info.doctree new file mode 100644 index 0000000000..fa95e871ce Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox6/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox7/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox7/configurations.doctree new file mode 100644 index 0000000000..780c6399d5 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox7/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox7/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox7/creating-services.doctree new file mode 100644 index 0000000000..9c3136c573 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox7/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox7/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox7/index.doctree new file mode 100644 index 0000000000..ea0db683d0 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox7/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree new file mode 100644 index 0000000000..ed1b51e1c2 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox7/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox7/sandbox-info.doctree new file mode 100644 index 0000000000..bc2da9ffe4 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox7/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox8/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox8/configurations.doctree new file mode 100644 index 0000000000..c5d3e1b209 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox8/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox8/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox8/creating-services.doctree new file mode 100644 index 0000000000..945c1f175d Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox8/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox8/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox8/index.doctree new file mode 100644 index 0000000000..236cb72b58 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox8/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree new file mode 100644 index 0000000000..fc7a23449c Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox8/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox8/sandbox-info.doctree new file mode 100644 index 0000000000..b97b694d46 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox8/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox9/configurations.doctree b/en/3.0/.doctrees/sandbox/Sandbox9/configurations.doctree new file mode 100644 index 0000000000..874020cd18 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox9/configurations.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox9/creating-services.doctree b/en/3.0/.doctrees/sandbox/Sandbox9/creating-services.doctree new file mode 100644 index 0000000000..097441e456 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox9/creating-services.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox9/index.doctree b/en/3.0/.doctrees/sandbox/Sandbox9/index.doctree new file mode 100644 index 0000000000..36793d6a86 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox9/index.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree b/en/3.0/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree new file mode 100644 index 0000000000..a1e4cebdd2 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree differ diff --git a/en/3.0/.doctrees/sandbox/Sandbox9/sandbox-info.doctree b/en/3.0/.doctrees/sandbox/Sandbox9/sandbox-info.doctree new file mode 100644 index 0000000000..904dcb7852 Binary files /dev/null and b/en/3.0/.doctrees/sandbox/Sandbox9/sandbox-info.doctree differ diff --git a/en/3.0/.doctrees/softgate-performance.doctree b/en/3.0/.doctrees/softgate-performance.doctree new file mode 100644 index 0000000000..eebf6702f3 Binary files /dev/null and b/en/3.0/.doctrees/softgate-performance.doctree differ diff --git a/en/3.0/.doctrees/supported-networks.doctree b/en/3.0/.doctrees/supported-networks.doctree new file mode 100644 index 0000000000..e148476c05 Binary files /dev/null and b/en/3.0/.doctrees/supported-networks.doctree differ diff --git a/en/3.0/.doctrees/switch-agent-installation.doctree b/en/3.0/.doctrees/switch-agent-installation.doctree new file mode 100644 index 0000000000..41f3197232 Binary files /dev/null and b/en/3.0/.doctrees/switch-agent-installation.doctree differ diff --git a/en/3.0/.doctrees/switch-ports.doctree b/en/3.0/.doctrees/switch-ports.doctree new file mode 100644 index 0000000000..ec514824bf Binary files /dev/null and b/en/3.0/.doctrees/switch-ports.doctree differ diff --git a/en/3.0/.doctrees/terraform-integration.doctree b/en/3.0/.doctrees/terraform-integration.doctree new file mode 100644 index 0000000000..39356acfd6 Binary files /dev/null and b/en/3.0/.doctrees/terraform-integration.doctree differ diff --git a/en/3.0/.doctrees/topology-management.doctree b/en/3.0/.doctrees/topology-management.doctree new file mode 100644 index 0000000000..ea92d52cd6 Binary files /dev/null and b/en/3.0/.doctrees/topology-management.doctree differ diff --git a/en/3.0/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree b/en/3.0/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree new file mode 100644 index 0000000000..7846919578 Binary files /dev/null and b/en/3.0/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree differ diff --git a/en/3.0/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree b/en/3.0/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree new file mode 100644 index 0000000000..fbf4ca4132 Binary files /dev/null and b/en/3.0/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree differ diff --git a/en/3.0/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree b/en/3.0/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree new file mode 100644 index 0000000000..835ae83195 Binary files /dev/null and b/en/3.0/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree differ diff --git a/en/3.0/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree b/en/3.0/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree new file mode 100644 index 0000000000..d6e0d8fa61 Binary files /dev/null and b/en/3.0/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree differ diff --git a/en/3.0/.doctrees/tutorials/getting-started-for-equinix-metal.doctree b/en/3.0/.doctrees/tutorials/getting-started-for-equinix-metal.doctree new file mode 100644 index 0000000000..9ba3cb5bbc Binary files /dev/null and b/en/3.0/.doctrees/tutorials/getting-started-for-equinix-metal.doctree differ diff --git a/en/3.0/.doctrees/tutorials/index.doctree b/en/3.0/.doctrees/tutorials/index.doctree new file mode 100644 index 0000000000..0232d9a529 Binary files /dev/null and b/en/3.0/.doctrees/tutorials/index.doctree differ diff --git a/en/3.0/.doctrees/tutorials/installing-netris-controller.doctree b/en/3.0/.doctrees/tutorials/installing-netris-controller.doctree new file mode 100644 index 0000000000..a64c7bc84e Binary files /dev/null and b/en/3.0/.doctrees/tutorials/installing-netris-controller.doctree differ diff --git a/en/3.0/.doctrees/tutorials/upgrading-netris.doctree b/en/3.0/.doctrees/tutorials/upgrading-netris.doctree new file mode 100644 index 0000000000..7d01c3d89c Binary files /dev/null and b/en/3.0/.doctrees/tutorials/upgrading-netris.doctree differ diff --git a/en/3.0/.doctrees/tutorials/using-l4-load-balancer.doctree b/en/3.0/.doctrees/tutorials/using-l4-load-balancer.doctree new file mode 100644 index 0000000000..06fb4c0b67 Binary files /dev/null and b/en/3.0/.doctrees/tutorials/using-l4-load-balancer.doctree differ diff --git a/en/3.0/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree b/en/3.0/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree new file mode 100644 index 0000000000..76fc0b1be9 Binary files /dev/null and b/en/3.0/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree differ diff --git a/en/3.0/.doctrees/visibility.doctree b/en/3.0/.doctrees/visibility.doctree new file mode 100644 index 0000000000..194d9acf1e Binary files /dev/null and b/en/3.0/.doctrees/visibility.doctree differ diff --git a/en/3.0/.doctrees/vnet.doctree b/en/3.0/.doctrees/vnet.doctree new file mode 100644 index 0000000000..729d5c3d0f Binary files /dev/null and b/en/3.0/.doctrees/vnet.doctree differ diff --git a/en/3.0/.nojekyll b/en/3.0/.nojekyll new file mode 100644 index 0000000000..e69de29bb2 diff --git a/en/3.0/SoftGate-PRO-installation.html b/en/3.0/SoftGate-PRO-installation.html new file mode 100644 index 0000000000..bedbc1ba3e --- /dev/null +++ b/en/3.0/SoftGate-PRO-installation.html @@ -0,0 +1,616 @@ + + + + + + + + + + + SoftGate PRO Installation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • SoftGate PRO Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

SoftGate PRO Installation

+
+

Minimum Hardware Requirements

+
    +
  • 2 x Intel® Xeon® Silver Processor with 10 physical cores per socket (20 cores total)

  • +
  • 128 GB (64 GB RAM per socket) in multichannel configuration

  • +
  • 300 GB HDD

  • +
  • Nvidia Mellanox Connect-X 5/6 SmartNIC card

  • +
+
+
+

BIOS Configuration

+

The following are some recommendations for BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference:

+
    +
  • Before starting consider resetting all BIOS settings to their defaults

  • +
  • Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report

  • +
  • Select Performance as the CPU Power and Performance policy

  • +
  • Enable Turbo Boost

  • +
  • Set memory frequency to the highest available number, NOT auto

  • +
  • Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d

  • +
  • Disable Hyper-Threading

  • +
+
+
+

Install the Netris Agent

+

Requires freshly installed Ubuntu Linux 18.04 LTS and internet connectivity configured from netplan via management port.

+
    +
  1. Add the SoftGate in the controller Inventory or Topology section. Detailed configuration documentation is available here: “Adding SoftGates”.

  2. +
  3. Once the SoftGate is created, navigate to the Inventory section, click the three vertical dots (⋮) on the right side of the newly created SoftGate and select the Install Agent option.

  4. +
  5. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user.

  6. +
  7. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration).

  8. +
+
+

Note

+

If the Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured.

+
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    up ip route add <Controller address> via <Management network gateway> # Please delete this line if Netris Controller is located in the same network with the SoftGate node.
+    gateway <Gateway IP address>
+
+ source /etc/network/interfaces.d/*
+
+
+
    +
  1. If everything seems ok, please remove/comment the Gateway line and save the file.

  2. +
+
+

Note

+

Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent.

+
+
    +
  1. Reboot the SoftGate

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/SoftGate-installation.html b/en/3.0/SoftGate-installation.html new file mode 100644 index 0000000000..a99c1945a0 --- /dev/null +++ b/en/3.0/SoftGate-installation.html @@ -0,0 +1,623 @@ + + + + + + + + + + + SoftGate Installation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • SoftGate Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

SoftGate Installation

+
+

Minimum Hardware Requirements

+
    +
  • 8 physical CPU cores

  • +
  • 16 GB RAM

  • +
  • 300 GB HDD

  • +
+
+
+

Install the Netris Agent

+

Requires freshly installed Ubuntu Linux 22.04 LTS and internet connectivity configured from netplan via management port.

+
    +
  1. Add the SoftGate in the controller Inventory or Topology section. Detailed configuration documentation is available here: “Adding SoftGates”.

  2. +
  3. Once the SoftGate is created, navigate to the Inventory section, click the three vertical dots (⋮) on the right side of the newly created SoftGate and select the Install Agent option.

  4. +
  5. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user.

  6. +
  7. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration).

  8. +
+
+

Note

+

If the Netris Controller is not in the same OOB network then it is required to add a route to Netris Controller. No default route or other IP addresses should be configured.

+
+
+

Note

+

For proper operation of SoftGate, it is required to configure a bond interface. Please have a look at the example configuration below.

+
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    up ip route add <Controller address> via <Management network gateway> # Pleasedelete this line if Netris Controller is located in the same network with the SoftGate node.
+    gateway <Gateway IP address>
+
+# First Softgate Interface
+auto ensX # Please replace the ensX with the actual interface name present in the OS.
+iface ensX inet static # Please replace the ensX with the actual interface name present in the OS.
+    address 0.0.0.0/0
+
+# Second Softgate Interface # This interface is optional
+auto ensY # Please replace the ensY with the actual interface name present in the OS.
+iface ensY inet static # Please replace the ensY with the actual interface name present in the OS.
+    address 0.0.0.0/0
+
+# Bond interface
+auto bond0
+iface bond0 inet static
+    address 0.0.0.0/0
+    bond-slaves ensX ensY # Please replace the ensX/Y with actual interface names present in the OS.
+    bond-mode active-backup # Optional, please adjust the bonding mode according to the desired functionality.
+
+source /etc/network/interfaces.d/*
+
+
+
    +
  1. If everything seems ok, please remove/comment the Gateway line and save the file.

  2. +
+
+

Note

+

Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent.

+
+
    +
  1. Reboot the SoftGate

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up, you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/_images/ACL_active.png b/en/3.0/_images/ACL_active.png new file mode 100644 index 0000000000..6ba98a50b8 Binary files /dev/null and b/en/3.0/_images/ACL_active.png differ diff --git a/en/3.0/_images/ACL_approval.png b/en/3.0/_images/ACL_approval.png new file mode 100644 index 0000000000..6345dea408 Binary files /dev/null and b/en/3.0/_images/ACL_approval.png differ diff --git a/en/3.0/_images/ACL_rule.png b/en/3.0/_images/ACL_rule.png new file mode 100644 index 0000000000..81c08c11db Binary files /dev/null and b/en/3.0/_images/ACL_rule.png differ diff --git a/en/3.0/_images/BGP_neighbors_listing.png b/en/3.0/_images/BGP_neighbors_listing.png new file mode 100644 index 0000000000..de768abbc4 Binary files /dev/null and b/en/3.0/_images/BGP_neighbors_listing.png differ diff --git a/en/3.0/_images/BGP_route.png b/en/3.0/_images/BGP_route.png new file mode 100644 index 0000000000..b2e702fc7e Binary files /dev/null and b/en/3.0/_images/BGP_route.png differ diff --git a/en/3.0/_images/EVPN_routing.png b/en/3.0/_images/EVPN_routing.png new file mode 100644 index 0000000000..c9196538a8 Binary files /dev/null and b/en/3.0/_images/EVPN_routing.png differ diff --git a/en/3.0/_images/IP-allocation.png b/en/3.0/_images/IP-allocation.png new file mode 100644 index 0000000000..a59909eb2c Binary files /dev/null and b/en/3.0/_images/IP-allocation.png differ diff --git a/en/3.0/_images/IPv4-Prefix.png b/en/3.0/_images/IPv4-Prefix.png new file mode 100644 index 0000000000..4726a74662 Binary files /dev/null and b/en/3.0/_images/IPv4-Prefix.png differ diff --git a/en/3.0/_images/IPv6-Prefix.png b/en/3.0/_images/IPv6-Prefix.png new file mode 100644 index 0000000000..c4ab1a9fb0 Binary files /dev/null and b/en/3.0/_images/IPv6-Prefix.png differ diff --git a/en/3.0/_images/ISP_Iris.png b/en/3.0/_images/ISP_Iris.png new file mode 100644 index 0000000000..638bc3a1df Binary files /dev/null and b/en/3.0/_images/ISP_Iris.png differ diff --git a/en/3.0/_images/MAC_listing.png b/en/3.0/_images/MAC_listing.png new file mode 100644 index 0000000000..abe057a8ac Binary files /dev/null and b/en/3.0/_images/MAC_listing.png differ diff --git a/en/3.0/_images/NATIP-address.png b/en/3.0/_images/NATIP-address.png new file mode 100644 index 0000000000..f6a12b62f9 Binary files /dev/null and b/en/3.0/_images/NATIP-address.png differ diff --git a/en/3.0/_images/Port-Forwarding.png b/en/3.0/_images/Port-Forwarding.png new file mode 100644 index 0000000000..fea266b6b3 Binary files /dev/null and b/en/3.0/_images/Port-Forwarding.png differ diff --git a/en/3.0/_images/SiteMesh_listing.png b/en/3.0/_images/SiteMesh_listing.png new file mode 100644 index 0000000000..7142f87e52 Binary files /dev/null and b/en/3.0/_images/SiteMesh_listing.png differ diff --git a/en/3.0/_images/SiteMesh_modes.png b/en/3.0/_images/SiteMesh_modes.png new file mode 100644 index 0000000000..5edfe756f0 Binary files /dev/null and b/en/3.0/_images/SiteMesh_modes.png differ diff --git a/en/3.0/_images/Site_Mesh.png b/en/3.0/_images/Site_Mesh.png new file mode 100644 index 0000000000..f9fb780b4d Binary files /dev/null and b/en/3.0/_images/Site_Mesh.png differ diff --git a/en/3.0/_images/Spine1.png b/en/3.0/_images/Spine1.png new file mode 100644 index 0000000000..9871609ae1 Binary files /dev/null and b/en/3.0/_images/Spine1.png differ diff --git a/en/3.0/_images/TCAM.png b/en/3.0/_images/TCAM.png new file mode 100644 index 0000000000..0bba84f117 Binary files /dev/null and b/en/3.0/_images/TCAM.png differ diff --git a/en/3.0/_images/V_NET.png b/en/3.0/_images/V_NET.png new file mode 100644 index 0000000000..74733e071a Binary files /dev/null and b/en/3.0/_images/V_NET.png differ diff --git a/en/3.0/_images/action_permit.png b/en/3.0/_images/action_permit.png new file mode 100644 index 0000000000..c478300133 Binary files /dev/null and b/en/3.0/_images/action_permit.png differ diff --git a/en/3.0/_images/add-allocation.png b/en/3.0/_images/add-allocation.png new file mode 100644 index 0000000000..0d4784990e Binary files /dev/null and b/en/3.0/_images/add-allocation.png differ diff --git a/en/3.0/_images/add-bgp-basic.png b/en/3.0/_images/add-bgp-basic.png new file mode 100644 index 0000000000..be55737f4a Binary files /dev/null and b/en/3.0/_images/add-bgp-basic.png differ diff --git a/en/3.0/_images/add-new-hardware.png b/en/3.0/_images/add-new-hardware.png new file mode 100644 index 0000000000..5e00df64a3 Binary files /dev/null and b/en/3.0/_images/add-new-hardware.png differ diff --git a/en/3.0/_images/add-softgate.png b/en/3.0/_images/add-softgate.png new file mode 100644 index 0000000000..0ea2b859ac Binary files /dev/null and b/en/3.0/_images/add-softgate.png differ diff --git a/en/3.0/_images/add-to-lag-port.png b/en/3.0/_images/add-to-lag-port.png new file mode 100644 index 0000000000..479e36897e Binary files /dev/null and b/en/3.0/_images/add-to-lag-port.png differ diff --git a/en/3.0/_images/add-vnet.png b/en/3.0/_images/add-vnet.png new file mode 100644 index 0000000000..4321573d7c Binary files /dev/null and b/en/3.0/_images/add-vnet.png differ diff --git a/en/3.0/_images/approve_reject.png b/en/3.0/_images/approve_reject.png new file mode 100644 index 0000000000..a3054f9912 Binary files /dev/null and b/en/3.0/_images/approve_reject.png differ diff --git a/en/3.0/_images/cloudflare-dns-record.png b/en/3.0/_images/cloudflare-dns-record.png new file mode 100644 index 0000000000..a0d7586c2b Binary files /dev/null and b/en/3.0/_images/cloudflare-dns-record.png differ diff --git a/en/3.0/_images/cloudflare-dns-record1.png b/en/3.0/_images/cloudflare-dns-record1.png new file mode 100644 index 0000000000..a0d7586c2b Binary files /dev/null and b/en/3.0/_images/cloudflare-dns-record1.png differ diff --git a/en/3.0/_images/community.png b/en/3.0/_images/community.png new file mode 100644 index 0000000000..53fd553d3f Binary files /dev/null and b/en/3.0/_images/community.png differ diff --git a/en/3.0/_images/create_link.png b/en/3.0/_images/create_link.png new file mode 100644 index 0000000000..572869cb6f Binary files /dev/null and b/en/3.0/_images/create_link.png differ diff --git a/en/3.0/_images/createboard.png b/en/3.0/_images/createboard.png new file mode 100644 index 0000000000..8040723586 Binary files /dev/null and b/en/3.0/_images/createboard.png differ diff --git a/en/3.0/_images/credentials.png b/en/3.0/_images/credentials.png new file mode 100644 index 0000000000..edbc78148f Binary files /dev/null and b/en/3.0/_images/credentials.png differ diff --git a/en/3.0/_images/defaultroute.png b/en/3.0/_images/defaultroute.png new file mode 100644 index 0000000000..1a4d407173 Binary files /dev/null and b/en/3.0/_images/defaultroute.png differ diff --git a/en/3.0/_images/diagrams_terraform.png b/en/3.0/_images/diagrams_terraform.png new file mode 100644 index 0000000000..33f33920f0 Binary files /dev/null and b/en/3.0/_images/diagrams_terraform.png differ diff --git a/en/3.0/_images/edit-port.png b/en/3.0/_images/edit-port.png new file mode 100644 index 0000000000..eed82b6388 Binary files /dev/null and b/en/3.0/_images/edit-port.png differ diff --git a/en/3.0/_images/edit_switch_port.png b/en/3.0/_images/edit_switch_port.png new file mode 100644 index 0000000000..711980ab9e Binary files /dev/null and b/en/3.0/_images/edit_switch_port.png differ diff --git a/en/3.0/_images/equinix-metal-activate-bgp.png b/en/3.0/_images/equinix-metal-activate-bgp.png new file mode 100644 index 0000000000..b03c10dd20 Binary files /dev/null and b/en/3.0/_images/equinix-metal-activate-bgp.png differ diff --git a/en/3.0/_images/equinix-metal-bgp-diagram.png b/en/3.0/_images/equinix-metal-bgp-diagram.png new file mode 100644 index 0000000000..50080c16ad Binary files /dev/null and b/en/3.0/_images/equinix-metal-bgp-diagram.png differ diff --git a/en/3.0/_images/equinix-metal-netris-bgp-up.png b/en/3.0/_images/equinix-metal-netris-bgp-up.png new file mode 100644 index 0000000000..8c993d3a97 Binary files /dev/null and b/en/3.0/_images/equinix-metal-netris-bgp-up.png differ diff --git a/en/3.0/_images/equinix-metal-netris-ipam-synced.png b/en/3.0/_images/equinix-metal-netris-ipam-synced.png new file mode 100644 index 0000000000..7095db4a4c Binary files /dev/null and b/en/3.0/_images/equinix-metal-netris-ipam-synced.png differ diff --git a/en/3.0/_images/equinix-metal-project-api-keys.png b/en/3.0/_images/equinix-metal-project-api-keys.png new file mode 100644 index 0000000000..1988e1e503 Binary files /dev/null and b/en/3.0/_images/equinix-metal-project-api-keys.png differ diff --git a/en/3.0/_images/equinix-metal-project-id.png b/en/3.0/_images/equinix-metal-project-id.png new file mode 100644 index 0000000000..de83c5d6c0 Binary files /dev/null and b/en/3.0/_images/equinix-metal-project-id.png differ diff --git a/en/3.0/_images/equinix-metal-request-ip-block.png b/en/3.0/_images/equinix-metal-request-ip-block.png new file mode 100644 index 0000000000..9a39600bac Binary files /dev/null and b/en/3.0/_images/equinix-metal-request-ip-block.png differ diff --git a/en/3.0/_images/globalIP.png b/en/3.0/_images/globalIP.png new file mode 100644 index 0000000000..bae8a860ce Binary files /dev/null and b/en/3.0/_images/globalIP.png differ diff --git a/en/3.0/_images/graphboard.png b/en/3.0/_images/graphboard.png new file mode 100644 index 0000000000..b777f11b69 Binary files /dev/null and b/en/3.0/_images/graphboard.png differ diff --git a/en/3.0/_images/hairpin.png b/en/3.0/_images/hairpin.png new file mode 100644 index 0000000000..8bfa508087 Binary files /dev/null and b/en/3.0/_images/hairpin.png differ diff --git a/en/3.0/_images/hairpin_topology.png b/en/3.0/_images/hairpin_topology.png new file mode 100644 index 0000000000..95b3c4fe5d Binary files /dev/null and b/en/3.0/_images/hairpin_topology.png differ diff --git a/en/3.0/_images/hardware_health.png b/en/3.0/_images/hardware_health.png new file mode 100644 index 0000000000..df70fc2ecc Binary files /dev/null and b/en/3.0/_images/hardware_health.png differ diff --git a/en/3.0/_images/installOS.png b/en/3.0/_images/installOS.png new file mode 100644 index 0000000000..cd36377e22 Binary files /dev/null and b/en/3.0/_images/installOS.png differ diff --git a/en/3.0/_images/install_agent.gif b/en/3.0/_images/install_agent.gif new file mode 100644 index 0000000000..0928829c40 Binary files /dev/null and b/en/3.0/_images/install_agent.gif differ diff --git a/en/3.0/_images/inventory-listing.png b/en/3.0/_images/inventory-listing.png new file mode 100644 index 0000000000..caa35c8f66 Binary files /dev/null and b/en/3.0/_images/inventory-listing.png differ diff --git a/en/3.0/_images/inventory-profile.png b/en/3.0/_images/inventory-profile.png new file mode 100644 index 0000000000..e58ef725b7 Binary files /dev/null and b/en/3.0/_images/inventory-profile.png differ diff --git a/en/3.0/_images/inventory_heartbeat.png b/en/3.0/_images/inventory_heartbeat.png new file mode 100644 index 0000000000..8e884600a3 Binary files /dev/null and b/en/3.0/_images/inventory_heartbeat.png differ diff --git a/en/3.0/_images/l3lb_srv01.png b/en/3.0/_images/l3lb_srv01.png new file mode 100644 index 0000000000..f2fe0bf3bb Binary files /dev/null and b/en/3.0/_images/l3lb_srv01.png differ diff --git a/en/3.0/_images/l3lb_srv02.png b/en/3.0/_images/l3lb_srv02.png new file mode 100644 index 0000000000..6d3a5bd423 Binary files /dev/null and b/en/3.0/_images/l3lb_srv02.png differ diff --git a/en/3.0/_images/leaf1_spine1.png b/en/3.0/_images/leaf1_spine1.png new file mode 100644 index 0000000000..8b9b4a3ff0 Binary files /dev/null and b/en/3.0/_images/leaf1_spine1.png differ diff --git a/en/3.0/_images/list-subnets.png b/en/3.0/_images/list-subnets.png new file mode 100644 index 0000000000..a3e9d60c73 Binary files /dev/null and b/en/3.0/_images/list-subnets.png differ diff --git a/en/3.0/_images/list-vnet-expanded.png b/en/3.0/_images/list-vnet-expanded.png new file mode 100644 index 0000000000..fdc5f16998 Binary files /dev/null and b/en/3.0/_images/list-vnet-expanded.png differ diff --git a/en/3.0/_images/list-vnet.png b/en/3.0/_images/list-vnet.png new file mode 100644 index 0000000000..2ad4b6fa49 Binary files /dev/null and b/en/3.0/_images/list-vnet.png differ diff --git a/en/3.0/_images/netris-architecture.png b/en/3.0/_images/netris-architecture.png new file mode 100644 index 0000000000..8c36dc9ac8 Binary files /dev/null and b/en/3.0/_images/netris-architecture.png differ diff --git a/en/3.0/_images/netris-create-common-subnets.png b/en/3.0/_images/netris-create-common-subnets.png new file mode 100644 index 0000000000..b47c8e952c Binary files /dev/null and b/en/3.0/_images/netris-create-common-subnets.png differ diff --git a/en/3.0/_images/netris-create-equinix-metal-site.png b/en/3.0/_images/netris-create-equinix-metal-site.png new file mode 100644 index 0000000000..39734075f1 Binary files /dev/null and b/en/3.0/_images/netris-create-equinix-metal-site.png differ diff --git a/en/3.0/_images/netris-create-nat-rule.png b/en/3.0/_images/netris-create-nat-rule.png new file mode 100644 index 0000000000..28c5913965 Binary files /dev/null and b/en/3.0/_images/netris-create-nat-rule.png differ diff --git a/en/3.0/_images/netris-creating-vnet-for-equinix-metal.png b/en/3.0/_images/netris-creating-vnet-for-equinix-metal.png new file mode 100644 index 0000000000..98fdbd910d Binary files /dev/null and b/en/3.0/_images/netris-creating-vnet-for-equinix-metal.png differ diff --git a/en/3.0/_images/netris-enable-elb.png b/en/3.0/_images/netris-enable-elb.png new file mode 100644 index 0000000000..36e7566443 Binary files /dev/null and b/en/3.0/_images/netris-enable-elb.png differ diff --git a/en/3.0/_images/netris-ipam-nat.png b/en/3.0/_images/netris-ipam-nat.png new file mode 100644 index 0000000000..d7ff2449c7 Binary files /dev/null and b/en/3.0/_images/netris-ipam-nat.png differ diff --git a/en/3.0/_images/netris-l4-load-balancer.png b/en/3.0/_images/netris-l4-load-balancer.png new file mode 100644 index 0000000000..e6e466f622 Binary files /dev/null and b/en/3.0/_images/netris-l4-load-balancer.png differ diff --git a/en/3.0/_images/netris-vnet-ready-in-equinix-metal.png b/en/3.0/_images/netris-vnet-ready-in-equinix-metal.png new file mode 100644 index 0000000000..cf7274bb86 Binary files /dev/null and b/en/3.0/_images/netris-vnet-ready-in-equinix-metal.png differ diff --git a/en/3.0/_images/netris_controller_diagram.png b/en/3.0/_images/netris_controller_diagram.png new file mode 100644 index 0000000000..610a8ae396 Binary files /dev/null and b/en/3.0/_images/netris_controller_diagram.png differ diff --git a/en/3.0/_images/netris_version_example.png b/en/3.0/_images/netris_version_example.png new file mode 100644 index 0000000000..55e23d8df9 Binary files /dev/null and b/en/3.0/_images/netris_version_example.png differ diff --git a/en/3.0/_images/password.png b/en/3.0/_images/password.png new file mode 100644 index 0000000000..a27b7725a9 Binary files /dev/null and b/en/3.0/_images/password.png differ diff --git a/en/3.0/_images/permission_group.png b/en/3.0/_images/permission_group.png new file mode 100644 index 0000000000..e4d1ad6ab0 Binary files /dev/null and b/en/3.0/_images/permission_group.png differ diff --git a/en/3.0/_images/ping.png b/en/3.0/_images/ping.png new file mode 100644 index 0000000000..c8e9e64e47 Binary files /dev/null and b/en/3.0/_images/ping.png differ diff --git a/en/3.0/_images/route-map.png b/en/3.0/_images/route-map.png new file mode 100644 index 0000000000..692cfd63f3 Binary files /dev/null and b/en/3.0/_images/route-map.png differ diff --git a/en/3.0/_images/sandbox-l4lb-kubeapi.png b/en/3.0/_images/sandbox-l4lb-kubeapi.png new file mode 100644 index 0000000000..6a592476df Binary files /dev/null and b/en/3.0/_images/sandbox-l4lb-kubeapi.png differ diff --git a/en/3.0/_images/sandbox-l4lbs.png b/en/3.0/_images/sandbox-l4lbs.png new file mode 100644 index 0000000000..c002c0fed7 Binary files /dev/null and b/en/3.0/_images/sandbox-l4lbs.png differ diff --git a/en/3.0/_images/sandbox-podinfo-prov.png b/en/3.0/_images/sandbox-podinfo-prov.png new file mode 100644 index 0000000000..079871e59a Binary files /dev/null and b/en/3.0/_images/sandbox-podinfo-prov.png differ diff --git a/en/3.0/_images/sandbox-podinfo-ready.png b/en/3.0/_images/sandbox-podinfo-ready.png new file mode 100644 index 0000000000..d6919ecf9d Binary files /dev/null and b/en/3.0/_images/sandbox-podinfo-ready.png differ diff --git a/en/3.0/_images/sandbox3-l4lb-kubeapi.png b/en/3.0/_images/sandbox3-l4lb-kubeapi.png new file mode 100644 index 0000000000..9faa757631 Binary files /dev/null and b/en/3.0/_images/sandbox3-l4lb-kubeapi.png differ diff --git a/en/3.0/_images/sandbox3-l4lbs.png b/en/3.0/_images/sandbox3-l4lbs.png new file mode 100644 index 0000000000..629b642e88 Binary files /dev/null and b/en/3.0/_images/sandbox3-l4lbs.png differ diff --git a/en/3.0/_images/sandbox3-podinfo-prov.png b/en/3.0/_images/sandbox3-podinfo-prov.png new file mode 100644 index 0000000000..5dec03019c Binary files /dev/null and b/en/3.0/_images/sandbox3-podinfo-prov.png differ diff --git a/en/3.0/_images/sandbox3-podinfo-ready.png b/en/3.0/_images/sandbox3-podinfo-ready.png new file mode 100644 index 0000000000..2bfb5864bb Binary files /dev/null and b/en/3.0/_images/sandbox3-podinfo-ready.png differ diff --git a/en/3.0/_images/sandbox_topology.png b/en/3.0/_images/sandbox_topology.png new file mode 100644 index 0000000000..ccb044f313 Binary files /dev/null and b/en/3.0/_images/sandbox_topology.png differ diff --git a/en/3.0/_images/sandbox_topology_n.png b/en/3.0/_images/sandbox_topology_n.png new file mode 100644 index 0000000000..2743b42a5c Binary files /dev/null and b/en/3.0/_images/sandbox_topology_n.png differ diff --git a/en/3.0/_images/saveasnormal.png b/en/3.0/_images/saveasnormal.png new file mode 100644 index 0000000000..6f5cd5bda3 Binary files /dev/null and b/en/3.0/_images/saveasnormal.png differ diff --git a/en/3.0/_images/siteDefault.png b/en/3.0/_images/siteDefault.png new file mode 100644 index 0000000000..f8b1cf2688 Binary files /dev/null and b/en/3.0/_images/siteDefault.png differ diff --git a/en/3.0/_images/slide-1.png b/en/3.0/_images/slide-1.png new file mode 100644 index 0000000000..b9c94ee198 Binary files /dev/null and b/en/3.0/_images/slide-1.png differ diff --git a/en/3.0/_images/slide-2.png b/en/3.0/_images/slide-2.png new file mode 100644 index 0000000000..99b0ae67e9 Binary files /dev/null and b/en/3.0/_images/slide-2.png differ diff --git a/en/3.0/_images/slide-3.png b/en/3.0/_images/slide-3.png new file mode 100644 index 0000000000..4f17eafc0e Binary files /dev/null and b/en/3.0/_images/slide-3.png differ diff --git a/en/3.0/_images/slide-4.png b/en/3.0/_images/slide-4.png new file mode 100644 index 0000000000..3324e389da Binary files /dev/null and b/en/3.0/_images/slide-4.png differ diff --git a/en/3.0/_images/softgate-green.png b/en/3.0/_images/softgate-green.png new file mode 100644 index 0000000000..5043c928bd Binary files /dev/null and b/en/3.0/_images/softgate-green.png differ diff --git a/en/3.0/_images/softgate-nodes-created-in-equinix.png b/en/3.0/_images/softgate-nodes-created-in-equinix.png new file mode 100644 index 0000000000..6d0971205d Binary files /dev/null and b/en/3.0/_images/softgate-nodes-created-in-equinix.png differ diff --git a/en/3.0/_images/softgate-nodes-recognized-in-netris.png b/en/3.0/_images/softgate-nodes-recognized-in-netris.png new file mode 100644 index 0000000000..ddf15d2121 Binary files /dev/null and b/en/3.0/_images/softgate-nodes-recognized-in-netris.png differ diff --git a/en/3.0/_images/softgate-one-liner-provisioning.png b/en/3.0/_images/softgate-one-liner-provisioning.png new file mode 100644 index 0000000000..f1c72da232 Binary files /dev/null and b/en/3.0/_images/softgate-one-liner-provisioning.png differ diff --git a/en/3.0/_images/softgate_diagram.png b/en/3.0/_images/softgate_diagram.png new file mode 100644 index 0000000000..a6ade1fd5f Binary files /dev/null and b/en/3.0/_images/softgate_diagram.png differ diff --git a/en/3.0/_images/static_route.png b/en/3.0/_images/static_route.png new file mode 100644 index 0000000000..a022e230d4 Binary files /dev/null and b/en/3.0/_images/static_route.png differ diff --git a/en/3.0/_images/switch_port.png b/en/3.0/_images/switch_port.png new file mode 100644 index 0000000000..8f0e537af5 Binary files /dev/null and b/en/3.0/_images/switch_port.png differ diff --git a/en/3.0/_images/telescope.png b/en/3.0/_images/telescope.png new file mode 100644 index 0000000000..7cfc53fcc7 Binary files /dev/null and b/en/3.0/_images/telescope.png differ diff --git a/en/3.0/_images/tenants.png b/en/3.0/_images/tenants.png new file mode 100644 index 0000000000..eb4172733d Binary files /dev/null and b/en/3.0/_images/tenants.png differ diff --git a/en/3.0/_images/topology_manager.png b/en/3.0/_images/topology_manager.png new file mode 100644 index 0000000000..ce7bb1effe Binary files /dev/null and b/en/3.0/_images/topology_manager.png differ diff --git a/en/3.0/_images/uninstallOS.png b/en/3.0/_images/uninstallOS.png new file mode 100644 index 0000000000..eb2e1965c0 Binary files /dev/null and b/en/3.0/_images/uninstallOS.png differ diff --git a/en/3.0/_images/updateONIE.png b/en/3.0/_images/updateONIE.png new file mode 100644 index 0000000000..093c50c2de Binary files /dev/null and b/en/3.0/_images/updateONIE.png differ diff --git a/en/3.0/_images/user_role.png b/en/3.0/_images/user_role.png new file mode 100644 index 0000000000..d285d9b688 Binary files /dev/null and b/en/3.0/_images/user_role.png differ diff --git a/en/3.0/_images/users.png b/en/3.0/_images/users.png new file mode 100644 index 0000000000..2f90077728 Binary files /dev/null and b/en/3.0/_images/users.png differ diff --git a/en/3.0/_images/waiting_approval.png b/en/3.0/_images/waiting_approval.png new file mode 100644 index 0000000000..ac769acb93 Binary files /dev/null and b/en/3.0/_images/waiting_approval.png differ diff --git a/en/3.0/_sources/SoftGate-PRO-installation.rst.txt b/en/3.0/_sources/SoftGate-PRO-installation.rst.txt new file mode 100644 index 0000000000..09f1133db1 --- /dev/null +++ b/en/3.0/_sources/SoftGate-PRO-installation.rst.txt @@ -0,0 +1,71 @@ +.. meta:: + :description: Netris SoftGate PRO Installation + +*************************** +SoftGate PRO Installation +*************************** + +Minimum Hardware Requirements +============================= +* 2 x Intel® Xeon® Silver Processor with 10 physical cores per socket (20 cores total) +* 128 GB (64 GB RAM per socket) in multichannel configuration +* 300 GB HDD +* Nvidia Mellanox Connect-X 5/6 SmartNIC card + +BIOS Configuration +================== +The following are some recommendations for BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference: + +* Before starting consider resetting all BIOS settings to their defaults +* Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report +* Select Performance as the CPU Power and Performance policy +* Enable Turbo Boost +* Set memory frequency to the highest available number, NOT auto +* Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d +* Disable Hyper-Threading + +Install the Netris Agent +======================== +Requires freshly installed Ubuntu Linux 18.04 LTS and internet connectivity configured from netplan via management port. + +1. Add the SoftGate in the controller **Inventory** or **Topology** section. Detailed configuration documentation is available here: :ref:`"Adding SoftGates"`. +2. Once the SoftGate is created, navigate to the **Inventory** section, click the **three vertical dots (⋮)** on the right side of the newly created SoftGate and select the **Install Agent** option. +3. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user. +4. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration). + +.. note:: + + If the Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured. + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The management network interface + auto ensZ + iface ensZ inet static + address + up ip route add via # Please delete this line if Netris Controller is located in the same network with the SoftGate node. + gateway + + source /etc/network/interfaces.d/* + +5. If everything seems ok, please remove/comment the **Gateway** line and save the file. + +.. note:: + + Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent. + +6. Reboot the SoftGate + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. diff --git a/en/3.0/_sources/SoftGate-installation.rst.txt b/en/3.0/_sources/SoftGate-installation.rst.txt new file mode 100644 index 0000000000..556de8c540 --- /dev/null +++ b/en/3.0/_sources/SoftGate-installation.rst.txt @@ -0,0 +1,79 @@ +.. meta:: + :description: Netris SoftGate Installation + +*************************** +SoftGate Installation +*************************** + +Minimum Hardware Requirements +============================= +* 8 physical CPU cores +* 16 GB RAM +* 300 GB HDD + +Install the Netris Agent +======================== +Requires freshly installed Ubuntu Linux 22.04 LTS and internet connectivity configured from netplan via management port. + +1. Add the SoftGate in the controller **Inventory** or **Topology** section. Detailed configuration documentation is available here: :ref:`"Adding SoftGates"`. +2. Once the SoftGate is created, navigate to the **Inventory** section, click the **three vertical dots (⋮)** on the right side of the newly created SoftGate and select the **Install Agent** option. +3. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user. +4. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration). + +.. note:: + + If the Netris Controller is not in the same OOB network then it is required to add a route to Netris Controller. No default route or other IP addresses should be configured. + +.. note:: + + For proper operation of SoftGate, it is required to configure a bond interface. Please have a look at the example configuration below. + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The management network interface + auto ensZ + iface ensZ inet static + address + up ip route add via # Pleasedelete this line if Netris Controller is located in the same network with the SoftGate node. + gateway + + # First Softgate Interface + auto ensX # Please replace the ensX with the actual interface name present in the OS. + iface ensX inet static # Please replace the ensX with the actual interface name present in the OS. + address 0.0.0.0/0 + + # Second Softgate Interface # This interface is optional + auto ensY # Please replace the ensY with the actual interface name present in the OS. + iface ensY inet static # Please replace the ensY with the actual interface name present in the OS. + address 0.0.0.0/0 + + # Bond interface + auto bond0 + iface bond0 inet static + address 0.0.0.0/0 + bond-slaves ensX ensY # Please replace the ensX/Y with actual interface names present in the OS. + bond-mode active-backup # Optional, please adjust the bonding mode according to the desired functionality. + + source /etc/network/interfaces.d/* + +5. If everything seems ok, please remove/comment the **Gateway** line and save the file. + +.. note:: + + Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent. + +6. Reboot the SoftGate + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up, you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. diff --git a/en/3.0/_sources/accounts.rst.txt b/en/3.0/_sources/accounts.rst.txt new file mode 100644 index 0000000000..92f4ab3caa --- /dev/null +++ b/en/3.0/_sources/accounts.rst.txt @@ -0,0 +1,74 @@ +.. meta:: + :description: Netris Controller User Account Management + +######## +Accounts +######## + +The accounts section is for the management of user accounts, access permissions, and tenants. + +Users +===== +Description of User account fields: + +* **Username** - Unique username. +* **Full Name** - Full Name of the user. +* **E-mail** - The email address of the user. Also used for system notifications and for password retrieval. +* **E-mail CC** - Send copies of email notifications to this address. +* **Phone Number** - User’s phone number. +* **Company** - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers. +* **Position** - Position within the company. +* **User Role** - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate. +* **Permission Group** - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used) +* **+Tenant** - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used) + +Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin. + +.. image:: images/users.png + :align: center + :class: with-shadow + :alt: User Management + +**Password**: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side. + +Example: Listing of user accounts. + + +.. image:: images/password.png + :align: center + :class: with-shadow + :alt: List User Accounts + +Tenants +======= +IP addresses and Switch Ports are network resources that can be assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs) as part of DevOps/NetOps pipeline. + +A Tenant has just two fields, the unique name and custom description. + +Example: Adding a tenant. + +.. image:: images/tenants.png + :align: center + :class: with-shadow + :alt: Adding Tenants + +Permission Groups +================= +Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections. + +Example: Permission Group. + +.. image:: images/permission_group.png + :align: center + :class: with-shadow + :alt: Managing Permissions + +User Roles +========== +Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username. + +.. image:: images/user_role.png + :align: center + :class: with-shadow + :alt: User Roles + diff --git a/en/3.0/_sources/acls.rst.txt b/en/3.0/_sources/acls.rst.txt new file mode 100644 index 0000000000..4ff5a7ab12 --- /dev/null +++ b/en/3.0/_sources/acls.rst.txt @@ -0,0 +1,115 @@ +.. meta:: + :description: Access Control Lists (ACLs) + +.. _acl_def: + +########################## +Access Control Lists (ACL) +########################## +Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access. + +Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches. + +Screenshot: TCAM utilization can be seen under Net→Inventory + +.. image:: images/TCAM.png + :align: center + :class: with-shadow + +Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements. + +ACL Default Policy +------------------ +The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules. + +Example: Changing “ACL Default Policy” for the site “siteDefault”. + +.. image:: images/siteDefault.png + :align: center + :class: with-shadow + + +ACL Rules +--------- +ACL rules can be created, listed, edited, approved under Services→ACL. + +Description of ACL fields. +General + +* **Name** - Unique name for the ACL entry. +* **Protocol** - IP protocol to match. + + * All - Any IP protocols. + * IP - Specific IP protocol number. + * TCP - TCP. + * UDP - UDP. + * ICMP ALL - Any IPv4 ICMP protocol. + * ICMP Custom - Custom IPv4 ICMP code. + * ICMPv6 ALL - Any IPv6 ICMP protocol. + * ICMPv6 Custom - Custom IPv6 ICMP code. + +* **Active Until** - Disable this rule at the defined date/time. +* **Action** - Permit or Deny forwarding of matched packets. +* **Established/Reverse** - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination. + +Source/Destination - Source and destination addresses and ports to match. + +* **Source** IPv4/IPv6 - IPv4/IPv6 address. +* **Ports Type** + + * Port Range - Match on the port or a port range defined in this window. + * Port Group - Match on a group of ports defined under Services→ ACL Port Group. + +* **From Port** - Port range starting from. +* **To Port** - Port range ending with. + +* **Comment** - Descriptive comment, commonly used for approval workflows. + +* **Check button** - Check if Another ACL on the system already permits the described network access. + +Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established). + +.. image:: images/action_permit.png + :align: center + :class: with-shadow + +Example: “Check” shows that requested access is already provided by a broader ACL rule. + +.. image:: images/ACL_rule.png + :align: center + :class: with-shadow + +ACL Approval Workflow +--------------------- +When one tenant (one team) needs to get network access to resources under the responsibility of another tenant (another team), an ACL can be created but will activate only after approval of the tenant responsible for the destination address resources. See the below example. + +Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant. + +.. image:: images/ACL_approval.png + :align: center + :class: with-shadow + +Screenshot: ACL stays in “waiting for approval” state until approved. + +.. image:: images/waiting_approval.png + :align: center + :class: with-shadow + +Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it. + +.. image:: images/approve_reject.png + :align: center + :class: with-shadow + +Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric. + +.. image:: images/ACL_active.png + :align: center + :class: with-shadow + +ACL Processing Order +-------------------- +#. User-defined Deny Rules +#. User-defined Permit Rules +#. Deny the rest + diff --git a/en/3.0/_sources/controller-k3s-installation.rst.txt b/en/3.0/_sources/controller-k3s-installation.rst.txt new file mode 100644 index 0000000000..4076d9eb1e --- /dev/null +++ b/en/3.0/_sources/controller-k3s-installation.rst.txt @@ -0,0 +1,167 @@ + +.. meta:: + :description: Controller Generic Linux Host + +###################################################### +Netris Controller installation on a generic Linux host +###################################################### + +Linux Host requirements +======================= + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +.. note:: + + K3s is expected to work on most modern Linux systems. + + Some OSs have specific requirements: + + * If you are using Raspbian Buster, follow `these steps `__ to switch to legacy iptables. + * If you are using Alpine Linux, follow `these steps `__ for additional setup. + * If you are using (Red Hat/CentOS) Enterprise Linux, follow `these steps `__ for additional setup. + + +Installation +============ + +The following command will install the Netris Controller on your Linux server: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh - + +Once installed, you will be able to log in to Netris Controller using your host's IP address. + + +.. note:: + The installation script does the following: + + * Installs `k3s `_ + * Installs the `Cert-Manager Helm chart `_ + * Installs the `Netris Controller Helm chart `_ + + + +Installation with the specific host name +---------------------------------------- + +In order to set the specific ingress host name to the Netris Controller, use the ``--ctl-hostname`` installation argument: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com + +A self-signed SSL certificate will be generated from that host name. + +Installation with the Let's Encrypt SSL +--------------------------------------- + +The installation script supports Let's Encrypt SSL generation out-of-box. To instruct the installation script to do that use ``--ctl-ssl-issuer`` argument. + +.. note:: + | The argument ``--ctl-ssl-issuer`` is passing ``cert-manager.io/cluster-issuer`` value to the ingress resource of the Netris Controller. The installation script can create two types of ClusterIssuer resource: ``selfsigned`` or ``letsencrypt``, where ``selfsigned`` is just `Cert-Manager self-signed `_ SSL and the ``letsencrypt`` is the ACME issuer with `HTTP01 challenge validation `_. + | If the ``--ctl-ssl-issuer`` argument is not set, the installation script will proceed with ``selfsigned`` ClusterIssuer type. + + +Run the following command to install Netris Controller and use ``letsencrypt`` ClusterIssuer for SSL generation: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + + To successfully validate and complete Let's Encrypt SSL generation, a valid A/CNAME record for the domain/subdomain name should exist prior, and that name must be accessible from the Internet. + + +Installation with the Custom SSL Issuer +--------------------------------------- + +The HTTP01 challenge validation is the simplest way of issuing the Let's Encrypt SSL, but it does not work when the host behind the FQDN is not accessible from the public internet. +The common approach of validating and completing Let's Encrypt SSL generation for private deployments is `DNS01 challenge validation `_. +If the ``DNS01`` does not work for you either, Cert-Manager supports a number of certificate issuers, get familiar with all types of issuers `here `_. + +In order to install Netris Controller with the custom SSL issuer, you need to run installation script with the specified host name: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com + +Once the installation is complete, create a yaml file with the ``ClusterIssuer`` resource, suitable for your requirements, and apply it: + +.. code-block:: shell-session + + kubectl apply -f my-cluster-issuer.yaml + +Then rerun the installation script with the ``--ctl-ssl-issuer`` argument: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh -s -- --ctl-ssl-issuer + + +Upgrading +========= + +To upgrade the Netris Controller to the latest version simply run the script: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh - + +If a newer version of Netris Controller is available, it will be updated in a few minutes. + + +Uninstalling +============ + +To uninstall Netris Controller and K3s from a server node, run: + +.. code-block:: shell-session + + /usr/local/bin/k3s-uninstall.sh + + +Backup and Restore +================== + +Netris Controller stores all critical data in MariaDB. It's highly recommended to create a cronjob with ``mysqldump``. + + +Backup +------ + +To take database snapshot run the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot-$(date +%Y-%m-%d-%H-%M-%S).sql + +After command execution, you can find ``db-snapshot-YYYY-MM-DD-HH-MM-SS.sql`` file in the current working directory. + + +Restore +------- + +In order to restore DB from a database snapshot, follow these steps: + +1. Copy snapshot file to the MariaDB container: + +.. code-block:: shell-session + + kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql + +2. Run the restore command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql' + +.. note:: + + In this example the snapshot file name is db-snapshot.sql and it's located in the current working directory + diff --git a/en/3.0/_sources/controller-k8s-installation.rst.txt b/en/3.0/_sources/controller-k8s-installation.rst.txt new file mode 100644 index 0000000000..33caaa221a --- /dev/null +++ b/en/3.0/_sources/controller-k8s-installation.rst.txt @@ -0,0 +1,57 @@ + +.. meta:: + :description: Controller Helm Chart Installation + +####################### +Helm Chart Installation +####################### + +Requirements +------------ + +* Kubernetes 1.12+ +* Helm 3.1+ +* PV provisioner support in the underlying infrastructure + +Get Repo Info +------------- + +Add the Netris Helm repository: + +.. code-block:: shell-session + + helm repo add netrisai https://netrisai.github.io/charts + helm repo update + +Installing the Chart +-------------------- + +In order to install the Helm chart, you must follow these steps: + +1. Create the namespace for netris-controller: + +.. code-block:: shell-session + + kubectl create namespace netris-controller + +1. Install helm chart with netris-controller: + +.. code-block:: + + helm install netris-controller netrisai/netris-controller \ + --namespace netris-controller \ + --set app.ingress.hosts={my.domain.com} + +Uninstalling the Chart +--------------------------- + +To uninstall/delete the ``netris-controller`` helm release: + +.. code-block:: + + helm uninstall netris-controller + +Chart Configuration +------------------- + +See the `netris-controller README `_ for details about configurable parameters and their default values. diff --git a/en/3.0/_sources/controller-k8s-quickstart.rst.txt b/en/3.0/_sources/controller-k8s-quickstart.rst.txt new file mode 100644 index 0000000000..e0ffd95c72 --- /dev/null +++ b/en/3.0/_sources/controller-k8s-quickstart.rst.txt @@ -0,0 +1,41 @@ +.. meta:: + :description: Controller Quickstart + +*********************** +Quickstart Installation +*********************** + +Netris offers a simplified deployment model for users who want to quickly install the Netris Controller in the shortest amount of time. + +**This installation process is streamlined for Linux servers that do not already have Kubernetes running.** The install does the following: + +* Installs `k3s `_ +* Installs the `Netris Controller Helm chart `_ + +If you wish to install the controller on an existing Kubernetes cluster, follow `these instructions `_ instead of this Quickstart. + +Quickstart Process +------------------ + +1. Install the Netris Controller w/ k3s by running the following command on your Linux server: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh - + +2. When the installation completes, you will be provided with the login for the web UI. Login with the provided credentials. + +3. Navigate in the UI to Net→IPAM and add a new subnet that contains the desired management IP addresses you wish to use for your SoftGates and switches. + + For example, if you are planning on using 192.168.1.100 as the IP address of your Ubuntu server, then create a subnet in Netris UI for 192.168.1.0/24. + + Detailed configuration documentation is available here: `Netris IPAM `_. + +4. Navigate in the UI to **Topology** +5. Click the **Add** in the upper right +6. Fill out the fields for the SoftGate you wish to add +7. Select the proper **Management IP address** from the subnet selector +8. Once the SoftGate is created in the Topology, **right-click** on the SoftGate and select the **Install Agent** option +9. Copy the agent install command to your clipboard and run the command on the Ubuntu server you are using as your SoftGate +10. Congratulations. The SoftGate should now be connected to your controller. + diff --git a/en/3.0/_sources/controller-vm-installation.rst.txt b/en/3.0/_sources/controller-vm-installation.rst.txt new file mode 100644 index 0000000000..9dcc70fca5 --- /dev/null +++ b/en/3.0/_sources/controller-vm-installation.rst.txt @@ -0,0 +1,191 @@ +.. meta:: + :description: Controller Virtual Machine Installation + +**************************** +Virtual Machine Installation +**************************** + +Requirements +============ + +Minimal system requirements for the VM: + +* CPU - 4 Core +* RAM - 4 Gb +* Disk - 100Gb +* Network - 1 virtual NIC + +Recommended system requirements for the VM: + +* CPU - 8 Core +* RAM - 16 Gb +* Disk - 100Gb +* Network - 1 virtual NIC + +KVM Hypervisor Installation +=========================== +If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04) + +.. code-block:: shell-session + + sudo apt-get install virt-manager + +VM Controller Installation +========================== + +1. Download the Netris Controller image. (contact Netris support for repository access permissions). + +.. code-block:: shell-session + + cd /var/lib/libvirt/images + + sudo wget http://img.netris.ai/netris-controller3.qcow2 + +2. Download VM definition file. + +.. code-block:: shell-session + + cd /etc/libvirt/qemu + + sudo wget http://img.netris.ai/netris-controller3.xml + +3. Define the KVM virtual machine + +.. code-block:: shell-session + + sudo virsh define netris-controller3.xml + +.. note:: + + Netris Controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below for the network interface configuration example. + +Example: Network configuration on host (hypervisor) machine. + +.. note:: + + replace , and + with the correct NIC and IP for your host machine. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + #Physical NIC connected to the management network + auto + iface inet static + address 0.0.0.0/0 + + #bridge interface + auto br-mgmt + iface br-mgmt inet static + address + gateway + bridge-ports + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +4. Set the virtual machine to autostart and start it. + +.. code-block:: shell-session + + sudo virsh autostart netris-controller + +.. code-block:: shell-session + + sudo virsh start netris-controller + +Accessing the Netris Controller +=============================== +By default, Netris Controller will obtain an IP address from a **DHCP** server. + +Below steps describe how to configure a **Static IP** address for the Netris Controller. + +1. Connecting to the VM console. + +default credentials. **login**: ``netris`` **password**: ``newNet0ps`` + +.. code-block:: shell-session + + sudo virsh console netris-controller + +.. note:: + + Do not forget to change the default password (using passwd command). + +2. Setting a static IP address. + +Edit network configuration file. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +Example: IP configuration file. + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +Reload the network config. + +.. code-block:: shell-session + + sudo ifreload -a + +.. note:: + + Make sure Netris Controller has Internet access. + +3. Reboot the controller + +.. code-block:: shell-session + + sudo reboot + +After reboot, the Netris Controller GUI should be accessible using a browser. Use ``netris/newNet0ps`` credentials. + +.. image:: images/credentials.png + :align: center + :class: with-shadow + :alt: Netris Credentials + +.. note::Don’t forget to change the default password by clicking your login name in the top right corner and then clicking “Change Password”. + +Replacing the SSL certificate +============================= + +1. Replace the below file with your SSL certificate file. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.cert.pem; + +2. Replace the below file with your SSL private key. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.key.pem; + +3. Restart Nginx service. + +.. code-block:: shell-session + + systemctl restart nginx.service diff --git a/en/3.0/_sources/definitions.rst.txt b/en/3.0/_sources/definitions.rst.txt new file mode 100644 index 0000000000..c49a4277cd --- /dev/null +++ b/en/3.0/_sources/definitions.rst.txt @@ -0,0 +1,26 @@ +.. meta:: + :description: Definitions + +=========== +Definitions +=========== + +When configuring and operating a Netris system, the following nomenclature is important to understand: + +* **User** - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is ``netris``, with password ``newNet0ps``. + +* **Tenant** - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request and manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs) via a DevOps/NetOps pipeline. + +* **Permission Group** - List of permissions on a per section basis can be attached individually to a User or a User Role + +* **User Role** - Group of user permissions and tenants for role-based access control (RBAC) + +* **Site** - Each separate deployment (each data center) should be defined as a *Site*. All network units and resources are attached to a site. Netris Controller comes with a "default" site preconfigured. Site entry defines global attributes such as; AS numbers, default ACL policy, and Site Mesh (site to site VPN) type. + +* **Subnet** - IPv4/IPv6 address resources linked to *Sites* and *Tenants* + +* **Switch Port** - Physical ports of all switches attached to the system + +* **Inventory** - Inventory of all network units that are operated using Netris Agent + +* **E-BGP** - Defines all External BGP peers (iBGP and eBGP) diff --git a/en/3.0/_sources/index.rst.txt b/en/3.0/_sources/index.rst.txt new file mode 100644 index 0000000000..dbea9c39ba --- /dev/null +++ b/en/3.0/_sources/index.rst.txt @@ -0,0 +1,84 @@ +.. Read the Docs Template documentation master file, created by + sphinx-quickstart on Tue Aug 26 14:19:49 2014. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to Netris Documentation +================================================== + +Netris is the Automatic NetOps platform that runs the physical network and provides cloud-like user experience for NetOps and DevOps engineers. + +.. toctree:: + :maxdepth: 2 + :caption: Tutorials + + tutorials/index + +.. toctree:: + :maxdepth: 4 + :caption: Cloud Native Tools + + kubernetes-integration + terraform-integration + +.. toctree:: + :maxdepth: 2 + :caption: Netris Fundamentals + + introduction + netris-architecture + supported-networks + +.. toctree:: + :maxdepth: 2 + :caption: Detailed Installation + + installation + switch-agent-installation + SoftGate-installation + SoftGate-PRO-installation + +.. toctree:: + :maxdepth: 2 + :caption: Switch-fabric Configuration + + definitions + ipam + topology-management + switch-ports + +.. toctree:: + :maxdepth: 4 + :caption: Network Policies + + network-policies + +.. toctree:: + :maxdepth: 4 + :caption: Network Services + + vnet + l3-load-balancer + l4-load-balancer + acls + roh + +.. toctree:: + :maxdepth: 4 + :caption: Operations + + visibility + accounts + +.. toctree:: + :maxdepth: 4 + :caption: Updates + + release-notes + softgate-performance + +.. toctree:: + :maxdepth: 4 + :caption: Lab Scenarios + + onprem-k8s diff --git a/en/3.0/_sources/installation.rst.txt b/en/3.0/_sources/installation.rst.txt new file mode 100644 index 0000000000..421ea55b9e --- /dev/null +++ b/en/3.0/_sources/installation.rst.txt @@ -0,0 +1,20 @@ +.. meta:: + :description: Controller Installation + +======================= +Controller Installation +======================= + +Netris Controller can be installed locally as a VM, deployed as a Kubernetes application, or hosted in the Netris Cloud. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime. + +.. note:: + + Select ONE installation option below. + + +.. toctree:: + :maxdepth: 2 + :caption: Controller Installation + + controller-k3s-installation + controller-k8s-installation diff --git a/en/3.0/_sources/installing-netris-controller.rst.txt b/en/3.0/_sources/installing-netris-controller.rst.txt new file mode 100644 index 0000000000..4ad210415f --- /dev/null +++ b/en/3.0/_sources/installing-netris-controller.rst.txt @@ -0,0 +1,37 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Installing a Netris Controller +============================== + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment. + +It doesn’t matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. So you can access the console, and nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface. + +Linux Host requirements + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +In this example I am running my Netris controller on an AWS hosted virtual machine (EC2) which has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +I’m using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my EC2 : 54.219.211.71. + +.. image:: images/cloudflare-dns-record.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-hostname ” will instruct the installer to generate a Let’s Encrypt SSL certificate for the provided domain name. That’s why it is important to create the DNS record before this step. + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com + +Once installation process is finished you will be able to access your newly installed Netris Controller web consonle using netris/newNet0ps credentials. + +Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller. diff --git a/en/3.0/_sources/introduction.rst.txt b/en/3.0/_sources/introduction.rst.txt new file mode 100644 index 0000000000..c3769c815a --- /dev/null +++ b/en/3.0/_sources/introduction.rst.txt @@ -0,0 +1,11 @@ +.. meta:: + :description: Introduction to Netris + +Introduction to Netris +====================== + +Netris is an automatic netops software for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network's health and either applies software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation. + +.. image:: images/netris-architecture.png + :align: center + \ No newline at end of file diff --git a/en/3.0/_sources/inventory-profiles.rst.txt b/en/3.0/_sources/inventory-profiles.rst.txt new file mode 100644 index 0000000000..98737f45a5 --- /dev/null +++ b/en/3.0/_sources/inventory-profiles.rst.txt @@ -0,0 +1,26 @@ +.. meta:: + :description: Inventory Profiles + +================== +Inventory Profiles +================== + +Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/softgate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except netris-defined and user-defined custom flows. Generated rules include: + +* SSH from user defined subnets +* NTP from user defined ntp services +* DNS from user defined DNS servers +* Custom user defined rules + +.. csv-table:: Inventory Profile Fields + :file: tables/inventory-profile-fields.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup). + +.. image:: images/inventory-profile.png + :align: center + :class: with-shadow + :alt: Adding an inventory profile diff --git a/en/3.0/_sources/ipam.rst.txt b/en/3.0/_sources/ipam.rst.txt new file mode 100644 index 0000000000..70f13c1453 --- /dev/null +++ b/en/3.0/_sources/ipam.rst.txt @@ -0,0 +1,96 @@ +.. meta:: + :description: IP Address Management + +.. _ipam_def: + +===================== +IP Address Management +===================== + +Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting. + +Purpose: +Users define specific roles(purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-net, NAT, etc… + + +Allocations and Subnets +----------------------- + +There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets. + + +.. image:: images/subnet-tree.png + :align: center + :alt: IPAM Tree View + :class: with-shadow + + IPAM Tree View + +-------------------------- + +Add an Allocation +----------------- + +#. Navigate to Net→IPAM +#. Click the **Add** button +#. Select **Allocation** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Allocation Fields + :widths: 25 50 + :header-rows: 0 + + * - Name + - Unique name for current allocation. + * - Prefix + - Unique prefix for allocation, must not overlap with other allocations. + * - Tenant + - Owner of the allocation. + +.. image:: images/add-allocation.png + :align: center + :class: with-shadow + :alt: Add a New IP Allocation + + Add Allocation Window + +-------------------------- + +Add a Subnet +------------ + +#. Navigate to Net→IPAM +#. Click the **Add** button +#. Select **Subnet** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Subnet fields + :widths: 25 50 + :header-rows: 0 + + * - **Name** + - Unique name for current subnet. + * - **Prefix** + - Unique prefix for subnet, ust be included in one of allocations. + * - **Tenant** + - Owner of the subnet. + * - **Purpose** + - This field describes for what kind of services the current subnet can be used. It can have the following values: + + - *common* - ordinary subnet, can be used in v-nets and ROH. + - *loopback* - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates). + - *management* - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates). + - *load-balancer* - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience. + - *nat* - hosts of this subnet or subnet itself can be used to define NAT services. + - *inactive* - can’t be used in any services, useful for reserving/documenting prefixes for future use. + +.. image:: images/add-subnet.png + :align: center + :alt: Add a New Subnet + :class: with-shadow + + Add Subnet Window diff --git a/en/3.0/_sources/kubernetes-integration.rst.txt b/en/3.0/_sources/kubernetes-integration.rst.txt new file mode 100644 index 0000000000..31e8c73948 --- /dev/null +++ b/en/3.0/_sources/kubernetes-integration.rst.txt @@ -0,0 +1,477 @@ +.. meta:: + :description: Kubernetes Integration + +######################## +Kubernetes Integration +######################## +Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris-Kubernetes integration is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters. + + +Install Netris Operator +======================= + +Integration between the Netris Controller and the Kubernetes API is completed by installing the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart: + +Helm Chart Method +----------------- +Instructions are available on Github: https://github.com/netrisai/netris-operator/tree/master/deploy/charts/netris-operator#installing-the-chart + +Regular Manifest Method +----------------------- + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='http://**your-netris-controller-ip-or-host**' \ + --from-literal=login='**your-netris-admin-username**' --from-literal=password='**your-netris-admin-password**' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Using Type 'LoadBalancer' +========================= + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :class: with-shadow + :alt: Sandbox pod provisioning + + +After provisioning has finished, inspect the service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.202 9898:32584/TCP,9999:30365/TCP 9m17s + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9898 active 50.117.59.202 9898/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9898 active 50.117.59.202 9898/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.203 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.203 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :class: with-shadow + :alt: View L4 LB instances + +V-Net Custom Resource +--------------------- + +You can also create Netris V-Nets (L2 segments) via Kubernetes with a simple manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +BGP Custom Resource +------------------- + +You can create BGP peers via Kubernetes manifests: + +1. Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1092 + localIP: 50.117.59.118/30 + remoteIP: 50.117.59.117/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.192/28 le 32 + EOF + +2. Apply the manifest file: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +3. Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.118/30 50.117.59.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.118/30 50.117.59.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Calico CNI Integration +====================== + +Netris Operator can integrate with Calico CNI. This annotation will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.118/30 50.117.59.117/30 7m59s + sandbox9-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox9-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox9-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.118/30 50.117.59.117/30 8m41s + sandbox9-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox9-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox9-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.202 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. diff --git a/en/3.0/_sources/l3-load-balancer.rst.txt b/en/3.0/_sources/l3-load-balancer.rst.txt new file mode 100644 index 0000000000..7dc6b0b778 --- /dev/null +++ b/en/3.0/_sources/l3-load-balancer.rst.txt @@ -0,0 +1,44 @@ +.. meta:: + :description: Layer-3 Load Balancer (Anycast) + +.. _l3lb_def: + +############################# +L3 Load Balancer (Anycast LB) +############################# +L3 (Anycast) load balancer leverages ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks. + +ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server. + +End-user traffic should be destined to the anycast IP address. The switch fabric uses ECMP to load balance the traffic towards every server, and will hash sessions based on IP/Protocol/Port such that TCP sessions will exist between the given end-user and server pair for the lifetime of the session. Optional health checks are used to identify application failures and reroute traffic in the case of a server outage. + +Creating an L3 Load Balancer +============================ + +To configure L3 (Anycast) load balancing: + +#. Navigate to **Services→Instances (ROH)** and locate an existing ROH instance +#. Click the ellipses and then the **Edit** button +#. Select an extra IPv4 address from the select box at the bottom, and check the **Anycast** option. +#. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances. + +.. image:: images/add-l3-lb.png + :align: center + :class: with-shadow + :alt: Add a L3 LB + + Example: Adding an Anycast IPv4 address + +.. image:: images/list-l3-lb.png + :align: center + :class: with-shadow + :alt: List L3 LBs + + Example: Under Services→Load Balancer, you can find the listing of L3 (Anycast) Load Balancers, service statuses, and you can add/remove more ROH instances and/or health checks. + +.. image:: images/list-l3-lb-detail.png + :align: center + :class: with-shadow + :alt: List L3 LB Details + + Screenshot: L3 (Anycast) Load Balancer Detail diff --git a/en/3.0/_sources/l4-load-balancer.rst.txt b/en/3.0/_sources/l4-load-balancer.rst.txt new file mode 100644 index 0000000000..90d2c3e72c --- /dev/null +++ b/en/3.0/_sources/l4-load-balancer.rst.txt @@ -0,0 +1,83 @@ +.. meta:: + :description: Netris Services and Configuration Examples + + +####################### +L4 Load Balancer (L4LB) +####################### +Netris L4 Load Balancer (L4LB) leverages SoftGate(Linux router) nodes to provide Layer-4 load balancing services, including on-demand cloud load balancing with native integration with Kubernetes. + +Enabling L4LB service +--------------------- +L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer). + +The IP address pool for L4LB can be defined in the Net→IPAM section by adding an Allocation and setting the purpose field to ‘load-balancer’. You can define multiple IP pools for L4LB at any given site. See the below example. + +Example: Adding a load-balancer IP pool assignment. + +.. image:: images/add-allocation.png + :align: center + :class: with-shadow + :alt: Add an IP Allocation + + +Screenshot: Listing of Net→IPAM after adding a load-balancer assignment + +.. image:: images/list-subnets.png + :align: center + :class: with-shadow + :alt: List IP Subnets + + +Consuming L4LB service +---------------------- +This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section. + +Click +add under Services→L4 Load Balancer to request an L4LB service. + +Add new L4 Load Balancer fields are described below: + +**General fields** + +* **Name** - Unique name. +* **Protocol** - TCP or UDP. +* **Tenant** - Requestor Tenant should have access to the backend IP space. +* **Site** - Site where L4LB service is being requested for. Backends should belong on this site. +* **State** - Administrative state. + +**Frontend** + +* **Address** - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses. +* **Port** - TCP or UDP port to be exposed. + +**Health-check** + +* **Type** - Probe backends on service availability. + + * **None** - load balance unconditionally. + * **TCP** - probe backend service availability through TCP connect checks. + * **HTTP** - probe backend service availability through HTTP GET checks. + +* **Timeout(ms)** - Probe timeout in milliseconds. +* **Request path** - HTTP request path. + +**Backend** + +* **+Add** - add a backend host. +* **Address** - IP address of the backend host. +* **Port** - Service port on the backend host. +* **Enabled** - Administrative state of particular backend. + +.. image:: images/request-L4.png + :align: center + :class: with-shadow + :alt: Request an L4 Load Balancer + + Example: Requesting an L4 Load Balancer service. + +.. image:: images/list-l4-load-balancers.png + :align: center + :class: with-shadow + :alt: List L4 Load Balancers + + Example: Listing of L4 Load Balancer services \ No newline at end of file diff --git a/en/3.0/_sources/netris-architecture.rst.txt b/en/3.0/_sources/netris-architecture.rst.txt new file mode 100644 index 0000000000..fbec46980b --- /dev/null +++ b/en/3.0/_sources/netris-architecture.rst.txt @@ -0,0 +1,53 @@ +.. meta:: + :description: Netris Architecture + +.. _netris_architecture: + +################### +Netris Architecture +################### + +A Netris system is composed of 4 elements: + +* Netris Controller +* Netris Switch Agent +* Netris SoftGate +* Customer Application Servers + +.. _netris_controller_def: + +Netris Controller +================= +Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. The Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud. + +Diagram: High level Netris architecture + +.. image:: images/netris_controller_diagram.png + :align: center + +* **Controller HA** We highly recommend running more than one copy of the controller for database replication. +* **Multiple sites** Netris is designed to operate multiple sites with just a single controller with HA. +* **What if the controller is unreachable.** Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers core operations will not be affected. + +.. _netris_sw_agent: + +Netris Switch Agent +=================== +Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet. + +.. _netris_sg_agent: + +Netris SoftGate +=============== +Netris SoftGate is automatic configuration software and reference architecture for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), and site-to-site VPN function on a regular x86 server with a SmartNIC card. + +Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2. + +The server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities. + +Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels. + +Diagram: Netris SoftGate high level architecture + +.. image:: images/softgate_diagram.png + :align: center diff --git a/en/3.0/_sources/network-policies.rst.txt b/en/3.0/_sources/network-policies.rst.txt new file mode 100644 index 0000000000..c8b5527ab1 --- /dev/null +++ b/en/3.0/_sources/network-policies.rst.txt @@ -0,0 +1,343 @@ +.. meta:: + :description: Netris Network Policies & Protocol Configuration + +.. _bgp_def: + +######### +Basic BGP +######### + +BGP neighbors can be declared in the Net→E-BGP section. Netris will automatically generate and program the network configuration to meet the requirements. + +Adding BGP Peers +---------------- +#. Navigate to Net→E-BGP in the web UI +#. Click the **Add** button +#. Fill in the fields as described in the table below +#. Click the **Add** button + +.. csv-table:: BGP Peer Fields + :file: tables/bgp-basic.csv + :widths: 25, 75 + :header-rows: 0 + +Example: Declare a basic BGP neighbor + +.. image:: images/add-bgp-basic.png + :align: center + :class: with-shadow + +Example2: Declare BGP neighbor terminated on V-Net. Netris will automatically configure BGP session on the switch closest to the remote IP. +.. image:: images/add-bgp-basic-2.png + :align: center + :class: with-shadow + + +############ +Advanced BGP +############ +BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies. + +Click Advanced to expand the BGP neighbor add/edit window. + +.. csv-table:: BGP Peer Fields - Advanced + :file: tables/bgp-advanced.csv + :widths: 25, 75 + :header-rows: 0 + + +-------------------------- + +BGP Objects +----------- +| Under Net→E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy. +| Supported objects include: + +* IPv4 Prefix +* IPv6 Prefix +* AS-PATH +* Community +* Extended Community +* Large Community + +IPv4 Prefix +^^^^^^^^^^^ +| Rules defined one per line. +| Each line in IPv4 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv4 prefix (mandatory). +* Length - Possible values are: le , ge or ge le . + +Example: Creating an IPv4 Prefix list. + +.. image:: images/IPv4-Prefix.png + :align: center + :class: with-shadow + +IPv6 Prefix +^^^^^^^^^^^ +| Rules defined one per line. +| Each line in IPv6 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv6 prefix (mandatory). +* Keyword - Possible values are: le , ge or ge le . + +Example: Creating an IPv6 Prefix list. + +.. image:: images/IPv6-Prefix.png + :align: center + :class: with-shadow + +Community +^^^^^^^^^ +| Community field has two parts: + +* Action - Possible values: permit or deny (mandatory). +* Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive). + +Example: Creating community. + +.. image:: images/community.png + :align: center + :class: with-shadow + +-------------------------- + +BGP route-maps +-------------- +| Under the Net→E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound. +| Description of route-map fields: + +* **Sequence Number** - Automatically assigned a sequence number. Drag and move sequences to organize the order. +* **Description** - Free description. +* **Policy** - Permit or deny the routes which match below all match clauses within the current sequence. +* **Match** - Rules for route matching. + + * **Type** - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag. + * **Object** - Select an object from the list. + +* **Action** - Action when all match clauses are met. + + * **Action type** - Define whether to manipulate a particular BGP attribute or go to another sequence. + * **Attribute** - The attribute to be manipulated. + * **Value** - New attribute value. + +Example: route-map + +.. image:: images/route-map.png + :align: center + :class: with-shadow + +-------------------------- + +############## +Static Routing +############## +Located under Net→Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. +We recommend using the Routes only if BGP is not supported by the remote end. + +| Typical use cases for static routing: +* To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported. +* Temporary interconnection with the old network for a migration. +* Routing a subnet behind a VM hypervisor machine for an internal VM network. +* Specifically routing traffic destined to a particular prefix through an out-of-band management network. + +| Add new static route fields description: +* **Prefix** - Route destination to match. +* **Next-Hop** - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network. +* **Description** - Free description. +* **Site** - Site where Route belongs. +* **State** - Administrative (enable/disable) state of the Route. +* **Apply to** - Limit the scope to particular units. It’s typically used for Null routes. + + +Example: Default route pointing to a Next-Hop that belongs to one of V-NETs. + +.. image:: images/defaultroute.png + :align: center + :class: with-shadow + +Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network. + +.. image:: images/static_route.png + :align: center + :class: with-shadow + +Screenshot: This Shows that my back route is actually applied on leaf1 and spine1. + +.. image:: images/leaf1_spine1.png + :align: center + :class: with-shadow + +-------------------------- + +.. _nat_def: + +### +NAT +### + +Netris SoftGate nodes are required to support NAT (Network Address Translation). + +Enabling NAT +------------ +To enable NAT for a given site, you first need to create a subnet with NAT purpose in the IPAM section. NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need. + +1. Allocate a public IP subnet for NAT under Net→IPAM. + +Example: Adding an IP allocation under Net→Subnets. + +.. image:: images/IP-allocation.png + :align: center + :class: with-shadow + +1. Attach NAT IP addresses and/or NAT IP Pools to just one SoftGate node. Other SoftGate Nodes on the same site will automatically add the same NAT IP/Pool resources for proper consistency and high availability. + +Example: Adding NAT IP addresses and NAT IP Address Pools to a SoftGate node. + +.. image:: images/NATIP-address.png + :align: center + :class: with-shadow + +Defining NAT rules +------------------ +NAT rules are defined under Net→NAT. + +.. list-table:: NAT Rule Fields + :widths: 25 75 + :header-rows: 1 + + * - Name + - Unique name + * - **State** + - State of rule (enabled or disabled) + * - **Site** + - Site to apply the rule + * - **Action** + - *SNAT* - Replace the source IP address with specified NAT IP along with port overloading + *DNAT* - Replace the destination IP address and/or destination port with specified NAT IP + *ACCEPT* - Silently forward, typically used to add an exclusion to broader SNAT or DNAT rule + *MASQUERADE* - Replace the source IP address with the IP address of the exit interface + * - **Protocol** + - *All* - Match any IP protocol + *TCP* - Match TCP traffic and ports + *UDP* - Match UDP traffic and ports + *ICMP* - Match ICMP traffic + * - **Source** + - *Address* - Source IP address to match + *Port* - Source ports range to match with this value (TCP/UDP) + * - **Destination** + - *Address* - Destination IP address to match. In the case of DNAT it should be one of the predefined NAT IP addresses + *Port* - For DNAT only, to match a single destination port + *Ports* - For SNAT/ACCEPT only. Destination ports range to match with this value (TCP/UDP) + * - **DNAT to IP** + - The global IP address for SNAT to be visible on the Public Internet. The internal IP address for DNAT to replace the original destination address with + * - **DNAT to Port** + - The Port to which destination Port of the packet should be NAT'd + * - **Status** + - Administrative state (enable/disable) + * - **Comment** + - Free optional comment + + +Example: SNAT all hosts on 10.0.0.0/8 to the Internet using 198.51.100.65 as a global IP. + +.. image:: images/globalIP.png + :align: center + :class: with-shadow + +Example: Port forwarding. DNAT the traffic destined to 198.51.100.66:80 to be forwarded to the host 10.0.4.10 on port tcp/1080. + +.. image:: images/Port-Forwarding.png + :align: center + :class: with-shadow + +-------------------------- + +######## +SiteMesh +######## +SiteMesh is a Netris service for site-to-site interconnects over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site. + +Edit Net->Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below. + +* **Disabled** - Do not participate in SiteMesh. +* **Hub** - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites) +* **Spoke** - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites) +* **Dynamic Spoke** - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites) + +Screenshot: Site Mesh parameter editing a Site under Net→Sites. + +.. image:: images/Site_Mesh.png + :align: center + +You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge. + +.. image:: images/SiteMesh_modes.png + :align: center + +Check the Net→Site Mesh section for the listing of tunnel statuses. + +Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh) + +.. image:: images/SiteMesh_listing.png + :align: center + +-------------------------- + +############# +Looking Glass +############# +The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Net→Looking Glass then selecting the device from the top-left dropdown menu. + +Looking Glass controls described for IPv4/IPv6 protocol families. + +* **BGP Summary** - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes. +* **BGP Route** - Lookup the BGP table (RIB) for the given address. +* **Route** - Lookup switch routing table for the given address. +* **Traceroute** - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address. +* **Ping** - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address. + +Example: Spine1: listing BGP neighbors and number of received prefixes. + +.. image:: images/Spine1.png + :align: center + :class: with-shadow + +Example: BGP Route - looking up my leaf1 switch’s loopback address from spine1’s perspective. Spine1 is load balancing between two available paths. + +.. image:: images/BGP_route.png + :align: center + :class: with-shadow + +Example: Ping. + +.. image:: images/ping.png + :align: center + :class: with-shadow + +| Looking Glass controls described for the EVPN family. +* **BGP Summary** - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received. +* **VNI** - List VNIs learned. +* **BGP EVPN** - List detailed EVPN routing information optionally for the given route distinguisher. +* **MAC table** - List MAC address table for the given VNI. + +Example: Listing of adjacent BGP neighbors and number of EVPN prefixes received. + +.. image:: images/BGP_neighbors_listing.png + :align: center + :class: with-shadow + +Example: Listing MAC addresses on VNI 2. + +.. image:: images/MAC_listing.png + :align: center + :class: with-shadow + +Example: EVPN routing information listing for a specified route distinguisher. + +.. image:: images/EVPN_routing.png + :align: center + :class: with-shadow diff --git a/en/3.0/_sources/reference-designs.rst.txt b/en/3.0/_sources/reference-designs.rst.txt new file mode 100644 index 0000000000..8c1b63a61d --- /dev/null +++ b/en/3.0/_sources/reference-designs.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Network Reference Designs + +######################### +Network Reference Designs +######################### + +Netris can support any type of standard network design. + +The majority of designs fall into one of four patterns: + +#. Unmanaged Switch +#. Single L2 Domain +#. Collapsed Core +#. Data Center Leaf/Spine Designs (Modern High Throughput Redundant Design) +#. Data Center Core/Distribution/Access (Legacy High Throughput Redundant Design) \ No newline at end of file diff --git a/en/3.0/_sources/release-notes.rst.txt b/en/3.0/_sources/release-notes.rst.txt new file mode 100644 index 0000000000..ea6c869c70 --- /dev/null +++ b/en/3.0/_sources/release-notes.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Netris Release Notes + +############# +Release notes +############# +* DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGate performance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on. +* L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, we now support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates with Kubernetes providing cloud-like load balancer service (type: load-balancer). +* Kubenet​ - a network service purpose-built for Kubernetes cluster nodes. Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking with modern physical networking. +* API logs​ - Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type. +* Site Mesh​ - a Netris service for automatically configuring site-to-site interconnect over the public Internet. Site Mesh supports configuration for WireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a few clicks, services in one site get connectivity to services in other sites over a mesh of WireGuard tunnels. +* Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loop cable. Removed the need for IP address reservation for V-NET, switching entirely to the anycast default gateway. +* Controller distributions​ - Netris controller, is now available in three deployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application. 3) Managed/Hosted in the cloud. +* Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes. +* Switch/SoftGate agents​ - New installer with easy initial config tool. Support for IP and FQDN as a controller address. Authentication key. +* GUI​ - Improved Net→Topology section, becoming the main and required place for defining the network topology. All sections got a column organizer, so every user can order and hide/show columns to their comfort. diff --git a/en/3.0/_sources/roh.rst.txt b/en/3.0/_sources/roh.rst.txt new file mode 100644 index 0000000000..ff9f901921 --- /dev/null +++ b/en/3.0/_sources/roh.rst.txt @@ -0,0 +1,58 @@ +.. meta:: + :description: Routing on the Host + +.. _roh_def: + +######################### +ROH (Routing on the Host) +######################### +To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly on their servers. This is commonly known as ROH (Routing on the Host). + +In ROH architectures, servers use a routing daemon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon/suite is FRR. + +Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead an IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. This is a modern and optimal design, leveraging Layer-3 networking from the fabric to the servers. + +By using only Layer-3 interfaces, Layer-2 protocols such as Spanning Tree (STP) can be minized and the reliability of the network increases. + +The ROH architecture that is configured by Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). For each instance of ROH, you’ll need to create a ROH entry in Netris Controller. + +Adding ROH Hosts +---------------- + +#. Navigate in the Netris UI to **Services→Instances (ROH)** +#. Click the **Add** button +#. Fill out the form based on the fields in the table below. +#. Click the **Add** button + +Description of ROH instance fields: + +- **Name** - Unique name for the ROH instance +- **Site** - Site where the current ROH instance belongs +- **Type** - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor. +- **ROH Routing Profile** - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances + + - **Default route only (most common design)** - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch + - **Default + Aggregate** - Will add prefixes of defined assignments + "Default" profile + - **Full table** - Will advertise all prefixes available in the routing table of the connected switch + - **Inherit** - Will inherit policy from site objects defined under Net→Sites + +- **Legacy Mode** - Switch from default zero-config mode to using /30 IP addresses. Used for MSFT Windows Servers or other OS that doesn’t support FRR. +- **+Port** - Physical Switch Ports anywhere on the network. +- **+IPv4** - IPv4 addresses for the loopback interface. +- **+Inbound Prefix List** - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks. + +.. tip:: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +.. image:: images/ROH-instance.png + :align: center + :class: with-shadow + :alt: ROH Instances + + Example: Adding an ROH instance. (Yes, you can use A.B.C.0/32 and A.B.C.255/32) + +.. image:: images/ROH-listing.png + :align: center + :class: with-shadow + :alt: ROH Listings + + Expanded view of ROH listing. BGP sessions are up, and the expected IP is in fact received from the actual ROH server. Traffic stats are available per port. diff --git a/en/3.0/_sources/sandbox/Sandbox1/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox1/configurations.rst.txt new file mode 100644 index 0000000000..fa8709313a --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox1/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s1-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox1.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc1::1** (from the "**2607:f358:11:ffc1::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.24 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox1/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox1/creating-services.rst.txt new file mode 100644 index 0000000000..9e510e86d7 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox1/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s1-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s1-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s1-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1012``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.0/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.1)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s1-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.4/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.4/32**" IP address. + + * This public IP is part of **45.38.161.4/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s1-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s1-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.8/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox1/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox1/index.rst.txt new file mode 100644 index 0000000000..7d11050256 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox1/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox1 # Sandbox name Uppercase(case sensitive) + sandbox1 # Sandbox name Lowercase + 166.88.17.24 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1011 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1012 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.0/28 # PUBLIC IPv4 Allocation + 45.38.161.0/30 # PUBLIC LOOPBACK subnet + 45.38.161.1 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.4/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.4/32 # CUSTOMER V-NET SNAT IP + 45.38.161.8/30 # L3LB Subnet + 45.38.161.8/32 # L3LB IP + 45.38.161.12/30 # L4LB Subnet + 45.38.161.13 # Second usable ip address in load-balancer subnet + 45.38.161.14 # Third usable ip address in load-balancer subnet + 45.38.161.18/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.17/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.22/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.21/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc1::/64 # public IPv6 subnet + 2607:f358:11:ffc1::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::3/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::2/127 # isp1-ipv6-example BGP peer remote IPv6 + s1-pre-configured # LINK + s1-learn-by-doing # LINK + s1-e-bgp # LINK + s1-v-net # LINK + s1-nat # LINK + s1-acl # LINK + s1-l3lb # LINK + s1-k8s # LINK + s1-topology # LINK + +Sandbox1 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt new file mode 100644 index 0000000000..528a7f7b8d --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s1-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox1-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox1.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox1.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox1-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.13 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.13:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox1-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.13 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.13 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.13 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.13 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.13 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.14 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.14 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox1-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.14 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.14 + + SRV05-NYC + curl 45.38.161.14 + + SRV05-NYC + curl 45.38.161.14 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1012 + localIP: 45.38.161.22/30 + remoteIP: 45.38.161.21/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.0/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.22/30 45.38.161.21/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.22/30 45.38.161.21/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.22/30 45.38.161.21/30 7m59s + sandbox1-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox1-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox1-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.22/30 45.38.161.21/30 8m41s + sandbox1-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox1-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox1-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.13 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox1/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox1/sandbox-info.rst.txt new file mode 100644 index 0000000000..20ec97c9d8 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox1/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s1-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox1.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.24 -p 30061 + srv02-nyc: ssh demo@166.88.17.24 -p 30062 + srv03-nyc: ssh demo@166.88.17.24 -p 30063 + srv04-nyc: ssh demo@166.88.17.24 -p 30064 + srv05-nyc: ssh demo@166.88.17.24 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1011 + Local Address: 45.38.161.18/30 + Remote Address: 45.38.161.17/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.0/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1011 + Local Address: 2607:f358:11:ffc0::3/127 + Remote Address: 2607:f358:11:ffc0::2/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc1::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1012 + Local Address: 45.38.161.22/30 + Remote Address: 45.38.161.21/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.0/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.0/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.0/30 + |___ NAT Subnet: 45.38.161.4/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.8/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.12/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc1::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc1::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox10/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox10/configurations.rst.txt new file mode 100644 index 0000000000..f4a1bc03ad --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox10/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s10-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox10.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffca::1** (from the "**2607:f358:11:ffca::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.19 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox10/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox10/creating-services.rst.txt new file mode 100644 index 0000000000..9c0576e46b --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox10/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s10-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s10-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s10-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1102``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.208/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.209)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s10-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.212/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.212/32**" IP address. + + * This public IP is part of **50.117.59.212/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s10-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s10-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.216/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox10/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox10/index.rst.txt new file mode 100644 index 0000000000..97891e80b2 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox10/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox10 # Sandbox name Uppercase(case sensitive) + sandbox10 # Sandbox name Lowercase + 166.88.17.19 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1101 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1102 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.208/28 # PUBLIC IPv4 Allocation + 50.117.59.208/30 # PUBLIC LOOPBACK subnet + 50.117.59.209 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.212/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.212/32 # CUSTOMER V-NET SNAT IP + 50.117.59.216/30 # L3LB Subnet + 50.117.59.216/32 # L3LB IP + 50.117.59.220/30 # L4LB Subnet + 50.117.59.221 # Second usable ip address in load-balancer subnet + 50.117.59.222 # Third usable ip address in load-balancer subnet + 50.117.59.122/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.121/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.126/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.125/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffca::/64 # public IPv6 subnet + 2607:f358:11:ffca::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::15/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::14/127 # isp1-ipv6-example BGP peer remote IPv6 + s10-pre-configured # LINK + s10-learn-by-doing # LINK + s10-e-bgp # LINK + s10-v-net # LINK + s10-nat # LINK + s10-acl # LINK + s10-l3lb # LINK + s10-k8s # LINK + s10-topology # LINK + +Sandbox10 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt new file mode 100644 index 0000000000..0fc28ec839 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s10-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox10.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox10.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.221 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.221:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.221 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.221 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.221 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.221 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.221 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.222 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.222 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.222 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.222 + + SRV05-NYC + curl 50.117.59.222 + + SRV05-NYC + curl 50.117.59.222 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1102 + localIP: 50.117.59.126/30 + remoteIP: 50.117.59.125/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.208/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.126/30 50.117.59.125/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.126/30 50.117.59.125/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.126/30 50.117.59.125/30 7m59s + sandbox10-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox10-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox10-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.126/30 50.117.59.125/30 8m41s + sandbox10-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox10-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox10-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.221 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox10/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox10/sandbox-info.rst.txt new file mode 100644 index 0000000000..9bb0d1ea4a --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox10/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s10-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox10.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.19 -p 30061 + srv02-nyc: ssh demo@166.88.17.19 -p 30062 + srv03-nyc: ssh demo@166.88.17.19 -p 30063 + srv04-nyc: ssh demo@166.88.17.19 -p 30064 + srv05-nyc: ssh demo@166.88.17.19 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1101 + Local Address: 50.117.59.122/30 + Remote Address: 50.117.59.121/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.208/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1101 + Local Address: 2607:f358:11:ffc0::15/127 + Remote Address: 2607:f358:11:ffc0::14/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffca::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1102 + Local Address: 50.117.59.126/30 + Remote Address: 50.117.59.125/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.208/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.208/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.208/30 + |___ NAT Subnet: 50.117.59.212/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.216/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.220/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffca::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffca::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox11/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox11/configurations.rst.txt new file mode 100644 index 0000000000..550672638e --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox11/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s11-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox11.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcb::1** (from the "**2607:f358:11:ffcb::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.82 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox11/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox11/creating-services.rst.txt new file mode 100644 index 0000000000..4362d69c7e --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox11/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s11-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s11-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.82 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s11-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1112``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.118``. + 10. In the **Remote IP** field, type in ``45.38.161.117``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.96/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.97)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s11-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.82 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.100/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.100/32**" IP address. + + * This public IP is part of **45.38.161.100/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s11-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.82 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s11-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.104/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.104**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.104**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.104**) into the browser's address bar or simply visit `http://45.38.161.104/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.104 (name_45.38.161.104)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.104** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox11/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox11/index.rst.txt new file mode 100644 index 0000000000..2f648ba500 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox11/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox11 # Sandbox name Uppercase(case sensitive) + sandbox11 # Sandbox name Lowercase + 50.117.27.82 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1111 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1112 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.96/28 # PUBLIC IPv4 Allocation + 45.38.161.96/30 # PUBLIC LOOPBACK subnet + 45.38.161.97 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.100/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.100/32 # CUSTOMER V-NET SNAT IP + 45.38.161.104/30 # L3LB Subnet + 45.38.161.104/32 # L3LB IP + 45.38.161.108/30 # L4LB Subnet + 45.38.161.109 # Second usable ip address in load-balancer subnet + 45.38.161.110 # Third usable ip address in load-balancer subnet + 45.38.161.114/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.113/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.118/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.117/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcb::/64 # public IPv6 subnet + 2607:f358:11:ffcb::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::17/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::16/127 # isp1-ipv6-example BGP peer remote IPv6 + s11-pre-configured # LINK + s11-learn-by-doing # LINK + s11-e-bgp # LINK + s11-v-net # LINK + s11-nat # LINK + s11-acl # LINK + s11-l3lb # LINK + s11-k8s # LINK + s11-topology # LINK + +Sandbox11 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt new file mode 100644 index 0000000000..7bfbecfa63 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s11-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox11.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox11.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.109 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.109:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.109 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.109 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.109 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.109 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.109 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.110 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.110 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.110 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.110 + + SRV05-NYC + curl 45.38.161.110 + + SRV05-NYC + curl 45.38.161.110 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1112 + localIP: 45.38.161.118/30 + remoteIP: 45.38.161.117/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.96/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.118/30 45.38.161.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.118/30 45.38.161.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.118/30 45.38.161.117/30 7m59s + sandbox11-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox11-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox11-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.118/30 45.38.161.117/30 8m41s + sandbox11-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox11-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox11-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.109 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox11/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox11/sandbox-info.rst.txt new file mode 100644 index 0000000000..f11dc53159 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox11/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s11-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox11.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.82 -p 30061 + srv02-nyc: ssh demo@50.117.27.82 -p 30062 + srv03-nyc: ssh demo@50.117.27.82 -p 30063 + srv04-nyc: ssh demo@50.117.27.82 -p 30064 + srv05-nyc: ssh demo@50.117.27.82 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1111 + Local Address: 45.38.161.114/30 + Remote Address: 45.38.161.113/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.96/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1111 + Local Address: 2607:f358:11:ffc0::17/127 + Remote Address: 2607:f358:11:ffc0::16/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcb::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1112 + Local Address: 45.38.161.118/30 + Remote Address: 45.38.161.117/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.96/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.96/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.96/30 + |___ NAT Subnet: 45.38.161.100/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.104/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.108/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcb::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcb::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox12/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox12/configurations.rst.txt new file mode 100644 index 0000000000..743ec4961f --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox12/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s12-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox12.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcc::1** (from the "**2607:f358:11:ffcc::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.83 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox12/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox12/creating-services.rst.txt new file mode 100644 index 0000000000..bf8178b954 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox12/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s12-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s12-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.83 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s12-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1122``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.126``. + 10. In the **Remote IP** field, type in ``45.38.161.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.128/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.129)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s12-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.83 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.132/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.132/32**" IP address. + + * This public IP is part of **45.38.161.132/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s12-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.83 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s12-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.136/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.136**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.136**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.136**) into the browser's address bar or simply visit `http://45.38.161.136/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.136 (name_45.38.161.136)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.136** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox12/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox12/index.rst.txt new file mode 100644 index 0000000000..004beb139b --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox12/index.rst.txt @@ -0,0 +1,61 @@ + +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox12 # Sandbox name Uppercase(case sensitive) + sandbox12 # Sandbox name Lowercase + 50.117.27.83 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1121 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1122 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.128/28 # PUBLIC IPv4 Allocation + 45.38.161.128/30 # PUBLIC LOOPBACK subnet + 45.38.161.129 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.132/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.132/32 # CUSTOMER V-NET SNAT IP + 45.38.161.136/30 # L3LB Subnet + 45.38.161.136/32 # L3lB IP + 45.38.161.140/30 # L4LB Subnet + 45.38.161.141 # Second usable ip address in load-balancer subnet + 45.38.161.142 # Third usable ip address in load-balancer subnet + 45.38.161.122/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.121/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.126/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.125/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcc::/64 # public IPv6 subnet + 2607:f358:11:ffcc::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::19/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::18/127 # isp1-ipv6-example BGP peer remote IPv6 + s12-pre-configured # LINK + s12-learn-by-doing # LINK + s12-e-bgp # LINK + s12-v-net # LINK + s12-nat # LINK + s12-acl # LINK + s12-l3lb # LINK + s12-k8s # LINK + s12-topology # LINK + +Sandbox12 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt new file mode 100644 index 0000000000..4b89e3f6a2 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s12-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox12.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox12.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.141 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.141:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.141 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.141 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.141 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.141 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.141 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.142 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.142 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.142 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.142 + + SRV05-NYC + curl 45.38.161.142 + + SRV05-NYC + curl 45.38.161.142 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1122 + localIP: 45.38.161.126/30 + remoteIP: 45.38.161.125/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.128/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.126/30 45.38.161.125/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.126/30 45.38.161.125/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.126/30 45.38.161.125/30 7m59s + sandbox12-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox12-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox12-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.126/30 45.38.161.125/30 8m41s + sandbox12-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox12-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox12-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.141 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox12/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox12/sandbox-info.rst.txt new file mode 100644 index 0000000000..c88c586403 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox12/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s12-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox12.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.83 -p 30061 + srv02-nyc: ssh demo@50.117.27.83 -p 30062 + srv03-nyc: ssh demo@50.117.27.83 -p 30063 + srv04-nyc: ssh demo@50.117.27.83 -p 30064 + srv05-nyc: ssh demo@50.117.27.83 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1121 + Local Address: 45.38.161.122/30 + Remote Address: 45.38.161.121/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.128/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1121 + Local Address: 2607:f358:11:ffc0::19/127 + Remote Address: 2607:f358:11:ffc0::18/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcc::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1122 + Local Address: 45.38.161.126/30 + Remote Address: 45.38.161.125/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.128/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.128/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.128/30 + |___ NAT Subnet: 45.38.161.132/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.136/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.140/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcc::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcc::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox13/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox13/configurations.rst.txt new file mode 100644 index 0000000000..5886c657ec --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox13/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s13-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox13.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcd::1** (from the "**2607:f358:11:ffcd::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.84 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox13/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox13/creating-services.rst.txt new file mode 100644 index 0000000000..8b0b9d04a8 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox13/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s13-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s13-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.84 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s13-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1132``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.166``. + 10. In the **Remote IP** field, type in ``45.38.161.165``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.144/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.145)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s13-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.84 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.148/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.148/32**" IP address. + + * This public IP is part of **45.38.161.148/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s13-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.84 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s13-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.152/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.152**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.152**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.152**) into the browser's address bar or simply visit `http://45.38.161.152/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.152 (name_45.38.161.152)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.152** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox13/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox13/index.rst.txt new file mode 100644 index 0000000000..4c48c2cf2b --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox13/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox13 # Sandbox name Uppercase(case sensitive) + sandbox13 # Sandbox name Lowercase + 50.117.27.84 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1131 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1132 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.144/28 # PUBLIC IPv4 Allocation + 45.38.161.144/30 # PUBLIC LOOPBACK subnet + 45.38.161.145 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.148/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.148/32 # CUSTOMER V-NET SNAT IP + 45.38.161.152/30 # L3LB Subnet + 45.38.161.152/32 # L3LB IP + 45.38.161.156/30 # L4LB Subnet + 45.38.161.157 # Second usable ip address in load-balancer subnet + 45.38.161.158 # Third usable ip address in load-balancer subnet + 45.38.161.162/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.161/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.166/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.165/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcd::/64 # public IPv6 subnet + 2607:f358:11:ffcd::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1b/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1a/127 # isp1-ipv6-example BGP peer remote IPv6 + s13-pre-configured # LINK + s13-learn-by-doing # LINK + s13-e-bgp # LINK + s13-v-net # LINK + s13-nat # LINK + s13-acl # LINK + s13-l3lb # LINK + s13-k8s # LINK + s13-topology # LINK + +Sandbox13 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt new file mode 100644 index 0000000000..b434841d6f --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s13-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox13.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox13.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.157 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.157:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.157 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.157 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.157 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.157 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.157 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.158 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.158 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.158 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.158 + + SRV05-NYC + curl 45.38.161.158 + + SRV05-NYC + curl 45.38.161.158 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1132 + localIP: 45.38.161.166/30 + remoteIP: 45.38.161.165/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.144/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.166/30 45.38.161.165/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.166/30 45.38.161.165/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.166/30 45.38.161.165/30 7m59s + sandbox13-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox13-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox13-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.166/30 45.38.161.165/30 8m41s + sandbox13-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox13-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox13-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.157 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox13/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox13/sandbox-info.rst.txt new file mode 100644 index 0000000000..28fa875f2c --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox13/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s13-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox13.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.84 -p 30061 + srv02-nyc: ssh demo@50.117.27.84 -p 30062 + srv03-nyc: ssh demo@50.117.27.84 -p 30063 + srv04-nyc: ssh demo@50.117.27.84 -p 30064 + srv05-nyc: ssh demo@50.117.27.84 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1131 + Local Address: 45.38.161.162/30 + Remote Address: 45.38.161.161/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.144/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1131 + Local Address: 2607:f358:11:ffc0::1b/127 + Remote Address: 2607:f358:11:ffc0::1a/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcd::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1132 + Local Address: 45.38.161.166/30 + Remote Address: 45.38.161.165/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.144/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.144/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.144/30 + |___ NAT Subnet: 45.38.161.148/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.152/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.156/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcd::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcd::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox14/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox14/configurations.rst.txt new file mode 100644 index 0000000000..24d55bb27f --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox14/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s14-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox14.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffce::1** (from the "**2607:f358:11:ffce::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.85 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox14/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox14/creating-services.rst.txt new file mode 100644 index 0000000000..cb0e23c8d7 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox14/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s14-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s14-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.85 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s14-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1142``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.174``. + 10. In the **Remote IP** field, type in ``45.38.161.173``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.176/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.177)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s14-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.85 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.180/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.180/32**" IP address. + + * This public IP is part of **45.38.161.180/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s14-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.85 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s14-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.184/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.184**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.184**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.184**) into the browser's address bar or simply visit `http://45.38.161.184/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.184 (name_45.38.161.184)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.184** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox14/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox14/index.rst.txt new file mode 100644 index 0000000000..f48f80f4f6 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox14/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox15 # Sandbox name Uppercase(case sensitive) + sandbox15 # Sandbox name Lowercase + 50.117.27.85 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1141 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1142 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.176/28 # PUBLIC IPv4 Allocation + 45.38.161.176/30 # PUBLIC LOOPBACK subnet + 45.38.161.177 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.180/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.180/32 # CUSTOMER V-NET SNAT IP + 45.38.161.184/30 # L3LB Subnet + 45.38.161.184/32 # L3LB IP + 45.38.161.188/30 # L4LB Subnet + 45.38.161.189 # Second usable ip address in load-balancer subnet + 45.38.161.190 # Third usable ip address in load-balancer subnet + 45.38.161.170/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.169/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.174/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.173/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffce::/64 # public IPv6 subnet + 2607:f358:11:ffce::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1d/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1c/127 # isp1-ipv6-example BGP peer remote IPv6 + s14-pre-configured # LINK + s14-learn-by-doing # LINK + s14-e-bgp # LINK + s14-v-net # LINK + s14-nat # LINK + s14-acl # LINK + s14-l3lb # LINK + s14-k8s # LINK + s14-topology # LINK + +Sandbox14 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt new file mode 100644 index 0000000000..58d3847693 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s14-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox15.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox15.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.189 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.189:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.189 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.189 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.189 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.189 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.189 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.190 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.190 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.190 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.190 + + SRV05-NYC + curl 45.38.161.190 + + SRV05-NYC + curl 45.38.161.190 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1142 + localIP: 45.38.161.174/30 + remoteIP: 45.38.161.173/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.176/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.174/30 45.38.161.173/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.174/30 45.38.161.173/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.174/30 45.38.161.173/30 7m59s + sandbox15-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox15-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox15-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.174/30 45.38.161.173/30 8m41s + sandbox15-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox15-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox15-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.189 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox14/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox14/sandbox-info.rst.txt new file mode 100644 index 0000000000..3d1ac9c9de --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox14/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s14-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox14.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.85 -p 30061 + srv02-nyc: ssh demo@50.117.27.85 -p 30062 + srv03-nyc: ssh demo@50.117.27.85 -p 30063 + srv04-nyc: ssh demo@50.117.27.85 -p 30064 + srv05-nyc: ssh demo@50.117.27.85 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1141 + Local Address: 45.38.161.170/30 + Remote Address: 45.38.161.169/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.176/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1141 + Local Address: 2607:f358:11:ffc0::1d/127 + Remote Address: 2607:f358:11:ffc0::1c/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffce::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1142 + Local Address: 45.38.161.174/30 + Remote Address: 45.38.161.173/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.176/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.176/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.176/30 + |___ NAT Subnet: 45.38.161.180/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.184/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.188/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffce::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffce::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox15/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox15/configurations.rst.txt new file mode 100644 index 0000000000..485dfa4586 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox15/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s15-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox15.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcf::1** (from the "**2607:f358:11:ffcf::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.86 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox15/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox15/creating-services.rst.txt new file mode 100644 index 0000000000..89011e4fbd --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox15/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s15-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s15-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.86 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s15-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1152``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.214``. + 10. In the **Remote IP** field, type in ``45.38.161.213``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.192/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.193)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s15-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.86 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.196/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.196/32**" IP address. + + * This public IP is part of **45.38.161.196/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s15-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.86 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s15-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.200/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.200**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.200**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.200**) into the browser's address bar or simply visit `http://45.38.161.200/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.200 (name_45.38.161.200)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.200** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox15/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox15/index.rst.txt new file mode 100644 index 0000000000..1fbb4baf56 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox15/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox15 # Sandbox name Uppercase(case sensitive) + sandbox15 # Sandbox name Lowercase + 50.117.27.86 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1151 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1152 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.192/28 # PUBLIC IPv4 Allocation + 45.38.161.192/30 # PUBLIC LOOPBACK subnet + 45.38.161.193 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.196/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.196/32 # CUSTOMER V-NET SNAT IP + 45.38.161.200/30 # L3LB Subnet + 45.38.161.200/32 # L3LB IP + 45.38.161.204/30 # L4LB Subnet + 45.38.161.205 # Second usable ip address in load-balancer subnet + 45.38.161.206 # Third usable ip address in load-balancer subnet + 45.38.161.210/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.209/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.214/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.213/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcf::/64 # public IPv6 subnet + 2607:f358:11:ffcf::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1f/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1e/127 # isp1-ipv6-example BGP peer remote IPv6 + s15-pre-configured # LINK + s15-learn-by-doing # LINK + s15-e-bgp # LINK + s15-v-net # LINK + s15-nat # LINK + s15-acl # LINK + s15-l3lb # LINK + s15-k8s # LINK + s15-topology # LINK + +Sandbox15 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt new file mode 100644 index 0000000000..c97a6dc012 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s15-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox15.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox15.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.205 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.205:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.205 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.205 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.205 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.205 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.205 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.206 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.206 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.206 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.206 + + SRV05-NYC + curl 45.38.161.206 + + SRV05-NYC + curl 45.38.161.206 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1152 + localIP: 45.38.161.214/30 + remoteIP: 45.38.161.213/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.192/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.214/30 45.38.161.213/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.214/30 45.38.161.213/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.214/30 45.38.161.213/30 7m59s + sandbox15-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox15-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox15-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.214/30 45.38.161.213/30 8m41s + sandbox15-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox15-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox15-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.205 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox15/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox15/sandbox-info.rst.txt new file mode 100644 index 0000000000..207b77248f --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox15/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s15-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox15.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.86 -p 30061 + srv02-nyc: ssh demo@50.117.27.86 -p 30062 + srv03-nyc: ssh demo@50.117.27.86 -p 30063 + srv04-nyc: ssh demo@50.117.27.86 -p 30064 + srv05-nyc: ssh demo@50.117.27.86 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1151 + Local Address: 45.38.161.210/30 + Remote Address: 45.38.161.209/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.192/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1151 + Local Address: 2607:f358:11:ffc0::1f/127 + Remote Address: 2607:f358:11:ffc0::1e/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcf::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1152 + Local Address: 45.38.161.214/30 + Remote Address: 45.38.161.213/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.192/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.192/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.192/30 + |___ NAT Subnet: 45.38.161.196/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.200/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.204/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcf::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcf::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox2/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox2/configurations.rst.txt new file mode 100644 index 0000000000..e133ecfa59 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox2/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s2-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox2.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc2::1** (from the "**2607:f358:11:ffc2::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.190 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox2/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox2/creating-services.rst.txt new file mode 100644 index 0000000000..19412c18f8 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox2/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s2-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s2-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s2-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1022``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.32/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.33)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s2-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.36/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.36/32**" IP address. + + * This public IP is part of **45.38.161.36/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s2-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s2-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.40/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox2/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox2/index.rst.txt new file mode 100644 index 0000000000..b5f4205fd0 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox2/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox2 # Sandbox name Uppercase(case sensitive) + sandbox2 # Sandbox name Lowercase + 166.88.17.190 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1021 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1022 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.32/28 # PUBLIC IPv4 Allocation + 45.38.161.32/30 # PUBLIC LOOPBACK subnet + 45.38.161.33 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.36/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.36/32 # CUSTOMER V-NET SNAT IP + 45.38.161.40/30 # L3LB Subnet + 45.38.161.40/32 # L3LB IP + 45.38.161.44/30 # L4LB Subnet + 45.38.161.45 # Second usable ip address in load-balancer subnet + 45.38.161.46 # Third usable ip address in load-balancer subnet + 45.38.161.26/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.25/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.30/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.29/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc2::/64 # public IPv6 subnet + 2607:f358:11:ffc2::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::5/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::4/127 # isp1-ipv6-example BGP peer remote IPv6 + s2-pre-configured # LINK + s2-learn-by-doing # LINK + s2-e-bgp # LINK + s2-v-net # LINK + s2-nat # LINK + s2-acl # LINK + s2-l3lb # LINK + s2-k8s # LINK + s2-topology # LINK + +Sandbox2 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt new file mode 100644 index 0000000000..9d10d53939 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s2-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox2-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox2.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox2.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox2-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.45 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.45:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox2-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.45 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.45 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.45 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.45 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.45 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.46 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.46 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox2-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.46 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.46 + + SRV05-NYC + curl 45.38.161.46 + + SRV05-NYC + curl 45.38.161.46 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1022 + localIP: 45.38.161.30/30 + remoteIP: 45.38.161.29/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.32/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.30/30 45.38.161.29/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.30/30 45.38.161.29/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.30/30 45.38.161.29/30 7m59s + sandbox2-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox2-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox2-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.30/30 45.38.161.29/30 8m41s + sandbox2-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox2-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox2-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.45 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox2/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox2/sandbox-info.rst.txt new file mode 100644 index 0000000000..0107417bdb --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox2/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s2-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox2.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.190 -p 30061 + srv02-nyc: ssh demo@166.88.17.190 -p 30062 + srv03-nyc: ssh demo@166.88.17.190 -p 30063 + srv04-nyc: ssh demo@166.88.17.190 -p 30064 + srv05-nyc: ssh demo@166.88.17.190 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1021 + Local Address: 45.38.161.26/30 + Remote Address: 45.38.161.25/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.32/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1021 + Local Address: 2607:f358:11:ffc0::5/127 + Remote Address: 2607:f358:11:ffc0::4/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc2::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1022 + Local Address: 45.38.161.30/30 + Remote Address: 45.38.161.29/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.32/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.32/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.32/30 + |___ NAT Subnet: 45.38.161.36/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.40/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.44/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc2::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc2::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox3/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox3/configurations.rst.txt new file mode 100644 index 0000000000..22924fd207 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox3/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s3-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox3.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc3::1** (from the "**2607:f358:11:ffc3::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.189 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox3/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox3/creating-services.rst.txt new file mode 100644 index 0000000000..f8b74af13b --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox3/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s3-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s3-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s3-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1032``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.48/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.49)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s3-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.52/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.52/32**" IP address. + + * This public IP is part of **45.38.161.52/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s3-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s3-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.56/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox3/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox3/index.rst.txt new file mode 100644 index 0000000000..45e3a10c1d --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox3/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox3 # Sandbox name Uppercase(case sensitive) + sandbox3 # Sandbox name Lowercase + 166.88.17.189 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1031 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1032 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.48/28 # PUBLIC IPv4 Allocation + 45.38.161.48/30 # PUBLIC LOOPBACK subnet + 45.38.161.49 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.52/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.52/32 # CUSTOMER V-NET SNAT IP + 45.38.161.56/30 # L3LB Subnet + 45.38.161.56/32 # L3LB IP + 45.38.161.60/30 # L4LB Subnet + 45.38.161.61 # Second usable ip address in load-balancer subnet + 45.38.161.62 # Third usable ip address in load-balancer subnet + 45.38.161.66/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.65/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.70/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.69/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc3::/64 # public IPv6 subnet + 2607:f358:11:ffc3::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::7/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::6/127 # isp1-ipv6-example BGP peer remote IPv6 + s3-pre-configured # LINK + s3-learn-by-doing # LINK + s3-e-bgp # LINK + s3-v-net # LINK + s3-nat # LINK + s3-acl # LINK + s3-l3lb # LINK + s3-k8s # LINK + s3-topology # LINK + +Sandbox3 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt new file mode 100644 index 0000000000..e60fa0aa7f --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s3-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox3.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox3.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.61 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.61:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.61 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.61 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.61 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.61 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.61 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.62 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.62 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.62 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.62 + + SRV05-NYC + curl 45.38.161.62 + + SRV05-NYC + curl 45.38.161.62 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1032 + localIP: 45.38.161.70/30 + remoteIP: 45.38.161.69/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.48/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.70/30 45.38.161.69/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.70/30 45.38.161.69/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.70/30 45.38.161.69/30 7m59s + sandbox3-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox3-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox3-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.70/30 45.38.161.69/30 8m41s + sandbox3-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox3-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox3-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.61 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox3/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox3/sandbox-info.rst.txt new file mode 100644 index 0000000000..43d59d3f2b --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox3/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s3-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox3.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.189 -p 30061 + srv02-nyc: ssh demo@166.88.17.189 -p 30062 + srv03-nyc: ssh demo@166.88.17.189 -p 30063 + srv04-nyc: ssh demo@166.88.17.189 -p 30064 + srv05-nyc: ssh demo@166.88.17.189 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1031 + Local Address: 45.38.161.66/30 + Remote Address: 45.38.161.65/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.48/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1031 + Local Address: 2607:f358:11:ffc0::7/127 + Remote Address: 2607:f358:11:ffc0::6/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc3::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1032 + Local Address: 45.38.161.70/30 + Remote Address: 45.38.161.69/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.48/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.48/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.48/30 + |___ NAT Subnet: 45.38.161.52/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.56/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.60/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc3::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc3::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox4/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox4/configurations.rst.txt new file mode 100644 index 0000000000..5286d7f58b --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox4/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s4-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox4.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc4::1** (from the "**2607:f358:11:ffc4::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.188 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox4/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox4/creating-services.rst.txt new file mode 100644 index 0000000000..bab4c1c92a --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox4/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s4-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s4-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s4-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1042``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.80/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.81)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s4-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.84/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.84/32**" IP address. + + * This public IP is part of **45.38.161.84/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s4-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s4-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.88/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox4/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox4/index.rst.txt new file mode 100644 index 0000000000..066815ccea --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox4/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox4 # Sandbox name Uppercase(case sensitive) + sandbox4 # Sandbox name Lowercase + 166.88.17.188 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1041 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1042 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.80/28 # PUBLIC IPv4 Allocation + 45.38.161.80/30 # PUBLIC LOOPBACK subnet + 45.38.161.81 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.84/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.84/32 # CUSTOMER V-NET SNAT IP + 45.38.161.88/30 # L3LB Subnet + 45.38.161.88/32 # L3LB IP + 45.38.161.92/30 # L4LB Subnet + 45.38.161.93 # Second usable ip address in load-balancer subnet + 45.38.161.94 # Third usable ip address in load-balancer subnet + 45.38.161.74/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.73/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.78/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.77/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc4::/64 # public IPv6 subnet + 2607:f358:11:ffc4::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::9/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::8/127 # isp1-ipv6-example BGP peer remote IPv6 + s4-pre-configured # LINK + s4-learn-by-doing # LINK + s4-e-bgp # LINK + s4-v-net # LINK + s4-nat # LINK + s4-acl # LINK + s4-l3lb # LINK + s4-k8s # LINK + s4-topology # LINK + +Sandbox4 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt new file mode 100644 index 0000000000..addf200aee --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s4-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox4.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox4.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.93 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.93:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.93 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.93 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.93 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.93 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.93 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.94 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.94 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.94 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.94 + + SRV05-NYC + curl 45.38.161.94 + + SRV05-NYC + curl 45.38.161.94 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1042 + localIP: 45.38.161.78/30 + remoteIP: 45.38.161.77/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.80/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.78/30 45.38.161.77/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.78/30 45.38.161.77/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.78/30 45.38.161.77/30 7m59s + sandbox4-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox4-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox4-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.78/30 45.38.161.77/30 8m41s + sandbox4-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox4-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox4-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.93 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox4/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox4/sandbox-info.rst.txt new file mode 100644 index 0000000000..f18d8c16c4 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox4/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s4-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox4.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.188 -p 30061 + srv02-nyc: ssh demo@166.88.17.188 -p 30062 + srv03-nyc: ssh demo@166.88.17.188 -p 30063 + srv04-nyc: ssh demo@166.88.17.188 -p 30064 + srv05-nyc: ssh demo@166.88.17.188 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1041 + Local Address: 45.38.161.74/30 + Remote Address: 45.38.161.73/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.80/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1041 + Local Address: 2607:f358:11:ffc0::9/127 + Remote Address: 2607:f358:11:ffc0::8/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc4::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1042 + Local Address: 45.38.161.78/30 + Remote Address: 45.38.161.77/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.80/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.80/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.80/30 + |___ NAT Subnet: 45.38.161.84/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.88/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.92/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc4::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc4::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox5/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox5/configurations.rst.txt new file mode 100644 index 0000000000..341b2e6c7c --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox5/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s5-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox5.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc5::1** (from the "**2607:f358:11:ffc5::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.187 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox5/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox5/creating-services.rst.txt new file mode 100644 index 0000000000..c6451abd28 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox5/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s5-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s5-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s5-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1052``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.128/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.129)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s5-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.132/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.132/32**" IP address. + + * This public IP is part of **50.117.59.132/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s5-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s5-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.136/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox5/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox5/index.rst.txt new file mode 100644 index 0000000000..fabe3e986e --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox5/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox5 # Sandbox name Uppercase(case sensitive) + sandbox5 # Sandbox name Lowercase + 166.88.17.187 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1051 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1052 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.128/28 # PUBLIC IPv4 Allocation + 50.117.59.128/30 # PUBLIC LOOPBACK subnet + 50.117.59.129 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.132/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.132/32 # CUSTOMER V-NET SNAT IP + 50.117.59.136/30 # L3LB Subnet + 50.117.59.136/32 # L3LB IP + 50.117.59.140/30 # L4LB Subnet + 50.117.59.141 # Second usable ip address in load-balancer subnet + 50.117.59.142 # Third usable ip address in load-balancer subnet + 50.117.59.82/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.81/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.86/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.85/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc5::/64 # public IPv6 subnet + 2607:f358:11:ffc5::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::b/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::a/127 # isp1-ipv6-example BGP peer remote IPv6 + s5-pre-configured # LINK + s5-learn-by-doing # LINK + s5-e-bgp # LINK + s5-v-net # LINK + s5-nat # LINK + s5-acl # LINK + s5-l3lb # LINK + s5-k8s # LINK + s5-topology # LINK + +Sandbox5 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt new file mode 100644 index 0000000000..9c6949c993 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s5-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox5.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox5.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.141 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.141:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.141 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.141 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.141 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.141 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.141 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.142 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.142 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.142 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.142 + + SRV05-NYC + curl 50.117.59.142 + + SRV05-NYC + curl 50.117.59.142 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1052 + localIP: 50.117.59.86/30 + remoteIP: 50.117.59.85/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.128/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.86/30 50.117.59.85/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.86/30 50.117.59.85/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.86/30 50.117.59.85/30 7m59s + sandbox5-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox5-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox5-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.86/30 50.117.59.85/30 8m41s + sandbox5-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox5-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox5-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.141 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox5/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox5/sandbox-info.rst.txt new file mode 100644 index 0000000000..fc919e2692 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox5/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s5-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox5.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.187 -p 30061 + srv02-nyc: ssh demo@166.88.17.187 -p 30062 + srv03-nyc: ssh demo@166.88.17.187 -p 30063 + srv04-nyc: ssh demo@166.88.17.187 -p 30064 + srv05-nyc: ssh demo@166.88.17.187 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1051 + Local Address: 50.117.59.82/30 + Remote Address: 50.117.59.81/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.128/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1051 + Local Address: 2607:f358:11:ffc0::b/127 + Remote Address: 2607:f358:11:ffc0::a/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc5::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1052 + Local Address: 50.117.59.86/30 + Remote Address: 50.117.59.85/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.128/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.128/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.128/30 + |___ NAT Subnet: 50.117.59.132/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.136/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.140/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc5::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc5::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox6/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox6/configurations.rst.txt new file mode 100644 index 0000000000..219e0222f8 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox6/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s6-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.186 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/3.0/_sources/sandbox/Sandbox6/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox6/creating-services.rst.txt new file mode 100644 index 0000000000..c5e13300d8 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox6/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s6-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s6-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s6-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1062``. + 10. In the **Local IP** field, type in ``50.117.59.94`` + 11. In the **Remote IP** field, type in ``50.117.59.93``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.144/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s6-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.150/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s6-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.ai `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/3.0/_sources/sandbox/Sandbox6/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox6/index.rst.txt new file mode 100644 index 0000000000..1481cb647a --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox6/index.rst.txt @@ -0,0 +1,11 @@ +Sandbox6 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt new file mode 100644 index 0000000000..fd23a37c59 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt @@ -0,0 +1,641 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox6 # sandbox name + 166.88.17.186 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1061 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1062 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.144/28 # customer public subnet + 50.117.59.154 # second usable ip address in load-balancer subnet + 50.117.59.155 # third usable ip address in load-balancer subnet + 50.117.59.90/30 # isp1-customer bgp peer local ip + 50.117.59.89/30 # isp1-customer bgp peer remote ip + 50.117.59.94/30 # isp2-customer bgp peer local ip + 50.117.59.93/30 # isp2-customer bgp peer remote ip + 50.117.59.150/32 # customer v-net nat ip + s6-pre-configured # LINKS + s6-learn-by-doing # LINKS + s6-e-bgp # LINKS + s6-v-net # LINKS + s6-nat # LINKS + s6-acl # LINKS + s6-k8s # LINKS + +.. _s6-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox6.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox6.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.154 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.154:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.154 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.155 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.155 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.155 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1062 + localIP: 50.117.59.94/30 + remoteIP: 50.117.59.93/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.144/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.94/30 50.117.59.93/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.94/30 50.117.59.93/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.94/30 50.117.59.93/30 7m59s + sandbox6-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox6-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox6-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.94/30 50.117.59.93/30 8m41s + sandbox6-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox6-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox6-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.154 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox6/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox6/sandbox-info.rst.txt new file mode 100644 index 0000000000..d4cdf947d0 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox6/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + +* **Netris Controller**: A cloud-hosted Netris controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all Netris-operated. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology_n.png + :align: center + +Netris GUI +========== +https://sandbox6.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.186 -p 30061 + srv02: ssh demo@166.88.17.186 -p 30062 + srv03: ssh demo@166.88.17.186 -p 30063 + srv04: ssh demo@166.88.17.186 -p 30064 + srv05: ssh demo@166.88.17.186 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1061 + IP customer: 50.117.59.90/30 + IP Iris: 50.117.59.89/30 + + Neighbor AS: 65007 + Vlan: 1062 + IP customer: 50.117.59.94/30 + IP Iris: 50.117.59.93/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 50.117.59.144/28 + diff --git a/en/3.0/_sources/sandbox/Sandbox7/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox7/configurations.rst.txt new file mode 100644 index 0000000000..9034591c4c --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox7/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s7-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox7.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc7::1** (from the "**2607:f358:11:ffc7::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.185 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox7/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox7/creating-services.rst.txt new file mode 100644 index 0000000000..55caf727af --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox7/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s7-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s7-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s7-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1072``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.160/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.161)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s7-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.164/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.164/32**" IP address. + + * This public IP is part of **50.117.59.164/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s7-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s7-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.168/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox7/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox7/index.rst.txt new file mode 100644 index 0000000000..04275ae820 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox7/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox7 # Sandbox name Uppercase(case sensitive) + sandbox7 # Sandbox name Lowercase + 166.88.17.185 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1071 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1072 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.160/28 # PUBLIC IPv4 Allocation + 50.117.59.160/30 # PUBLIC LOOPBACK subnet + 50.117.59.161 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.164/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.164/32 # CUSTOMER V-NET SNAT IP + 50.117.59.168/30 # L3LB Subnet + 50.117.59.168/32 # L3LB IP + 50.117.59.172/30 # L4LB Subnet + 50.117.59.173 # Second usable ip address in load-balancer subnet + 50.117.59.174 # Third usable ip address in load-balancer subnet + 50.117.59.98/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.97/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.102/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.101/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc7::/64 # public IPv6 subnet + 2607:f358:11:ffc7::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::f/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::e/127 # isp1-ipv6-example BGP peer remote IPv6 + s7-pre-configured # LINK + s7-learn-by-doing # LINK + s7-e-bgp # LINK + s7-v-net # LINK + s7-nat # LINK + s7-acl # LINK + s7-l3lb # LINK + s7-k8s # LINK + s7-topology # LINK + +Sandbox7 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt new file mode 100644 index 0000000000..f94333eab1 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s7-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox7.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox7.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.173 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.173:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.173 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.173 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.173 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.173 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.173 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.174 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.174 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.174 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.174 + + SRV05-NYC + curl 50.117.59.174 + + SRV05-NYC + curl 50.117.59.174 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1072 + localIP: 50.117.59.102/30 + remoteIP: 50.117.59.101/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.160/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.102/30 50.117.59.101/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.102/30 50.117.59.101/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.102/30 50.117.59.101/30 7m59s + sandbox7-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox7-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox7-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.102/30 50.117.59.101/30 8m41s + sandbox7-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox7-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox7-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.173 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox7/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox7/sandbox-info.rst.txt new file mode 100644 index 0000000000..44828ae1f6 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox7/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s7-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox7.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.185 -p 30061 + srv02-nyc: ssh demo@166.88.17.185 -p 30062 + srv03-nyc: ssh demo@166.88.17.185 -p 30063 + srv04-nyc: ssh demo@166.88.17.185 -p 30064 + srv05-nyc: ssh demo@166.88.17.185 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1071 + Local Address: 50.117.59.98/30 + Remote Address: 50.117.59.97/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.160/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1071 + Local Address: 2607:f358:11:ffc0::f/127 + Remote Address: 2607:f358:11:ffc0::e/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc7::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1072 + Local Address: 50.117.59.102/30 + Remote Address: 50.117.59.101/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.160/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.160/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.160/30 + |___ NAT Subnet: 50.117.59.164/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.168/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.172/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc7::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc7::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox8/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox8/configurations.rst.txt new file mode 100644 index 0000000000..43ed2fd843 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox8/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s8-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox8.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc8::1** (from the "**2607:f358:11:ffc8::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.29 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox8/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox8/creating-services.rst.txt new file mode 100644 index 0000000000..47504f6852 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox8/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s8-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s8-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.29 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s8-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1082``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.176/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.177)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s8-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.29 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.180/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.180/32**" IP address. + + * This public IP is part of **50.117.59.180/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s8-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.29 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s8-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.184/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox8/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox8/index.rst.txt new file mode 100644 index 0000000000..53b95ef0d1 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox8/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox8 # Sandbox name Uppercase(case sensitive) + sandbox8 # Sandbox name Lowercase + 166.88.17.29 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1081 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1082 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.176/28 # PUBLIC IPv4 Allocation + 50.117.59.176/30 # PUBLIC LOOPBACK subnet + 50.117.59.177 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.180/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.180/32 # CUSTOMER V-NET SNAT IP + 50.117.59.184/30 # L3LB Subnet + 50.117.59.184/32 # L3LB IP + 50.117.59.188/30 # L4LB Subnet + 50.117.59.189 # Second usable ip address in load-balancer subnet + 50.117.59.190 # Third usable ip address in load-balancer subnet + 50.117.59.106/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.105/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.110/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.109/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc8::/64 # public IPv6 subnet + 2607:f358:11:ffc8::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::11/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::10/127 # isp1-ipv6-example BGP peer remote IPv6 + s8-pre-configured # LINK + s8-learn-by-doing # LINK + s8-e-bgp # LINK + s8-v-net # LINK + s8-nat # LINK + s8-acl # LINK + s8-l3lb # LINK + s8-k8s # LINK + s8-topology # LINK + +Sandbox8 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt new file mode 100644 index 0000000000..5514339fd1 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s8-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox8.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox8.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.189 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.189:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.189 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.189 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.189 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.189 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.189 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.190 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.190 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.190 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.190 + + SRV05-NYC + curl 50.117.59.190 + + SRV05-NYC + curl 50.117.59.190 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1082 + localIP: 50.117.59.110/30 + remoteIP: 50.117.59.109/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.176/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.110/30 50.117.59.109/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.110/30 50.117.59.109/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.110/30 50.117.59.109/30 7m59s + sandbox8-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox8-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox8-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.110/30 50.117.59.109/30 8m41s + sandbox8-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox8-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox8-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.189 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox8/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox8/sandbox-info.rst.txt new file mode 100644 index 0000000000..ef9a9780ee --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox8/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s8-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox8.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.29 -p 30061 + srv02-nyc: ssh demo@166.88.17.29 -p 30062 + srv03-nyc: ssh demo@166.88.17.29 -p 30063 + srv04-nyc: ssh demo@166.88.17.29 -p 30064 + srv05-nyc: ssh demo@166.88.17.29 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1081 + Local Address: 50.117.59.106/30 + Remote Address: 50.117.59.105/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.176/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1081 + Local Address: 2607:f358:11:ffc0::11/127 + Remote Address: 2607:f358:11:ffc0::10/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc8::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1082 + Local Address: 50.117.59.110/30 + Remote Address: 50.117.59.109/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.176/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.176/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.176/30 + |___ NAT Subnet: 50.117.59.180/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.184/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.188/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc8::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc8::/64 + diff --git a/en/3.0/_sources/sandbox/Sandbox9/configurations.rst.txt b/en/3.0/_sources/sandbox/Sandbox9/configurations.rst.txt new file mode 100644 index 0000000000..a7fdf0e168 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox9/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s9-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox9.netris.ai `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc9::1** (from the "**2607:f358:11:ffc9::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric, you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.22 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox9/creating-services.rst.txt b/en/3.0/_sources/sandbox/Sandbox9/creating-services.rst.txt new file mode 100644 index 0000000000..6357f310c4 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox9/creating-services.rst.txt @@ -0,0 +1,201 @@ +.. _s9-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes. + + +.. _s9-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give the **srv05-nyc** server the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.22 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.ai `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Owner** drop-down menu, select "**Demo**". + 5. From the **Sites** drop-down menu, select "**US/NYC**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Port** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s9-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.ai `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1092``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.192/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.193)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s9-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.22 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.ai `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.196/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.196/32**" IP address. + + * This public IP is part of **50.117.59.196/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s9-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.22 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.ai `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s9-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.ai `_ and navigate to **Services → Instances(ROH)**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.200/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.0/_sources/sandbox/Sandbox9/index.rst.txt b/en/3.0/_sources/sandbox/Sandbox9/index.rst.txt new file mode 100644 index 0000000000..351ff318dd --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox9/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox9 # Sandbox name Uppercase(case sensitive) + sandbox9 # Sandbox name Lowercase + 166.88.17.22 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1091 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1092 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.192/28 # PUBLIC IPv4 Allocation + 50.117.59.192/30 # PUBLIC LOOPBACK subnet + 50.117.59.193 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.196/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.196/32 # CUSTOMER V-NET SNAT IP + 50.117.59.200/30 # L3LB Subnet + 50.117.59.200/32 # L3LB IP + 50.117.59.204/30 # L4LB Subnet + 50.117.59.205 # Second usable ip address in load-balancer subnet + 50.117.59.206 # Third usable ip address in load-balancer subnet + 50.117.59.114/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.113/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.118/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.117/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc9::/64 # public IPv6 subnet + 2607:f358:11:ffc9::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::13/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::12/127 # isp1-ipv6-example BGP peer remote IPv6 + s9-pre-configured # LINK + s9-learn-by-doing # LINK + s9-e-bgp # LINK + s9-v-net # LINK + s9-nat # LINK + s9-acl # LINK + s9-l3lb # LINK + s9-k8s # LINK + s9-topology # LINK + +Sandbox9 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.0/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt b/en/3.0/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt new file mode 100644 index 0000000000..50dd46c10f --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt @@ -0,0 +1,608 @@ +.. _s9-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox3-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you’ve successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox9.netris.ai:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox9.netris.ai' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox3-podinfo-prov.png + :align: center + +After provisioning has finished, let’s one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.205 9898:32584/TCP,9999:30365/TCP 9m17s + +Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.205:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let’s switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox3-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.205 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.205 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.205 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.205 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.205 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.206 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.206 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox3-l4lbs.png + :align: center + +VNet Custom Resource +-------------------- + +If you see the same as shown in the previous screenshot, it means you didn’t create "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"` manual. If so, let’s create it from Kubernetes using the VNet custom resource. + +Let’s create our VNet manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +Let’s curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.206 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.206 + + SRV05-NYC + curl 50.117.59.206 + + SRV05-NYC + curl 50.117.59.206 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" VNet as stated in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in VNet manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +After applying the manifest containing "import" annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +BGP Custom Resource +------------------- + +Let’s create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1092 + localIP: 50.117.59.118/30 + remoteIP: 50.117.59.117/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.192/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.118/30 50.117.59.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.118/30 50.117.59.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.118/30 50.117.59.117/30 7m59s + sandbox9-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox9-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox9-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.118/30 50.117.59.117/30 8m41s + sandbox9-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox9-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox9-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.205 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.0/_sources/sandbox/Sandbox9/sandbox-info.rst.txt b/en/3.0/_sources/sandbox/Sandbox9/sandbox-info.rst.txt new file mode 100644 index 0000000000..4eac66ee66 --- /dev/null +++ b/en/3.0/_sources/sandbox/Sandbox9/sandbox-info.rst.txt @@ -0,0 +1,133 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack + +The credentials for the sandbox have been provided to you by email in response to your sandbox request. + +This environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all Netris-operated. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s9-topology: +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +========== +https://sandbox9.netris.ai + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +Accessing Linux servers: + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.22 -p 30061 + srv02-nyc: ssh demo@166.88.17.22 -p 30062 + srv03-nyc: ssh demo@166.88.17.22 -p 30063 + srv04-nyc: ssh demo@166.88.17.22 -p 30064 + srv05-nyc: ssh demo@166.88.17.22 -p 30065 + + +Kubernetes cluster +================== +This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This sandbox provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1091 + Local Address: 50.117.59.114/30 + Remote Address: 50.117.59.113/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.192/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1091 + Local Address: 2607:f358:11:ffc0::13/127 + Remote Address: 2607:f358:11:ffc0::12/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc9::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1092 + Local Address: 50.117.59.118/30 + Remote Address: 50.117.59.117/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.192/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.192/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.192/30 + |___ NAT Subnet: 50.117.59.196/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.200/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.204/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc9::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc9::/64 + diff --git a/en/3.0/_sources/softgate-performance.rst.txt b/en/3.0/_sources/softgate-performance.rst.txt new file mode 100644 index 0000000000..1b1de3a508 --- /dev/null +++ b/en/3.0/_sources/softgate-performance.rst.txt @@ -0,0 +1,15 @@ +.. meta:: + :description: Netris Network Policies & Protocol Configuration + +#################### +SoftGate Performance +#################### + +The following tested results are offered to help properly size the hardware needed for a SoftGate with various types of services: + + +.. csv-table:: SoftGate Performance + :file: tables/softgate-performance.csv + :widths: 30, 10, 15, 15, 10, 10, 10 + :header-rows: 0 + diff --git a/en/3.0/_sources/supported-networks.rst.txt b/en/3.0/_sources/supported-networks.rst.txt new file mode 100644 index 0000000000..88cfe41107 --- /dev/null +++ b/en/3.0/_sources/supported-networks.rst.txt @@ -0,0 +1,43 @@ +.. meta:: + :description: Reference Network Architectures + +############################### +Reference Network Architectures +############################### + + +Unmanaged Switch & Netris SoftGate +--------------------------- + +.. image:: images/slide-1.png + :align: center + :alt: Unmanaged Switch & SoftGate + +-------------------------- + +Unmanaged Switch & SoftGate (HA) +--------------------------------------- + +.. image:: images/slide-2.png + :align: center + :alt: Unmanaged Switch & SoftGate (HA) + +-------------------------- + +Netris Managed Switch & SoftGate (HA) +-------------------------------------------- + +.. image:: images/slide-3.png + :align: center + :alt: Netris Managed Switch & SoftGate small data center (HA) + +-------------------------- + +Netris Managed Switch & SoftGate scalable data center (HA) +-------------------------------------------- + +.. image:: images/slide-4.png + :align: center + :alt: Netris Managed Switch & SoftGate scalable data center (HA) + +-------------------------- diff --git a/en/3.0/_sources/switch-agent-installation.rst.txt b/en/3.0/_sources/switch-agent-installation.rst.txt new file mode 100644 index 0000000000..eb6d03bf6f --- /dev/null +++ b/en/3.0/_sources/switch-agent-installation.rst.txt @@ -0,0 +1,239 @@ +.. _switch-agent-installation: +.. meta:: + :description: Netris Switch Agent Installation + +######################### +Switch Agent Installation +######################### + +****************** +Prerequisite Steps +****************** + +============================ +Nvidia Cumulus Linux Devices +============================ +Requirements: +* Fresh install of Cumulus Linux v. 3.7.(x) - Cumulus 4.X is in the process of validation and will be supported in the next Netris release. + +Configure the OOB Management IP address +*************************************** +Configure internet connectivity via management port. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +Configure Nvidia Cumulus Linux License +************************************** + +.. code-block:: shell-session + + sudo cl-license -i + +Copy/paste the Cumulus Linux license string then press ctrl-d. + +Continue to :ref:`"Install the Netris Agent"` section. + +============================ +Ubuntu SwitchDev Devices +============================ + +.. note:: + + Further installation requires a Console and Internet connectivity via management port! + +1. NOS Uninstall + +Uninstall current NOS using **Uninstall OS** from grub menu: + +.. image:: images/uninstallOS.png + :align: center + +Once the uninstallation is completed, the switch will reboot automatically. + +2. Update ONIE + +Select **Update ONIE** from grub menu: + +.. image:: images/updateONIE.png + :align: center + +In case you don't have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually: + +.. code-block:: shell-session + + onie-discovery-stop + ip addr add dev eth0 + ip route add default via + echo "nameserver " > /etc/resolv.conf + +Update ONIE to the supported version. + +.. note:: + + ONIE image available for Mellanox switches only! + +.. code-block:: shell-session + + onie-self-update http://downloads.netris.ai/onie-updater-x86_64-mlnx_x86-r0 + +3. NOS Install + +Select **Install OS** from grub menu: + +.. image:: images/installOS.png + :align: center + +In case you don't have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually: + +.. code-block:: shell-session + + onie-discovery-stop + ip addr add dev eth0 + ip route add default via + echo "nameserver " > /etc/resolv.conf + +Install Ubuntu-SwitchDev from the Netris custom image: + +.. code-block:: shell-session + + onie-nos-install http://downloads.netris.ai/netris-ubuntu-18.04.1.bin + +Default username/password + +``netris/newNet0ps`` + +Configure the OOB Management IP address +*************************************** +Configure internet connectivity via management port. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +Continue to :ref:`"Install the Netris Agent"` section. + +============================ +EdgeCore SONiC Devices +============================ + +.. note:: + + Further installation requires a Console and Internet connectivity via management port! + +1. NOS Uninstall + +Uninstall current NOS using **Uninstall OS** from grub menu: + +.. image:: images/uninstallOS.png + :align: center + +Once the uninstallation is completed, the switch will reboot automatically. + +2. NOS Install + +Select **Install OS** from grub menu: + +.. image:: images/installOS.png + :align: center + +In case you don't have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually: + +.. code-block:: shell-session + + onie-discovery-stop + ip addr add dev eth0 + ip route add default via + echo "nameserver " > /etc/resolv.conf + +Install EdgeCore SONiC image from the Netris repository: + +.. code-block:: shell-session + + onie-nos-install http://downloads.netris.ai/Edgecore-SONiC_20211125_074752_ec202012_227.bin + +Default username/password + +``admin/YourPaSsWoRd`` + +Configure the OOB Management IP address +*************************************** +Disable Zero Touch Provisioning for time being. + +.. code-block:: shell-session + + ztp disable -y + +.. note:: + This will take some time, please be patient. + +Configure internet connectivity via management port. + +.. code-block:: shell-session + + ip addr add dev eth0 + ip route add default via + echo "nameserver " > /etc/resolv.conf + +Continue to :ref:`"Install the Netris Agent"` section. + +.. _switch-agent-installation-install-the-netris-agent: + +************************ +Install the Netris Agent +************************ + +1. Add the Switch in the controller **Inventory**. Detailed configuration documentation is available here: :ref:`"Adding Switches"` +2. Once the Switch is created in the **Inventory**, click on **three vertical dots (⋮)** on the right side on the Switch and select the **Install Agent** option +3. Copy the agent install command to your clipboard and run the command on the Switch +4. Reboot the Switch when the installation completes + +.. code-block:: shell-session + + sudo reboot + +Once the switch boots up you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and switch color will reflect its health in Net→Topology + +Screenshot: Net→Inventory + +.. image:: images/inventory_heartbeat.png + :align: center diff --git a/en/3.0/_sources/switch-ports.rst.txt b/en/3.0/_sources/switch-ports.rst.txt new file mode 100644 index 0000000000..4113bd2f07 --- /dev/null +++ b/en/3.0/_sources/switch-ports.rst.txt @@ -0,0 +1,46 @@ +.. meta:: + :description: Switch Ports + +============ +Switch Ports +============ + +Switch ports can be directly managed in the **Switch Port** UI section. Both physical and virtual ports (extended, aggregate, etc…) will appear in this section once they have been added to inventory. The Netris Controller will automatically sync the list of available ports that appear on each device. + +The following options are available for editing on each port: + +* Description - Description of the port. +* Tenant - Tenant to whom the port is assigned, by default it is the owner tenant of the device to whom the port belongs to. +* Breakout - Available only for physical switch ports, used to split physical ports into multiple physical ports. When there is a need to use other supported option supported by switch not shown in the dropdown list user must set breakout to "Manual" and configure breakout manually on the switch. For certain platforms some ports need to be disabled to support breakout into other ports, for that option use "Disable" mode of breakout. For Cumulus, after configuration, user must manually restart switchd daemon on the switch via command "systemctl restart switchd". +* MTU - Maximum transmission unit of the port. +* Autoneg - Toggle autonegotiation. Available only for physical ports. +* Speed - Toggle speed. Available only for physical ports. +* Duplex - Toggle duplex. Available only for physical ports. +* Extension - Create extension ports. Available for physical and aggregate ports. +* Extension Name - Name for new extension. +* VLAN Range - VLAN id range for new extension port. + +.. image:: images/edit-port.png + :align: center + :class: with-shadow + +Example: Edit physical port + + +Quick action menu provides following actions for ports (note that Bulk Action also available for multiple ports: + +Edit - Edit the port. +Admin UP/Down - Toggle admin status of the port. +Add to V-net - Add selected port(s) to a V-net. + +.. image:: images/quick-action-port.png + :align: center + :class: with-shadow + + Add to LAG - Add selected ports into a LAG. + +.. image:: images/add-to-lag-port.png + :align: center + :class: with-shadow + +Free Up Port - Detach port from all resources. diff --git a/en/3.0/_sources/terraform-integration.rst.txt b/en/3.0/_sources/terraform-integration.rst.txt new file mode 100644 index 0000000000..e79dc8cdbb --- /dev/null +++ b/en/3.0/_sources/terraform-integration.rst.txt @@ -0,0 +1,317 @@ +.. meta:: + :description: Terraform: Netris provider + +########################## +Terraform: Netris provider +########################## + +Use Netris provider to interact with the many resources supported by Netris. You must configure the provider with the proper credentials before you can use it. +To learn the basics of Terraform using this provider, follow the hands-on `get started tutorials `_ on HashiCorp’s Learn platform. + +.. image:: images/diagrams_terraform.png + :align: center + +When you make changes in the Terraform files and apply them, Terraform automatically decides which part of your configuration is already deployed into Netris controller and what should be added or removed. + + +.. contents:: To create your first Terraform configuration: + :local: + + +Install Terraform +================= + +Download and install the `Terraform `_ + + + +Create a directory for Terraform files +====================================== + +#. Create a directory with any name, for example, ``netris-terraform``. It stores the configuration files and saved states for Terraform and your infrastructure. +#. Create a configuration file with the ``.tf`` extension in this directory, such as ``main.tf``. + + +Configure a provider +==================== + +1. At the beginning of the configuration file, specify the provider settings. + +.. code-block:: + + terraform { + required_providers { + netris = { + source = "netrisai/netris" + version = "1.1.0" + } + } + } + + provider "netris" { + address = "" + login = "" + password = "" + } + +Specify the provider required arguments: + +* ``address`` - This is your Netris-Controller address (http://example.com). This can also be specified with the ``NETRIS_ADDRESS`` environment variable. +* ``login`` - This is your Netris-Controller login. This can also be specified with the ``NETRIS_LOGIN`` environment variable. +* ``password`` - This is your Netris-Controller password. This can also be specified with the ``NETRIS_PASSWORD`` environment variable. + +2. Execute the command ``terraform init`` in the folder with the configuration file. This command initializes the providers specified in the configuration files and lets you work with the provider resources and data sources. + + +Prepare an infrastructure plan +============================== + +By using Netris Terraform Provider, you can create all kinds of resources, such as Sites, IPAMs, Topology, Inventory, etc. +To create a resource, specify a set of required and optional parameters that define the resource properties. Such resource descriptions make up an infrastructure plan. + +Infrastructure provisioning in Netris starts with Site resources. The Netris-Controller comes with the initial site ``Default``. You can use it in your Terraform configuration files by getting its ID with the Terraform `Data source element `_. + +Let's create a separate file for site resource, and get its ID via Terraform Data source element. + +.. code-block:: + + cat << EOF > site.tf + data "netris_site" "default" { + name = "Default" + } + EOF + +Or, you can create a new ``Site`` resource, `here `_ is the detailed documentation with examples. + +Now, when we're clear on ``Site`` resource usage, let's define our IPAM. There are two types of IPAM resources in the Netris-Controller it’s ``Allocation`` and ``Subnet``. +IPAM resources only require ``tenantid`` field, let's get our default Admin tenant ID with the Data source element. + +.. code-block:: + + cat << EOF > tenant.tf + data "netris_tenant" "admin"{ + name = "Admin" + } + EOF + +Then, when we have the ``tenantid``, we can create IPAM resources. + +.. code-block:: + + cat << EOF > ipam.tf + resource "netris_allocation" "my-allocation-mgmt" { + name = "my-allocation-mgmt" + prefix = "192.0.2.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_allocation" "my-allocation-loopback" { + name = "my-allocation-loopback" + prefix = "198.51.100.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_allocation" "my-allocation-common" { + name = "my-allocation-common" + prefix = "203.0.113.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_subnet" "my-subnet-mgmt" { + name = "my-subnet-mgmt" + prefix = "192.0.2.0/24" + tenantid = data.netris_tenant.admin.id + purpose = "management" + defaultgateway = "192.0.2.254" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-mgmt, + ] + } + + resource "netris_subnet" "my-subnet-loopback" { + name = "my-subnet-loopback" + prefix = "198.51.100.0/24" + tenantid = data.netris_tenant.admin.id + purpose = "loopback" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-loopback, + ] + } + + resource "netris_subnet" "my-subnet-common" { + name = "my-subnet-common" + prefix = "203.0.113.0/25" + tenantid = data.netris_tenant.admin.id + purpose = "common" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-common, + ] + } + EOF + +With the command above, we've defined 6 resources, 3 of the type of Allocation, 3 of the type of Subnet, each Subnet resource has a different purpose. +For more details, get familiar with the IPAM `docs `_. + +Now, when we have all the required resources let's define our Inventory. +We're going to create 1 SoftGate, 1 switch and connect them with a link. + +.. code-block:: + + cat << EOF > inventory.tf + resource "netris_softgate" "my-softgate" { + name = "my-softgate" + tenantid = data.netris_tenant.admin.id + siteid = data.netris_site.default.id + description = "Softgate 1" + mainip = "auto" + mgmtip = "auto" + depends_on = [ + netris_subnet.my-subnet-mgmt, + netris_subnet.my-subnet-loopback, + ] + } + + resource "netris_switch" "my-switch" { + name = "my-switch" + tenantid = data.netris_tenant.admin.id + siteid = data.netris_site.default.id + description = "Switch 01" + nos = "cumulus_linux" + asnumber = "auto" + mainip = "auto" + mgmtip = "auto" + portcount = 16 + depends_on = [ + netris_subnet.my-subnet-mgmt, + netris_subnet.my-subnet-loopback, + ] + } + + resource "netris_link" "sg-to-sw" { + ports = [ + "swp1@my-softgate", + "swp16@my-switch" + ] + depends_on = [ + netris_softgate.my-softgate, + netris_switch.my-switch, + ] + } + EOF + +Next, let's define a local L3 network for our servers, suppose we want to connect 3 servers to our switch first 3 ports + +.. code-block:: + + cat << EOF > vnet.tf + resource "netris_vnet" "my-vnet" { + name = "my-vnet" + tenantid = data.netris_tenant.admin.id + state = "active" + sites{ + id = data.netris_site.default.id + gateways { + prefix = "203.0.113.1/25" + } + ports { + name = "swp1@my-switch" + vlanid = 1050 + } + ports { + name = "swp2@my-switch" + vlanid = 1050 + } + ports { + name = "swp3@my-switch" + } + } + depends_on = [ + netris_switch.my-switch, + netris_subnet.my-subnet-common, + ] + } + EOF + +And finally, we have to provide internet connectivity to our fabric, for that we'll define BGP resource. Suppose we're going to connect our ISP cable to the 10th port of our switch, and want to establish the BGP session on our Softgate. + +.. code-block:: + + cat << EOF > bgp.tf + data "netris_port" "swp10_my_switch"{ + name = "swp10@my-switch" + depends_on = [netris_switch.my-switch] + } + + resource "netris_bgp" "my-bgp" { + name = "my-bgp" + siteid = data.netris_site.default.id + hardware = "my-softgate" + neighboras = 23456 + portid = data.netris_port.swp10_my_switch.id + vlanid = 3000 + localip = "172.16.0.2/30" + remoteip = "172.16.0.1/30" + description = "My First BGP" + prefixlistinbound = ["deny 127.0.0.0/8 le 32", "permit 0.0.0.0/0 le 24"] + prefixlistoutbound = ["permit 192.0.2.0/24", "permit 198.51.100.0/24 le 25", "permit 203.0.113.0/24 le 26"] + depends_on = [netris_link.sg-to-sw] + } + EOF + +.. note:: + + For more information about all resources, how to create and manage them in Terraform, see the `provider's documentation `_. + +Now, when we've done with the configuration files, let's check whether they are valid + +.. code-block:: shell-session + + terraform validate + +If the configuration is valid, the following message is returned: + +.. code-block:: shell-session + + Success! The configuration is valid. + + + +Create resources +================ + +1. After preparing and checking the configuration, run the command: + +.. code-block:: shell-session + + terraform plan + +The terminal will display a list of resources with parameters. This is a test step. No resources are created. If there are errors in the configuration, Terraform points them out. + +2. To create resources, run the command: + +.. code-block:: shell-session + + terraform apply + +3. Confirm the resource creation: type ``yes`` in the terminal and press **Enter**. + +Terraform will create all the required resources and the terminal will display the progress. After creation, you can check resource availability and their settings in the Netris-Controller UI. + + +Delete resources +================ + +1. To delete resources created using Terraform: + +Run the command: + +.. code-block:: shell-session + + terraform destroy + +After the command is executed, the terminal will display a list of resources to be deleted. + +2. Type ``yes`` to confirm their deletion and press **Enter**. diff --git a/en/3.0/_sources/topology-management.rst.txt b/en/3.0/_sources/topology-management.rst.txt new file mode 100644 index 0000000000..0de0381821 --- /dev/null +++ b/en/3.0/_sources/topology-management.rst.txt @@ -0,0 +1,154 @@ +.. _topology-management: +.. meta:: + :description: Topology Management + +========= +Inventory +========= +The Inventory section allows you to add/edit/delete network switches and SoftGates. Initial setup of a Netris managed network is a three part process: + +#. Create Inventory Profiles +#. Adding Switches +#. Adding Softgates + +Inventory Profiles +================== +Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/SoftGate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except Netris-defined and user-defined custom flows. Generated rules include: + +* SSH from user defined subnets +* NTP from user defined ntp services +* DNS from user defined DNS servers +* Custom user defined rules + +.. csv-table:: Inventory Profile Fields + :file: tables/inventory-profile-fields.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup). + +.. image:: images/inventory-profile.png + :align: center + :class: with-shadow + +.. _topology-management-adding-switches: + +Adding Switches +=============== +Every switch needs to be added to the Netris Controller inventory. You can add new devices with the following process: + +#. Navigate to **Net→Inventory** +#. Click the **Add** button +#. Fill in the fields as described below +#. Click the **Add** button + + .. csv-table:: Add Inventory Fields - Switch + :file: tables/inventory-add-switch.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** Add a new Switch. + + .. image:: images/add-new-hardware.png + :align: center + :class: with-shadow + +.. note:: Repeat this process to define all your switches. + +.. _topology-management-adding-softgates: + +Adding SoftGates +================ +Every SoftGate node needs to be added to the Netris Controller inventory. To add a SoftGate node: + +#. Navigate to **Net→Topology** +#. Click **Add** +#. Fill in the fields as described below +#. Click the **Add** button + +.. csv-table:: Add Inventory Fields - SoftGate + :file: tables/inventory-add-softgate.csv + :widths: 25, 75 + :header-rows: 0 + +Example: Adding a SoftGate Node to Topology. + +.. image:: images/add-softgate.png + :align: center + :class: with-shadow + +Viewing Inventory +================= + +Inventory Listing shows also Heartbeat and monitoring statuses of each device. + +Heartbeat - Shows the status of device reachability. +Health - Shows number of successful and failed checks on the device. + + .. image:: images/inventory-listing.png + :align: center + :class: with-shadow + +.. note:: You can also add new devices in the Topology view. + +================ +Topology Manager +================ + +The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents will configure the underlying network devices according to this topology dynamically and start watching against potential failures. + +Adding Links +============ + +To define the links in the network: + +#. Right-click on the spine switch +#. Click **Create Link** +#. Select the **From Port** and the **To Port** + +See the example below: + +.. image:: images/create_link.png + :align: center + :class: with-shadow + +.. image:: images/topology_manager.png + :align: center + :class: with-shadow + +Once the links have been defined, the network is automatically configured as long as physical connectivity is in place and Netris Agents can communicate with Netris Controller. + +.. tip:: You can drag/move the units to your desired positions and click “Save positions”. + +Hairpin Links (Nvidia Cumulus only) +=================================== +With Nvidia Cumulus Linux only, we need to loop two ports on spine switches (hairpin cable) in the current release, usually two upstream (higher capacity) ports. We are planning to lift this requirement in the next Netris release (v2.10). + +To define what ports will be used as a hairpin, navigate to Net→Switch Ports, or right-click on the spine switch, click Ports in Net-->Topology. + +Example: Accessing Switch Ports from Net→Topology + +.. image:: images/switch_port.png + :align: center + :class: with-shadow + +For each spine switch, find the two ports that you are going to connect (loop/hairpin) and configure one port as a “hairpin **l2**” and another port as “hairpin **l3**”. The order doesn’t matter. The system needs to know which ports you have dedicated for the hairpin/loop on each spine switch. (do not do this for non-Cumulus switches) +| +Example: Editing Switch Port from Net→Switch Ports. + +.. image:: images/edit_switch_port.png + :align: center + :class: with-shadow + +Example: Setting port types to “hairpin l2” and “hairpin l3”. + +.. image:: images/hairpin.png + :align: center + :class: with-shadow + +Screenshot: Hairpin visualized in Net→Topology + +.. image:: images/hairpin_topology.png + :align: center + :class: with-shadow diff --git a/en/3.0/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt b/en/3.0/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..4ff8166f44 --- /dev/null +++ b/en/3.0/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt @@ -0,0 +1,27 @@ +####################################### +Activating BGP on Equinix Metal Project +####################################### + +Why use BGP with Equinix Metal? +SoftGate nodes are like border routers to your VPC, they are routing traffic between hosts inside your project and the Internet. We are going to establish 2 BGP sessions between SoftGate nodes and Equinix Metal. So there will be 4 BGP sessions total. + +We need these BGP sessions for moving further. In the next chapters we are going to request pools of public IP addresses, that Netris will automatically advertise to Equinix Metal, so inbound traffic “knows” how to reach Load Balancer, NAT, and other services that you will use within your VPC. + +.. image:: /tutorials/images/equinix-metal-bgp-diagram.png + :align: center + +You only need to activate BGP on the Equinix Metal Project. Netris will handle the rest. +In the Equinix Metal web console go to IPs & Networks → BGP then click Activate BGP on This Project. (see below screenshots) + +.. image:: /tutorials/images/equinix-metal-activate-bgp.png + :align: center + +Netris will handle the rest behind the scenes automatically. Netris will enable BGP peering on the Equinix Metal side, Netris will pull the metadata with the BGP info, and will automatically configure FRR (Free Range Routing BGP daemon) on both SoftGate nodes to bring up the BGP sessions up. + +After a few minutes you should see 4 new BGP sessions in your Netris web console under Net → E-BGP. (example screenshot below). + +.. image:: /tutorials/images/equinix-metal-netris-bgp-up.png + :align: center + + +Now your Netris VPC has established BGP sessions with Equinix Metal Project, and you can proceed to the next step. diff --git a/en/3.0/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt b/en/3.0/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt new file mode 100644 index 0000000000..3e865a8da0 --- /dev/null +++ b/en/3.0/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt @@ -0,0 +1,36 @@ +########################################################### +Provisioning Netris SoftGate nodes in Equinix Metal Project +########################################################### + +For SoftGate nodes you can start with two c3.small.x86 or larger servers. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one. + +Request two servers(c3.small.x86) from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned. + +1) At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory + +.. image:: /tutorials/images/softgate-nodes-created-in-equinix.png + :align: center + +2) When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”. + + Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location). + +Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + +.. image:: /tutorials/images/softgate-nodes-recognized-in-netris.png + :align: center + +3) Provision SoftGate nodes. + +Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there. + +.. image:: /tutorials/images/softgate-one-liner-provisioning.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/softgate-green.png + :align: center diff --git a/en/3.0/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt b/en/3.0/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..d41de776a9 --- /dev/null +++ b/en/3.0/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt @@ -0,0 +1,68 @@ +####################################################### +Enabling services (NAT, V-Net, Load Balancer, IP pools) +####################################################### + +Although bare metal servers in Equinix Metal Project get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities. + +Both NAT and on-demand Load Balancer services need public IP addresses. + + +1) Requesting new Public IP address block +========================================= + +Go to Equinix Metal web console and click on IPs & Networks → IPs (see the screenshot below) + +In this example, I’m requesting two IP address blocks, one /30 (4 IPs) for NAT and one /28 (16 IPs) for Load Balancer. + +It’s important to tag IP blocks as “netris”. This is a signal for Netris Controller that this block is intended for Netris VPC. + +You can always request more IP address blocks in the future. Also it is possible to request a large block and then use Netris IPAM for crushing it into smaller blocks. You can read more about Netris IPAM in Netris docs. + +.. image:: /tutorials/images/equinix-metal-request-ip-block.png + :align: center + +Once IP address blocks are provisioned on Equinix Metal Project you should be able to find them automatically replicated in Netris web console under Net → IPAM + +.. image:: /tutorials/images/equinix-metal-netris-ipam-synced.png + :align: center + +You don’t need to worry about advertising them over BGP, Netris will handle that automatically when that makes sense (associated with any service). + + +2) Enable on-demand (elastic) Load Balancer +=========================================== + +To Enable on-demand (elastic) Load Balancer you only need to change the “purpose” field of appropriate IP address block from “common” into “load-balancer” + +Click on the 3 dots menu (in this example of /28 IP address block), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field. + +.. image:: /tutorials/images/netris-enable-elb.png + :align: center + +Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform. + +Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it. + + +3) Enable V-Net +=============== + +V-Net is a service for virtual private networks. You need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources. + +.. image:: /tutorials/images/netris-create-common-subnets.png + :align: center + +4) Enable NAT +============= + +To enable NAT, you need to repurpose a block of IP addresses for NAT. In the below example I’m repurposing the newly requested /30 subnet for NAT. + +.. image:: /tutorials/images/netris-ipam-nat.png + :align: center + +Then You need to create a NAT rule. In the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. In this example I’m enabling SNAT for the entire 10.0.0.0/8 private network, so basically I just want to ensure that VMs that will get IPs from private networks will get outbound Internet access through NAT. You can always have more granular control either through NAT rule or using Services → ACLs. + +.. image:: /tutorials/images/netris-create-nat-rule.png + :align: center + +At this point the minimal configuration of Netris VPC networking is done, next chapters will describe how to consume the VPC, how to request resources and services. diff --git a/en/3.0/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt b/en/3.0/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt new file mode 100644 index 0000000000..421bbab5ee --- /dev/null +++ b/en/3.0/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt @@ -0,0 +1,51 @@ +.. meta:: + :description: Getting Started for Equinix Metal + +######################################## +Equinix Metal API integration enablement +######################################## + + +For each Equinix Metal Project+location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "Equinix Metal" from the dropdown menu. + * - Name + - Type a descriptive name for your Equinix Metal Project+location. + * - Equinix Project ID + - Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID. + * - Equinix Project API key + - Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here. + * - Equinix Location + - Select your equinix location from the dropdown menu. + + +Equinix Metal Project ID + +.. image:: /tutorials/images/equinix-metal-project-id.png + :align: center + + +Equinix Metal Project API key + +.. image:: /tutorials/images/equinix-metal-project-api-keys.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/netris-create-equinix-metal-site.png + :align: center + + diff --git a/en/3.0/_sources/tutorials/getting-started-for-equinix-metal.rst.txt b/en/3.0/_sources/tutorials/getting-started-for-equinix-metal.rst.txt new file mode 100644 index 0000000000..14d5f88ed9 --- /dev/null +++ b/en/3.0/_sources/tutorials/getting-started-for-equinix-metal.rst.txt @@ -0,0 +1,86 @@ +.. meta:: + :description: Getting Started for Equinix Metal + +######################################## +Equinix Metal API integration enablement +######################################## + + +For each Equinix Metal Project+location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "Equinix Metal" from the dropdown menu. + * - Name + - Type a descriptive name for your Equinix Metal Project+location. + * - Equinix Project ID + - Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID. + * - Equinix Project API key + - Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here. + * - Equinix Location + - Select your equinix location from the dropdown menu. + + +Equinix Metal Project ID + +.. image:: /tutorials/images/equinix-metal-project-id.png + :align: center + + +Equinix Metal Project API key + +.. image:: /tutorials/images/equinix-metal-project-api-keys.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/netris-create-equinix-metal-site.png + :align: center + + +Adding Netris SoftGate nodes +============================ + +For SoftGate nodes you can start with two servers of the smallest flavor. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one. + +Request two servers from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned. + +1) At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory + +.. image:: /tutorials/images/softgate-nodes-created-in-equinix.png + :align: center + +2) When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”. + + Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location). + +Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + +.. image:: /tutorials/images/softgate-nodes-recognized-in-netris.png + :align: center + +3) Provision SoftGate nodes. + +Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there. + +.. image:: /tutorials/images/softgate-one-liner-provisioning.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/softgate-green.png + :align: center diff --git a/en/3.0/_sources/tutorials/index.rst.txt b/en/3.0/_sources/tutorials/index.rst.txt new file mode 100644 index 0000000000..159ded0a0b --- /dev/null +++ b/en/3.0/_sources/tutorials/index.rst.txt @@ -0,0 +1,31 @@ +======================== +Generic Netris Tutorials +======================== + + +.. toctree:: + :maxdepth: 2 + + installing-netris-controller + upgrading-netris + +====================================== +Netris VPC for Equinix Metal Tutorials +====================================== + + +.. toctree:: + :maxdepth: 2 + :caption: Enabling Netris VPC for Equinix Metal Project + + equinix-metal-api-integration-enablement + adding-netris-softgate-nodes-in-equinix-metal + activating-bgp-on-equinix-metal-project + enable-services-on-equinix-metal-project + +.. toctree:: + :maxdepth: 2 + :caption: Consuming Netris VPC services for Equinix Metal Project + + using-vnet-in-equinix-metal-project + using-l4-load-balancer diff --git a/en/3.0/_sources/tutorials/installing-netris-controller.rst.txt b/en/3.0/_sources/tutorials/installing-netris-controller.rst.txt new file mode 100644 index 0000000000..91c8b95e58 --- /dev/null +++ b/en/3.0/_sources/tutorials/installing-netris-controller.rst.txt @@ -0,0 +1,37 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Installing a Netris Controller +============================== + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment. + +It doesn’t matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. 1) So you can access the web console. 2) Nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface. + +Linux Host requirements + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +In this example my host has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +I’m using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my host : 54.219.211.71. + +.. image:: images/cloudflare-dns-record.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. For more details, get familiar with the controller installation `doc `_. + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + +Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller. diff --git a/en/3.0/_sources/tutorials/upgrading-netris.rst.txt b/en/3.0/_sources/tutorials/upgrading-netris.rst.txt new file mode 100644 index 0000000000..5f6286752c --- /dev/null +++ b/en/3.0/_sources/tutorials/upgrading-netris.rst.txt @@ -0,0 +1,193 @@ +.. meta:: + :description: Upgrading Netris + +.. raw:: html + + + + +.. role:: green + +.. role:: red + +************************************** +Netris Upgrade and Rollback Procedures +************************************** + +Upgrade Procedure +================= + +Due to potential database structural changes between Netris versions, it's highly recommended to take a backup of the database before upgrading. The backup will be used in the unlikely event of the need to perform a rollback. + +1. To create a database backup, run the following command by first SSHing the Controller: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot.sql + +Ensure that SQL file ``db-snapshot.sql`` is generated and present in the current directory. + +2. Stop all Netris agents on devices managed by the controller (switch & SoftGate). + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sg + +Ensure that all devices in the *Net → Inventory* section are ":red:`red`" with the "**check_agent**" status being "**Agent is unavailable**". + +.. note:: + + A stopped Netris agent has no impact on production traffic through the device. + +.. _upgrade 3: + +3. Before upgrading the Netris Controller, take a note of the "*Netris Version*" by navigating to *Setting → General* in the Controller web interface. This version number will be used in case of the need to perform a rollback procedure. + +.. image:: /tutorials/images/netris_version_example.png + :align: center + :alt: Netris Version Example + +4. Start the upgrade of the Netris Controller using the one-liner after SSHing to the Controller. + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh - + +.. note:: + + This process can take up to 5 minutes + + +Afterwards, make sure that all pods have either "*Running*" or "*Completed*" status by executing the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller get pods + + +The output is similar to this: + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + svclb-netris-controller-haproxy-6tkgj 4/4 Running 0 38d + netris-controller-haproxy-bcb944b7c-qcbf8 1/1 Running 0 13d + netris-controller-squid-7f6fdc6cf9-7fdx8 1/1 Running 0 38d + svclb-netris-controller-squid-58rnp 1/1 Running 0 38d + netris-controller-graphite-0 1/1 Running 0 38d + netris-controller-mongodb-0 1/1 Running 0 38d + netris-controller-redis-master-0 1/1 Running 0 38d + netris-controller-smtp-76778cf85f-lw5v5 1/1 Running 0 10d + netris-controller-mariadb-0 1/1 Running 0 10d + netris-controller-web-session-generator-8b9dbbcd8-8snhd 1/1 Running 0 10d + netris-controller-telescope-notifier-647975848f-fs5dn 1/1 Running 0 10d + netris-controller-app-b9b8d8f8d-4ssqb 1/1 Running 0 10d + netris-controller-grpc-987669fb9-jjskp 1/1 Running 0 10d + netris-controller-telescope-777c98c5d9-mqwl6 1/1 Running 0 10d + helm-install-netris-controller-lqmq7 0/1 Completed 0 20h + + +.. warning:: + + If, after 5 minutes, you see pods with a status other than "*Running*" or "*Completed*", please reach out to us via `Slack `__. + +Then verify that the "*Netris Version*" reflects the version change by navigating to *Setting → General* in the Controller web interface. + +5. Once you have verified that the Netris controller is up-to-date, it is time to update the switch and SoftGate agents. + +Upgrade the switch & SoftGate agents by copying the one-liner from the "*Install Agent*" option of the device's 3-dot menu found under the *Net → Inventory* section and pasting it after SSHing to the corresponding device. + +.. image:: /tutorials/images/install_agent.gif + :align: center + :alt: Install Agent + +After all the agents have finished the upgrade process, make sure all devices in the *Net → Inventory* section have a ":green:`green`" status and the *Netris version* for each device reflects the version change. + +In the event the "**check_agent**" status is "**Agent is unavailable**" after the agent upgrade has finished, perform agent restart on the affected device(s). + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sg + +Rollback Procedure +================== + +A rollback procedure can be executed in the event the upgrade introduced any adverse impact on the production traffic. + +1. Stop all Netris agents on the devices managed by the controller (switch & SoftGate). + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sg + +2. Restore the database from the previously taken snapshot. + +Copy the backup file from the controller host system to the MariaDB container by running the following command after SSHing to the Controller: + +.. code-block:: shell-session + + kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql + +While still connected to the Controller, restore the database: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql' + +3. Downgrade Netris Controller application with the following command. + +.. note:: + + For the version number, use the number collected from :ref:`step #3` during the upgrade procedure. + +Example: + +.. code-block:: shell-session + + curl -sfL https://get.netris.ai | sh -s -- --ctl-version 3.0.10-031 + +Afterwards, verify that the version of the "*Netris Version*" reflects the downgraded version by navigating to *Setting → General* in the Netris Controller. + +4. Once you have verified that the Netris controller has been downgraded to the correct version, it is time to downgrade the switch and SoftGate agents. + +Install the correct and appropriate versions of the switch & SoftGate agents simply by copying the one-liner from the "*Install Agent*" option of the device's 3-dot menu found under the *Net → Inventory* section and pasting it after SSHing to the corresponding device. + +After all the switches and SoftGates have been successfully downgraded, make sure all the devices in the *Net → Inventory* section have a ":green:`green`" status and the *Netris version* for each device reflects the version downgrade. + +In case the "**check_agent**" status is "**Agent is unavailable**" after agent downgrade, perform agent restart. + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sg diff --git a/en/3.0/_sources/tutorials/using-l4-load-balancer.rst.txt b/en/3.0/_sources/tutorials/using-l4-load-balancer.rst.txt new file mode 100644 index 0000000000..40b7477c34 --- /dev/null +++ b/en/3.0/_sources/tutorials/using-l4-load-balancer.rst.txt @@ -0,0 +1,13 @@ +################################################## +Using on-demand (elastic) L4 Load Balancer service +################################################## + +Services --> L4 Load Balancer is an on-demand (elastic) server load balancer. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console. + +.. image:: /tutorials/images/netris-l4-load-balancer.png + :align: center + diff --git a/en/3.0/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt b/en/3.0/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..6a230c129d --- /dev/null +++ b/en/3.0/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt @@ -0,0 +1,19 @@ +######################################################################## +Using V-Net (isolated virtual network) services in Equinix Metal Project +######################################################################## + +V-Net, under Services --> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from Equinix Metal Project into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network. + +Netris V-Net (in Equinix Metal scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Layer-2 network using Equinix Metal API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Layer-2 network. SoftGate nodes are the default gateway for the V-Net services. + +You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, Equinix, and your Compute. + +.. image:: /tutorials/images/netris-creating-vnet-for-equinix-metal.png + :align: center + +In this example, the new V-NET has VLAN ID 2, subnet 10.0.0.0/24, and gateway 10.0.0.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 2, and they should use IP addresses from 10.0.0.2-254 pointing to 10.0.0.1 as the default gateway. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 2 will have Internet access over the NAT. + +.. image:: /tutorials/images/netris-vnet-ready-in-equinix-metal.png + :align: center + +Note that you can use Services --> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other. remote sites) diff --git a/en/3.0/_sources/visibility.rst.txt b/en/3.0/_sources/visibility.rst.txt new file mode 100644 index 0000000000..cba565de9d --- /dev/null +++ b/en/3.0/_sources/visibility.rst.txt @@ -0,0 +1,84 @@ +.. meta:: + :description: Netris System Visibility, Monitoring & Telemetry + +********************** +Visibility (Telescope) +********************** + +Graph Boards +================= +You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view. + +To start with Graph Boards, first, you need to add a new Graph Board. + +1. Navigate to Telescope→Graph Boards, open the dropdown menu in the top left corner, then click +Add board. + +.. image:: images/telescope.png + :align: center + +2. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants. + +.. image:: images/createboard.png + :align: center + +Now you can add graphs by clicking +Add graph. + +Description of +Add graph fields: + +- **Title** - Title for the new graph. +- **Type** - Type of data source. + + - Bps - Traffic bits per second. + - Pps - Traffic packets per second. + - Errors - Errors per second. + - Optical - Optical signal statistics/history. + - MAC Count - History of the number of MAC addresses on the port. +- **Function** - Currently, only summing is supported. +- **+Member** - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port. + +Example: Sum of traffic on two ISP(Iris1 + Iris2) links. + +.. image:: images/ISP_Iris.png + :align: center + +Example: Sum of the traffic on all ports under the service called “my V-NET” + +.. image:: images/V_NET.png + :align: center + +Screenshot: Listing of a Graph Board with the explanation of the controls. + +.. image:: images/graphboard.png + :align: center + +API Logs +======== +Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type. + +Dashboard +========= +Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems. + +Telescope→Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner. + +Description of the pie charts. + +* **Hardware Health** - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions. +* **E-BGP** - Statuses of external BGP sessions. +* **LB VIP** - Statuses of Load Balancer frontend / VIP availability. +* **LB Members** - Statuses of Load Balancer backend members. + +By clicking on each title you can see the details of the checks on the right side. + +Screenshot: Dashboard showing details of “Hardware Health.” + +.. image:: images/hardware_health.png + :align: center + +Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state. + +Screenshot: “Save as normal” on selected ports. + +.. image:: images/saveasnormal.png + :align: center + diff --git a/en/3.0/_sources/vnet.rst.txt b/en/3.0/_sources/vnet.rst.txt new file mode 100644 index 0000000000..eec88d9176 --- /dev/null +++ b/en/3.0/_sources/vnet.rst.txt @@ -0,0 +1,52 @@ +.. meta:: + :description: V-Net + +.. _v-net_def: + +##### +V-Net +##### +V-Net is a virtual networking service that provide a Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or they can be created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization). +Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes. + + +V-Net Fields +============ + +- **Name** - Unique name for the V-Net +- **Owner** - Tenant, who can make any changes to current V-Net +- **V-Net state** - Active/Disable state for entire V-Net +- **VLAN aware** - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible +- **Guest tenants** - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters +- **Sites** - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites). +- **IPv4 Gateway** - IPv4 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site where V-Net is intended to span. +- **IPv6 Gateway** - IPv6 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site or sites where V-Net is intended to span. +- **Port** - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports. + + - **Enabled** - Enable or disable individual Switch Port under current V-Net + - **Port Name** - Switch Port format: (swp)@ + - **VLAN ID / Untag** - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port’s ingress/egress unless VLAN aware mode is used. + - **LAG Mode** - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes). + +.. tip:: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +.. image:: images/add-vnet.png + :align: center + :class: with-shadow + +.. centered:: + Example: Adding a new V-Net. + +.. image:: images/list-vnet.png + :align: center + :class: with-shadow + +.. centered:: + Example: Listing of V-Nets. + +.. image:: images/list-vnet-expanded.png + :align: center + :class: with-shadow + +.. centered:: + Example: Expanded view of a V-Net listing. \ No newline at end of file diff --git a/en/3.0/_static/_sphinx_javascript_frameworks_compat.js b/en/3.0/_static/_sphinx_javascript_frameworks_compat.js new file mode 100644 index 0000000000..8549469dc2 --- /dev/null +++ b/en/3.0/_static/_sphinx_javascript_frameworks_compat.js @@ -0,0 +1,134 @@ +/* + * _sphinx_javascript_frameworks_compat.js + * ~~~~~~~~~~ + * + * Compatability shim for jQuery and underscores.js. + * + * WILL BE REMOVED IN Sphinx 6.0 + * xref RemovedInSphinx60Warning + * + */ + +/** + * select a different prefix for underscore + */ +$u = _.noConflict(); + + +/** + * small helper function to urldecode strings + * + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL + */ +jQuery.urldecode = function(x) { + if (!x) { + return x + } + return decodeURIComponent(x.replace(/\+/g, ' ')); +}; + +/** + * small helper function to urlencode strings + */ +jQuery.urlencode = encodeURIComponent; + +/** + * This function returns the parsed url parameters of the + * current request. Multiple values per key are supported, + * it will always return arrays of strings for the value parts. + */ +jQuery.getQueryParameters = function(s) { + if (typeof s === 'undefined') + s = document.location.search; + var parts = s.substr(s.indexOf('?') + 1).split('&'); + var result = {}; + for (var i = 0; i < parts.length; i++) { + var tmp = parts[i].split('=', 2); + var key = jQuery.urldecode(tmp[0]); + var value = jQuery.urldecode(tmp[1]); + if (key in result) + result[key].push(value); + else + result[key] = [value]; + } + return result; +}; + +/** + * highlight a given string on a jquery object by wrapping it in + * span elements with the given class name. + */ +jQuery.fn.highlightText = function(text, className) { + function highlight(node, addItems) { + if (node.nodeType === 3) { + var val = node.nodeValue; + var pos = val.toLowerCase().indexOf(text); + if (pos >= 0 && + !jQuery(node.parentNode).hasClass(className) && + !jQuery(node.parentNode).hasClass("nohighlight")) { + var span; + var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.className = className; + } + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + node.parentNode.insertBefore(span, node.parentNode.insertBefore( + document.createTextNode(val.substr(pos + text.length)), + node.nextSibling)); + node.nodeValue = val.substr(0, pos); + if (isInSVG) { + var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect"); + var bbox = node.parentElement.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute('class', className); + addItems.push({ + "parent": node.parentNode, + "target": rect}); + } + } + } + else if (!jQuery(node).is("button, select, textarea")) { + jQuery.each(node.childNodes, function() { + highlight(this, addItems); + }); + } + } + var addItems = []; + var result = this.each(function() { + highlight(this, addItems); + }); + for (var i = 0; i < addItems.length; ++i) { + jQuery(addItems[i].parent).before(addItems[i].target); + } + return result; +}; + +/* + * backward compatibility for jQuery.browser + * This will be supported until firefox bug is fixed. + */ +if (!jQuery.browser) { + jQuery.uaMatch = function(ua) { + ua = ua.toLowerCase(); + + var match = /(chrome)[ \/]([\w.]+)/.exec(ua) || + /(webkit)[ \/]([\w.]+)/.exec(ua) || + /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) || + /(msie) ([\w.]+)/.exec(ua) || + ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) || + []; + + return { + browser: match[ 1 ] || "", + version: match[ 2 ] || "0" + }; + }; + jQuery.browser = {}; + jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true; +} diff --git a/en/3.0/_static/basic.css b/en/3.0/_static/basic.css new file mode 100644 index 0000000000..eeb0519a69 --- /dev/null +++ b/en/3.0/_static/basic.css @@ -0,0 +1,899 @@ +/* + * basic.css + * ~~~~~~~~~ + * + * Sphinx stylesheet -- basic theme. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +div.section::after { + display: block; + content: ''; + clear: left; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; + word-wrap: break-word; + overflow-wrap : break-word; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar #searchbox form.search { + overflow: hidden; +} + +div.sphinxsidebar #searchbox input[type="text"] { + float: left; + width: 80%; + padding: 0.25em; + box-sizing: border-box; +} + +div.sphinxsidebar #searchbox input[type="submit"] { + float: left; + width: 20%; + border-left: none; + padding: 0.25em; + box-sizing: border-box; +} + + +img { + border: 0; + max-width: 100%; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin: 10px 0 0 20px; + padding: 0; +} + +ul.search li { + padding: 5px 0 5px 20px; + background-image: url(file.png); + background-repeat: no-repeat; + background-position: 0 7px; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li p.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; + margin-left: auto; + margin-right: auto; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable ul { + margin-top: 0; + margin-bottom: 0; + list-style-type: none; +} + +table.indextable > tbody > tr > td > ul { + padding-left: 0em; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- domain module index --------------------------------------------------- */ + +table.modindextable td { + padding: 2px; + border-collapse: collapse; +} + +/* -- general body styles --------------------------------------------------- */ + +div.body { + min-width: 360px; + max-width: 800px; +} + +div.body p, div.body dd, div.body li, div.body blockquote { + -moz-hyphens: auto; + -ms-hyphens: auto; + -webkit-hyphens: auto; + hyphens: auto; +} + +a.headerlink { + visibility: hidden; +} +a.brackets:before, +span.brackets > a:before{ + content: "["; +} + +a.brackets:after, +span.brackets > a:after { + content: "]"; +} + + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { + visibility: visible; +} + +div.body p.caption { + text-align: inherit; +} + +div.body td { + text-align: left; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +img.align-left, figure.align-left, .figure.align-left, object.align-left { + clear: left; + float: left; + margin-right: 1em; +} + +img.align-right, figure.align-right, .figure.align-right, object.align-right { + clear: right; + float: right; + margin-left: 1em; +} + +img.align-center, figure.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +img.align-default, figure.align-default, .figure.align-default { + display: block; + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left; +} + +.align-center { + text-align: center; +} + +.align-default { + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar, +aside.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px; + background-color: #ffe; + width: 40%; + float: right; + clear: right; + overflow-x: auto; +} + +p.sidebar-title { + font-weight: bold; +} +div.admonition, div.topic, blockquote { + clear: left; +} + +/* -- topics ---------------------------------------------------------------- */ +div.topic { + border: 1px solid #ccc; + padding: 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- content of sidebars/topics/admonitions -------------------------------- */ + +div.sidebar > :last-child, +aside.sidebar > :last-child, +div.topic > :last-child, +div.admonition > :last-child { + margin-bottom: 0; +} + +div.sidebar::after, +aside.sidebar::after, +div.topic::after, +div.admonition::after, +blockquote::after { + display: block; + content: ''; + clear: both; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + margin-top: 10px; + margin-bottom: 10px; + border: 0; + border-collapse: collapse; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +table.align-default { + margin-left: auto; + margin-right: auto; +} + +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +th > :first-child, +td > :first-child { + margin-top: 0px; +} + +th > :last-child, +td > :last-child { + margin-bottom: 0px; +} + +/* -- figures --------------------------------------------------------------- */ + +div.figure, figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption, figcaption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number, +figcaption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text, +figcaption span.caption-text { +} + +/* -- field list styles ----------------------------------------------------- */ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.field-name { + -moz-hyphens: manual; + -ms-hyphens: manual; + -webkit-hyphens: manual; + hyphens: manual; +} + +/* -- hlist styles ---------------------------------------------------------- */ + +table.hlist { + margin: 1em 0; +} + +table.hlist td { + vertical-align: top; +} + +/* -- object description styles --------------------------------------------- */ + +.sig { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; +} + +.sig-name, code.descname { + background-color: transparent; + font-weight: bold; +} + +.sig-name { + font-size: 1.1em; +} + +code.descname { + font-size: 1.2em; +} + +.sig-prename, code.descclassname { + background-color: transparent; +} + +.optional { + font-size: 1.3em; +} + +.sig-paren { + font-size: larger; +} + +.sig-param.n { + font-style: italic; +} + +/* C++ specific styling */ + +.sig-inline.c-texpr, +.sig-inline.cpp-texpr { + font-family: unset; +} + +.sig.c .k, .sig.c .kt, +.sig.cpp .k, .sig.cpp .kt { + color: #0033B3; +} + +.sig.c .m, +.sig.cpp .m { + color: #1750EB; +} + +.sig.c .s, .sig.c .sc, +.sig.cpp .s, .sig.cpp .sc { + color: #067D17; +} + + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +:not(li) > ol > li:first-child > :first-child, +:not(li) > ul > li:first-child > :first-child { + margin-top: 0px; +} + +:not(li) > ol > li:last-child > :last-child, +:not(li) > ul > li:last-child > :last-child { + margin-bottom: 0px; +} + +ol.simple ol p, +ol.simple ul p, +ul.simple ol p, +ul.simple ul p { + margin-top: 0; +} + +ol.simple > li:not(:first-child) > p, +ul.simple > li:not(:first-child) > p { + margin-top: 0; +} + +ol.simple p, +ul.simple p { + margin-bottom: 0; +} +dl.footnote > dt, +dl.citation > dt { + float: left; + margin-right: 0.5em; +} + +dl.footnote > dd, +dl.citation > dd { + margin-bottom: 0em; +} + +dl.footnote > dd:after, +dl.citation > dd:after { + content: ""; + clear: both; +} + +dl.field-list { + display: grid; + grid-template-columns: fit-content(30%) auto; +} + +dl.field-list > dt { + font-weight: bold; + word-break: break-word; + padding-left: 0.5em; + padding-right: 5px; +} +dl.field-list > dt:after { + content: ":"; +} + + +dl.field-list > dd { + padding-left: 0.5em; + margin-top: 0em; + margin-left: 0em; + margin-bottom: 0em; +} + +dl { + margin-bottom: 15px; +} + +dd > :first-child { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +dl > dd:last-child, +dl > dd:last-child > :last-child { + margin-bottom: 0; +} + +dt:target, span.highlighted { + background-color: #fbe54e; +} + +rect.highlighted { + fill: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa; +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +.classifier:before { + font-style: normal; + margin: 0 0.5em; + content: ":"; + display: inline-block; +} + +abbr, acronym { + border-bottom: dotted 1px; + cursor: help; +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; + overflow-y: hidden; /* fixes display issues on Chrome browsers */ +} + +pre, div[class*="highlight-"] { + clear: both; +} + +span.pre { + -moz-hyphens: none; + -ms-hyphens: none; + -webkit-hyphens: none; + hyphens: none; + white-space: nowrap; +} + +div[class*="highlight-"] { + margin: 1em 0; +} + +td.linenos pre { + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + display: block; +} + +table.highlighttable tbody { + display: block; +} + +table.highlighttable tr { + display: flex; +} + +table.highlighttable td { + margin: 0; + padding: 0; +} + +table.highlighttable td.linenos { + padding-right: 0.5em; +} + +table.highlighttable td.code { + flex: 1; + overflow: hidden; +} + +.highlight .hll { + display: block; +} + +div.highlight pre, +table.highlighttable pre { + margin: 0; +} + +div.code-block-caption + div { + margin-top: 0; +} + +div.code-block-caption { + margin-top: 1em; + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +table.highlighttable td.linenos, +span.linenos, +div.highlight span.gp { /* gp: Generic.Prompt */ + user-select: none; + -webkit-user-select: text; /* Safari fallback only */ + -webkit-user-select: none; /* Chrome/Safari */ + -moz-user-select: none; /* Firefox */ + -ms-user-select: none; /* IE10+ */ +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + margin: 1em 0; +} + +code.xref, a code { + background-color: transparent; + font-weight: bold; +} + +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.body div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +span.eqno a.headerlink { + position: absolute; + z-index: 1; +} + +div.math:hover a.headerlink { + visibility: visible; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} \ No newline at end of file diff --git a/en/3.0/_static/check-solid.svg b/en/3.0/_static/check-solid.svg new file mode 100644 index 0000000000..92fad4b5c0 --- /dev/null +++ b/en/3.0/_static/check-solid.svg @@ -0,0 +1,4 @@ + + + + diff --git a/en/3.0/_static/clipboard.min.js b/en/3.0/_static/clipboard.min.js new file mode 100644 index 0000000000..54b3c46381 --- /dev/null +++ b/en/3.0/_static/clipboard.min.js @@ -0,0 +1,7 @@ +/*! + * clipboard.js v2.0.8 + * https://clipboardjs.com/ + * + * Licensed MIT © Zeno Rocha + */ +!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1 + + + + diff --git a/en/3.0/_static/copybutton.css b/en/3.0/_static/copybutton.css new file mode 100644 index 0000000000..f1916ec7d1 --- /dev/null +++ b/en/3.0/_static/copybutton.css @@ -0,0 +1,94 @@ +/* Copy buttons */ +button.copybtn { + position: absolute; + display: flex; + top: .3em; + right: .3em; + width: 1.7em; + height: 1.7em; + opacity: 0; + transition: opacity 0.3s, border .3s, background-color .3s; + user-select: none; + padding: 0; + border: none; + outline: none; + border-radius: 0.4em; + /* The colors that GitHub uses */ + border: #1b1f2426 1px solid; + background-color: #f6f8fa; + color: #57606a; +} + +button.copybtn.success { + border-color: #22863a; + color: #22863a; +} + +button.copybtn svg { + stroke: currentColor; + width: 1.5em; + height: 1.5em; + padding: 0.1em; +} + +div.highlight { + position: relative; +} + +/* Show the copybutton */ +.highlight:hover button.copybtn, button.copybtn.success { + opacity: 1; +} + +.highlight button.copybtn:hover { + background-color: rgb(235, 235, 235); +} + +.highlight button.copybtn:active { + background-color: rgb(187, 187, 187); +} + +/** + * A minimal CSS-only tooltip copied from: + * https://codepen.io/mildrenben/pen/rVBrpK + * + * To use, write HTML like the following: + * + *

Short

+ */ + .o-tooltip--left { + position: relative; + } + + .o-tooltip--left:after { + opacity: 0; + visibility: hidden; + position: absolute; + content: attr(data-tooltip); + padding: .2em; + font-size: .8em; + left: -.2em; + background: grey; + color: white; + white-space: nowrap; + z-index: 2; + border-radius: 2px; + transform: translateX(-102%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); +} + +.o-tooltip--left:hover:after { + display: block; + opacity: 1; + visibility: visible; + transform: translateX(-100%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); + transition-delay: .5s; +} + +/* By default the copy button shouldn't show up when printing a page */ +@media print { + button.copybtn { + display: none; + } +} diff --git a/en/3.0/_static/copybutton.js b/en/3.0/_static/copybutton.js new file mode 100644 index 0000000000..2ea7ff3e21 --- /dev/null +++ b/en/3.0/_static/copybutton.js @@ -0,0 +1,248 @@ +// Localization support +const messages = { + 'en': { + 'copy': 'Copy', + 'copy_to_clipboard': 'Copy to clipboard', + 'copy_success': 'Copied!', + 'copy_failure': 'Failed to copy', + }, + 'es' : { + 'copy': 'Copiar', + 'copy_to_clipboard': 'Copiar al portapapeles', + 'copy_success': '¡Copiado!', + 'copy_failure': 'Error al copiar', + }, + 'de' : { + 'copy': 'Kopieren', + 'copy_to_clipboard': 'In die Zwischenablage kopieren', + 'copy_success': 'Kopiert!', + 'copy_failure': 'Fehler beim Kopieren', + }, + 'fr' : { + 'copy': 'Copier', + 'copy_to_clipboard': 'Copier dans le presse-papier', + 'copy_success': 'Copié !', + 'copy_failure': 'Échec de la copie', + }, + 'ru': { + 'copy': 'Скопировать', + 'copy_to_clipboard': 'Скопировать в буфер', + 'copy_success': 'Скопировано!', + 'copy_failure': 'Не удалось скопировать', + }, + 'zh-CN': { + 'copy': '复制', + 'copy_to_clipboard': '复制到剪贴板', + 'copy_success': '复制成功!', + 'copy_failure': '复制失败', + }, + 'it' : { + 'copy': 'Copiare', + 'copy_to_clipboard': 'Copiato negli appunti', + 'copy_success': 'Copiato!', + 'copy_failure': 'Errore durante la copia', + } +} + +let locale = 'en' +if( document.documentElement.lang !== undefined + && messages[document.documentElement.lang] !== undefined ) { + locale = document.documentElement.lang +} + +let doc_url_root = DOCUMENTATION_OPTIONS.URL_ROOT; +if (doc_url_root == '#') { + doc_url_root = ''; +} + +/** + * SVG files for our copy buttons + */ +let iconCheck = ` + ${messages[locale]['copy_success']} + + +` + +// If the user specified their own SVG use that, otherwise use the default +let iconCopy = ``; +if (!iconCopy) { + iconCopy = ` + ${messages[locale]['copy_to_clipboard']} + + + +` +} + +/** + * Set up copy/paste for code blocks + */ + +const runWhenDOMLoaded = cb => { + if (document.readyState != 'loading') { + cb() + } else if (document.addEventListener) { + document.addEventListener('DOMContentLoaded', cb) + } else { + document.attachEvent('onreadystatechange', function() { + if (document.readyState == 'complete') cb() + }) + } +} + +const codeCellId = index => `codecell${index}` + +// Clears selected text since ClipboardJS will select the text when copying +const clearSelection = () => { + if (window.getSelection) { + window.getSelection().removeAllRanges() + } else if (document.selection) { + document.selection.empty() + } +} + +// Changes tooltip text for a moment, then changes it back +// We want the timeout of our `success` class to be a bit shorter than the +// tooltip and icon change, so that we can hide the icon before changing back. +var timeoutIcon = 2000; +var timeoutSuccessClass = 1500; + +const temporarilyChangeTooltip = (el, oldText, newText) => { + el.setAttribute('data-tooltip', newText) + el.classList.add('success') + // Remove success a little bit sooner than we change the tooltip + // So that we can use CSS to hide the copybutton first + setTimeout(() => el.classList.remove('success'), timeoutSuccessClass) + setTimeout(() => el.setAttribute('data-tooltip', oldText), timeoutIcon) +} + +// Changes the copy button icon for two seconds, then changes it back +const temporarilyChangeIcon = (el) => { + el.innerHTML = iconCheck; + setTimeout(() => {el.innerHTML = iconCopy}, timeoutIcon) +} + +const addCopyButtonToCodeCells = () => { + // If ClipboardJS hasn't loaded, wait a bit and try again. This + // happens because we load ClipboardJS asynchronously. + if (window.ClipboardJS === undefined) { + setTimeout(addCopyButtonToCodeCells, 250) + return + } + + // Add copybuttons to all of our code cells + const COPYBUTTON_SELECTOR = 'div.highlight pre'; + const codeCells = document.querySelectorAll(COPYBUTTON_SELECTOR) + codeCells.forEach((codeCell, index) => { + const id = codeCellId(index) + codeCell.setAttribute('id', id) + + const clipboardButton = id => + `` + codeCell.insertAdjacentHTML('afterend', clipboardButton(id)) + }) + +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} + + +var copyTargetText = (trigger) => { + var target = document.querySelector(trigger.attributes['data-clipboard-target'].value); + + // get filtered text + let exclude = '.linenos'; + + let text = filterText(target, exclude); + return formatCopyText(text, '', false, true, true, true, '', '') +} + + // Initialize with a callback so we can modify the text before copy + const clipboard = new ClipboardJS('.copybtn', {text: copyTargetText}) + + // Update UI with error/success messages + clipboard.on('success', event => { + clearSelection() + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_success']) + temporarilyChangeIcon(event.trigger) + }) + + clipboard.on('error', event => { + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_failure']) + }) +} + +runWhenDOMLoaded(addCopyButtonToCodeCells) \ No newline at end of file diff --git a/en/3.0/_static/copybutton_funcs.js b/en/3.0/_static/copybutton_funcs.js new file mode 100644 index 0000000000..dbe1aaad79 --- /dev/null +++ b/en/3.0/_static/copybutton_funcs.js @@ -0,0 +1,73 @@ +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +export function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +export function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} diff --git a/en/3.0/_static/css/badge_only.css b/en/3.0/_static/css/badge_only.css new file mode 100644 index 0000000000..e380325bc6 --- /dev/null +++ b/en/3.0/_static/css/badge_only.css @@ -0,0 +1 @@ +.fa:before{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-style:normal;font-weight:400;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#FontAwesome) format("svg")}.fa:before{font-family:FontAwesome;font-style:normal;font-weight:400;line-height:1}.fa:before,a .fa{text-decoration:inherit}.fa:before,a .fa,li .fa{display:inline-block}li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-.8em}ul.fas li .fa{width:.8em}ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before,.icon-book:before{content:"\f02d"}.fa-caret-down:before,.icon-caret-down:before{content:"\f0d7"}.fa-caret-up:before,.icon-caret-up:before{content:"\f0d8"}.fa-caret-left:before,.icon-caret-left:before{content:"\f0d9"}.fa-caret-right:before,.icon-caret-right:before{content:"\f0da"}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60}.rst-versions .rst-current-version:after{clear:both;content:"";display:block}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}} \ No newline at end of file diff --git a/en/3.0/_static/css/fonts/Roboto-Slab-Bold.woff b/en/3.0/_static/css/fonts/Roboto-Slab-Bold.woff new file mode 100644 index 0000000000..6cb6000018 Binary files /dev/null and b/en/3.0/_static/css/fonts/Roboto-Slab-Bold.woff differ diff --git a/en/3.0/_static/css/fonts/Roboto-Slab-Bold.woff2 b/en/3.0/_static/css/fonts/Roboto-Slab-Bold.woff2 new file mode 100644 index 0000000000..7059e23142 Binary files /dev/null and b/en/3.0/_static/css/fonts/Roboto-Slab-Bold.woff2 differ diff --git a/en/3.0/_static/css/fonts/Roboto-Slab-Regular.woff b/en/3.0/_static/css/fonts/Roboto-Slab-Regular.woff new file mode 100644 index 0000000000..f815f63f99 Binary files /dev/null and b/en/3.0/_static/css/fonts/Roboto-Slab-Regular.woff differ diff --git a/en/3.0/_static/css/fonts/Roboto-Slab-Regular.woff2 b/en/3.0/_static/css/fonts/Roboto-Slab-Regular.woff2 new file mode 100644 index 0000000000..f2c76e5bda Binary files /dev/null and b/en/3.0/_static/css/fonts/Roboto-Slab-Regular.woff2 differ diff --git a/en/3.0/_static/css/fonts/fontawesome-webfont.eot b/en/3.0/_static/css/fonts/fontawesome-webfont.eot new file mode 100644 index 0000000000..e9f60ca953 Binary files /dev/null and b/en/3.0/_static/css/fonts/fontawesome-webfont.eot differ diff --git a/en/3.0/_static/css/fonts/fontawesome-webfont.svg b/en/3.0/_static/css/fonts/fontawesome-webfont.svg new file mode 100644 index 0000000000..855c845e53 --- /dev/null +++ b/en/3.0/_static/css/fonts/fontawesome-webfont.svg @@ -0,0 +1,2671 @@ + + + + +Created by FontForge 20120731 at Mon Oct 24 17:37:40 2016 + By ,,, +Copyright Dave Gandy 2016. All rights reserved. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/en/3.0/_static/css/fonts/fontawesome-webfont.ttf b/en/3.0/_static/css/fonts/fontawesome-webfont.ttf new file mode 100644 index 0000000000..35acda2fa1 Binary files /dev/null and b/en/3.0/_static/css/fonts/fontawesome-webfont.ttf differ diff --git a/en/3.0/_static/css/fonts/fontawesome-webfont.woff b/en/3.0/_static/css/fonts/fontawesome-webfont.woff new file mode 100644 index 0000000000..400014a4b0 Binary files /dev/null and b/en/3.0/_static/css/fonts/fontawesome-webfont.woff differ diff --git a/en/3.0/_static/css/fonts/fontawesome-webfont.woff2 b/en/3.0/_static/css/fonts/fontawesome-webfont.woff2 new file mode 100644 index 0000000000..4d13fc6040 Binary files /dev/null and b/en/3.0/_static/css/fonts/fontawesome-webfont.woff2 differ diff --git a/en/3.0/_static/css/fonts/lato-bold-italic.woff b/en/3.0/_static/css/fonts/lato-bold-italic.woff new file mode 100644 index 0000000000..88ad05b9ff Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-bold-italic.woff differ diff --git a/en/3.0/_static/css/fonts/lato-bold-italic.woff2 b/en/3.0/_static/css/fonts/lato-bold-italic.woff2 new file mode 100644 index 0000000000..c4e3d804b5 Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-bold-italic.woff2 differ diff --git a/en/3.0/_static/css/fonts/lato-bold.woff b/en/3.0/_static/css/fonts/lato-bold.woff new file mode 100644 index 0000000000..c6dff51f06 Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-bold.woff differ diff --git a/en/3.0/_static/css/fonts/lato-bold.woff2 b/en/3.0/_static/css/fonts/lato-bold.woff2 new file mode 100644 index 0000000000..bb195043cf Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-bold.woff2 differ diff --git a/en/3.0/_static/css/fonts/lato-normal-italic.woff b/en/3.0/_static/css/fonts/lato-normal-italic.woff new file mode 100644 index 0000000000..76114bc033 Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-normal-italic.woff differ diff --git a/en/3.0/_static/css/fonts/lato-normal-italic.woff2 b/en/3.0/_static/css/fonts/lato-normal-italic.woff2 new file mode 100644 index 0000000000..3404f37e2e Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-normal-italic.woff2 differ diff --git a/en/3.0/_static/css/fonts/lato-normal.woff b/en/3.0/_static/css/fonts/lato-normal.woff new file mode 100644 index 0000000000..ae1307ff5f Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-normal.woff differ diff --git a/en/3.0/_static/css/fonts/lato-normal.woff2 b/en/3.0/_static/css/fonts/lato-normal.woff2 new file mode 100644 index 0000000000..3bf9843328 Binary files /dev/null and b/en/3.0/_static/css/fonts/lato-normal.woff2 differ diff --git a/en/3.0/_static/css/theme.css b/en/3.0/_static/css/theme.css new file mode 100644 index 0000000000..8cd4f101a9 --- /dev/null +++ b/en/3.0/_static/css/theme.css @@ -0,0 +1,4 @@ +html{box-sizing:border-box}*,:after,:before{box-sizing:inherit}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}[hidden],audio:not([controls]){display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:active,a:hover{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:700}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;text-decoration:none}ins,mark{color:#000}mark{background:#ff0;font-style:italic;font-weight:700}.rst-content code,.rst-content tt,code,kbd,pre,samp{font-family:monospace,serif;_font-family:courier new,monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:after,q:before{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sup{top:-.5em}sub{bottom:-.25em}dl,ol,ul{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure,form{margin:0}label{cursor:pointer}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type=button],input[type=reset],input[type=submit]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type=search]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}textarea{resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:.2em 0;background:#ccc;color:#000;padding:.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none!important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{body,html,section{background:none!important}*{box-shadow:none!important;text-shadow:none!important;filter:none!important;-ms-filter:none!important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="#"]:after,a[href^="javascript:"]:after{content:""}blockquote,pre{page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}img{max-width:100%!important}@page{margin:.5cm}.rst-content .toctree-wrapper>p.caption,h2,h3,p{orphans:3;widows:3}.rst-content .toctree-wrapper>p.caption,h2,h3{page-break-after:avoid}}.btn,.fa:before,.icon:before,.rst-content .admonition,.rst-content .admonition-title:before,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .code-block-caption .headerlink:before,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-alert,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before,.wy-nav-top a,.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a,input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week],select,textarea{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}/*! + * Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */@font-face{font-family:FontAwesome;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713);src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix&v=4.7.0) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#fontawesomeregular) format("svg");font-weight:400;font-style:normal}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14286em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14286em;width:2.14286em;top:.14286em;text-align:center}.fa-li.fa-lg{left:-1.85714em}.fa-border{padding:.2em .25em .15em;border:.08em solid #eee;border-radius:.1em}.fa-pull-left{float:left}.fa-pull-right{float:right}.fa-pull-left.icon,.fa.fa-pull-left,.rst-content .code-block-caption .fa-pull-left.headerlink,.rst-content .fa-pull-left.admonition-title,.rst-content code.download span.fa-pull-left:first-child,.rst-content dl dt .fa-pull-left.headerlink,.rst-content h1 .fa-pull-left.headerlink,.rst-content h2 .fa-pull-left.headerlink,.rst-content h3 .fa-pull-left.headerlink,.rst-content h4 .fa-pull-left.headerlink,.rst-content h5 .fa-pull-left.headerlink,.rst-content h6 .fa-pull-left.headerlink,.rst-content p.caption .fa-pull-left.headerlink,.rst-content table>caption .fa-pull-left.headerlink,.rst-content tt.download span.fa-pull-left:first-child,.wy-menu-vertical li.current>a span.fa-pull-left.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-left.toctree-expand,.wy-menu-vertical li span.fa-pull-left.toctree-expand{margin-right:.3em}.fa-pull-right.icon,.fa.fa-pull-right,.rst-content .code-block-caption .fa-pull-right.headerlink,.rst-content .fa-pull-right.admonition-title,.rst-content code.download span.fa-pull-right:first-child,.rst-content dl dt .fa-pull-right.headerlink,.rst-content h1 .fa-pull-right.headerlink,.rst-content h2 .fa-pull-right.headerlink,.rst-content h3 .fa-pull-right.headerlink,.rst-content h4 .fa-pull-right.headerlink,.rst-content h5 .fa-pull-right.headerlink,.rst-content h6 .fa-pull-right.headerlink,.rst-content p.caption .fa-pull-right.headerlink,.rst-content table>caption .fa-pull-right.headerlink,.rst-content tt.download span.fa-pull-right:first-child,.wy-menu-vertical li.current>a span.fa-pull-right.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-right.toctree-expand,.wy-menu-vertical li span.fa-pull-right.toctree-expand{margin-left:.3em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left,.pull-left.icon,.rst-content .code-block-caption .pull-left.headerlink,.rst-content .pull-left.admonition-title,.rst-content code.download span.pull-left:first-child,.rst-content dl dt .pull-left.headerlink,.rst-content h1 .pull-left.headerlink,.rst-content h2 .pull-left.headerlink,.rst-content h3 .pull-left.headerlink,.rst-content h4 .pull-left.headerlink,.rst-content h5 .pull-left.headerlink,.rst-content h6 .pull-left.headerlink,.rst-content p.caption .pull-left.headerlink,.rst-content table>caption .pull-left.headerlink,.rst-content tt.download span.pull-left:first-child,.wy-menu-vertical li.current>a span.pull-left.toctree-expand,.wy-menu-vertical li.on a span.pull-left.toctree-expand,.wy-menu-vertical li span.pull-left.toctree-expand{margin-right:.3em}.fa.pull-right,.pull-right.icon,.rst-content .code-block-caption .pull-right.headerlink,.rst-content .pull-right.admonition-title,.rst-content code.download span.pull-right:first-child,.rst-content dl dt .pull-right.headerlink,.rst-content h1 .pull-right.headerlink,.rst-content h2 .pull-right.headerlink,.rst-content h3 .pull-right.headerlink,.rst-content h4 .pull-right.headerlink,.rst-content h5 .pull-right.headerlink,.rst-content h6 .pull-right.headerlink,.rst-content p.caption .pull-right.headerlink,.rst-content table>caption .pull-right.headerlink,.rst-content tt.download span.pull-right:first-child,.wy-menu-vertical li.current>a span.pull-right.toctree-expand,.wy-menu-vertical li.on a span.pull-right.toctree-expand,.wy-menu-vertical li span.pull-right.toctree-expand{margin-left:.3em}.fa-spin{-webkit-animation:fa-spin 2s linear infinite;animation:fa-spin 2s linear infinite}.fa-pulse{-webkit-animation:fa-spin 1s steps(8) infinite;animation:fa-spin 1s steps(8) infinite}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";-webkit-transform:scaleX(-1);-ms-transform:scaleX(-1);transform:scaleX(-1)}.fa-flip-vertical{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";-webkit-transform:scaleY(-1);-ms-transform:scaleY(-1);transform:scaleY(-1)}:root .fa-flip-horizontal,:root .fa-flip-vertical,:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:""}.fa-music:before{content:""}.fa-search:before,.icon-search:before{content:""}.fa-envelope-o:before{content:""}.fa-heart:before{content:""}.fa-star:before{content:""}.fa-star-o:before{content:""}.fa-user:before{content:""}.fa-film:before{content:""}.fa-th-large:before{content:""}.fa-th:before{content:""}.fa-th-list:before{content:""}.fa-check:before{content:""}.fa-close:before,.fa-remove:before,.fa-times:before{content:""}.fa-search-plus:before{content:""}.fa-search-minus:before{content:""}.fa-power-off:before{content:""}.fa-signal:before{content:""}.fa-cog:before,.fa-gear:before{content:""}.fa-trash-o:before{content:""}.fa-home:before,.icon-home:before{content:""}.fa-file-o:before{content:""}.fa-clock-o:before{content:""}.fa-road:before{content:""}.fa-download:before,.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{content:""}.fa-arrow-circle-o-down:before{content:""}.fa-arrow-circle-o-up:before{content:""}.fa-inbox:before{content:""}.fa-play-circle-o:before{content:""}.fa-repeat:before,.fa-rotate-right:before{content:""}.fa-refresh:before{content:""}.fa-list-alt:before{content:""}.fa-lock:before{content:""}.fa-flag:before{content:""}.fa-headphones:before{content:""}.fa-volume-off:before{content:""}.fa-volume-down:before{content:""}.fa-volume-up:before{content:""}.fa-qrcode:before{content:""}.fa-barcode:before{content:""}.fa-tag:before{content:""}.fa-tags:before{content:""}.fa-book:before,.icon-book:before{content:""}.fa-bookmark:before{content:""}.fa-print:before{content:""}.fa-camera:before{content:""}.fa-font:before{content:""}.fa-bold:before{content:""}.fa-italic:before{content:""}.fa-text-height:before{content:""}.fa-text-width:before{content:""}.fa-align-left:before{content:""}.fa-align-center:before{content:""}.fa-align-right:before{content:""}.fa-align-justify:before{content:""}.fa-list:before{content:""}.fa-dedent:before,.fa-outdent:before{content:""}.fa-indent:before{content:""}.fa-video-camera:before{content:""}.fa-image:before,.fa-photo:before,.fa-picture-o:before{content:""}.fa-pencil:before{content:""}.fa-map-marker:before{content:""}.fa-adjust:before{content:""}.fa-tint:before{content:""}.fa-edit:before,.fa-pencil-square-o:before{content:""}.fa-share-square-o:before{content:""}.fa-check-square-o:before{content:""}.fa-arrows:before{content:""}.fa-step-backward:before{content:""}.fa-fast-backward:before{content:""}.fa-backward:before{content:""}.fa-play:before{content:""}.fa-pause:before{content:""}.fa-stop:before{content:""}.fa-forward:before{content:""}.fa-fast-forward:before{content:""}.fa-step-forward:before{content:""}.fa-eject:before{content:""}.fa-chevron-left:before{content:""}.fa-chevron-right:before{content:""}.fa-plus-circle:before{content:""}.fa-minus-circle:before{content:""}.fa-times-circle:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before{content:""}.fa-check-circle:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before{content:""}.fa-question-circle:before{content:""}.fa-info-circle:before{content:""}.fa-crosshairs:before{content:""}.fa-times-circle-o:before{content:""}.fa-check-circle-o:before{content:""}.fa-ban:before{content:""}.fa-arrow-left:before{content:""}.fa-arrow-right:before{content:""}.fa-arrow-up:before{content:""}.fa-arrow-down:before{content:""}.fa-mail-forward:before,.fa-share:before{content:""}.fa-expand:before{content:""}.fa-compress:before{content:""}.fa-plus:before{content:""}.fa-minus:before{content:""}.fa-asterisk:before{content:""}.fa-exclamation-circle:before,.rst-content .admonition-title:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before{content:""}.fa-gift:before{content:""}.fa-leaf:before{content:""}.fa-fire:before,.icon-fire:before{content:""}.fa-eye:before{content:""}.fa-eye-slash:before{content:""}.fa-exclamation-triangle:before,.fa-warning:before{content:""}.fa-plane:before{content:""}.fa-calendar:before{content:""}.fa-random:before{content:""}.fa-comment:before{content:""}.fa-magnet:before{content:""}.fa-chevron-up:before{content:""}.fa-chevron-down:before{content:""}.fa-retweet:before{content:""}.fa-shopping-cart:before{content:""}.fa-folder:before{content:""}.fa-folder-open:before{content:""}.fa-arrows-v:before{content:""}.fa-arrows-h:before{content:""}.fa-bar-chart-o:before,.fa-bar-chart:before{content:""}.fa-twitter-square:before{content:""}.fa-facebook-square:before{content:""}.fa-camera-retro:before{content:""}.fa-key:before{content:""}.fa-cogs:before,.fa-gears:before{content:""}.fa-comments:before{content:""}.fa-thumbs-o-up:before{content:""}.fa-thumbs-o-down:before{content:""}.fa-star-half:before{content:""}.fa-heart-o:before{content:""}.fa-sign-out:before{content:""}.fa-linkedin-square:before{content:""}.fa-thumb-tack:before{content:""}.fa-external-link:before{content:""}.fa-sign-in:before{content:""}.fa-trophy:before{content:""}.fa-github-square:before{content:""}.fa-upload:before{content:""}.fa-lemon-o:before{content:""}.fa-phone:before{content:""}.fa-square-o:before{content:""}.fa-bookmark-o:before{content:""}.fa-phone-square:before{content:""}.fa-twitter:before{content:""}.fa-facebook-f:before,.fa-facebook:before{content:""}.fa-github:before,.icon-github:before{content:""}.fa-unlock:before{content:""}.fa-credit-card:before{content:""}.fa-feed:before,.fa-rss:before{content:""}.fa-hdd-o:before{content:""}.fa-bullhorn:before{content:""}.fa-bell:before{content:""}.fa-certificate:before{content:""}.fa-hand-o-right:before{content:""}.fa-hand-o-left:before{content:""}.fa-hand-o-up:before{content:""}.fa-hand-o-down:before{content:""}.fa-arrow-circle-left:before,.icon-circle-arrow-left:before{content:""}.fa-arrow-circle-right:before,.icon-circle-arrow-right:before{content:""}.fa-arrow-circle-up:before{content:""}.fa-arrow-circle-down:before{content:""}.fa-globe:before{content:""}.fa-wrench:before{content:""}.fa-tasks:before{content:""}.fa-filter:before{content:""}.fa-briefcase:before{content:""}.fa-arrows-alt:before{content:""}.fa-group:before,.fa-users:before{content:""}.fa-chain:before,.fa-link:before,.icon-link:before{content:""}.fa-cloud:before{content:""}.fa-flask:before{content:""}.fa-cut:before,.fa-scissors:before{content:""}.fa-copy:before,.fa-files-o:before{content:""}.fa-paperclip:before{content:""}.fa-floppy-o:before,.fa-save:before{content:""}.fa-square:before{content:""}.fa-bars:before,.fa-navicon:before,.fa-reorder:before{content:""}.fa-list-ul:before{content:""}.fa-list-ol:before{content:""}.fa-strikethrough:before{content:""}.fa-underline:before{content:""}.fa-table:before{content:""}.fa-magic:before{content:""}.fa-truck:before{content:""}.fa-pinterest:before{content:""}.fa-pinterest-square:before{content:""}.fa-google-plus-square:before{content:""}.fa-google-plus:before{content:""}.fa-money:before{content:""}.fa-caret-down:before,.icon-caret-down:before,.wy-dropdown .caret:before{content:""}.fa-caret-up:before{content:""}.fa-caret-left:before{content:""}.fa-caret-right:before{content:""}.fa-columns:before{content:""}.fa-sort:before,.fa-unsorted:before{content:""}.fa-sort-desc:before,.fa-sort-down:before{content:""}.fa-sort-asc:before,.fa-sort-up:before{content:""}.fa-envelope:before{content:""}.fa-linkedin:before{content:""}.fa-rotate-left:before,.fa-undo:before{content:""}.fa-gavel:before,.fa-legal:before{content:""}.fa-dashboard:before,.fa-tachometer:before{content:""}.fa-comment-o:before{content:""}.fa-comments-o:before{content:""}.fa-bolt:before,.fa-flash:before{content:""}.fa-sitemap:before{content:""}.fa-umbrella:before{content:""}.fa-clipboard:before,.fa-paste:before{content:""}.fa-lightbulb-o:before{content:""}.fa-exchange:before{content:""}.fa-cloud-download:before{content:""}.fa-cloud-upload:before{content:""}.fa-user-md:before{content:""}.fa-stethoscope:before{content:""}.fa-suitcase:before{content:""}.fa-bell-o:before{content:""}.fa-coffee:before{content:""}.fa-cutlery:before{content:""}.fa-file-text-o:before{content:""}.fa-building-o:before{content:""}.fa-hospital-o:before{content:""}.fa-ambulance:before{content:""}.fa-medkit:before{content:""}.fa-fighter-jet:before{content:""}.fa-beer:before{content:""}.fa-h-square:before{content:""}.fa-plus-square:before{content:""}.fa-angle-double-left:before{content:""}.fa-angle-double-right:before{content:""}.fa-angle-double-up:before{content:""}.fa-angle-double-down:before{content:""}.fa-angle-left:before{content:""}.fa-angle-right:before{content:""}.fa-angle-up:before{content:""}.fa-angle-down:before{content:""}.fa-desktop:before{content:""}.fa-laptop:before{content:""}.fa-tablet:before{content:""}.fa-mobile-phone:before,.fa-mobile:before{content:""}.fa-circle-o:before{content:""}.fa-quote-left:before{content:""}.fa-quote-right:before{content:""}.fa-spinner:before{content:""}.fa-circle:before{content:""}.fa-mail-reply:before,.fa-reply:before{content:""}.fa-github-alt:before{content:""}.fa-folder-o:before{content:""}.fa-folder-open-o:before{content:""}.fa-smile-o:before{content:""}.fa-frown-o:before{content:""}.fa-meh-o:before{content:""}.fa-gamepad:before{content:""}.fa-keyboard-o:before{content:""}.fa-flag-o:before{content:""}.fa-flag-checkered:before{content:""}.fa-terminal:before{content:""}.fa-code:before{content:""}.fa-mail-reply-all:before,.fa-reply-all:before{content:""}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:""}.fa-location-arrow:before{content:""}.fa-crop:before{content:""}.fa-code-fork:before{content:""}.fa-chain-broken:before,.fa-unlink:before{content:""}.fa-question:before{content:""}.fa-info:before{content:""}.fa-exclamation:before{content:""}.fa-superscript:before{content:""}.fa-subscript:before{content:""}.fa-eraser:before{content:""}.fa-puzzle-piece:before{content:""}.fa-microphone:before{content:""}.fa-microphone-slash:before{content:""}.fa-shield:before{content:""}.fa-calendar-o:before{content:""}.fa-fire-extinguisher:before{content:""}.fa-rocket:before{content:""}.fa-maxcdn:before{content:""}.fa-chevron-circle-left:before{content:""}.fa-chevron-circle-right:before{content:""}.fa-chevron-circle-up:before{content:""}.fa-chevron-circle-down:before{content:""}.fa-html5:before{content:""}.fa-css3:before{content:""}.fa-anchor:before{content:""}.fa-unlock-alt:before{content:""}.fa-bullseye:before{content:""}.fa-ellipsis-h:before{content:""}.fa-ellipsis-v:before{content:""}.fa-rss-square:before{content:""}.fa-play-circle:before{content:""}.fa-ticket:before{content:""}.fa-minus-square:before{content:""}.fa-minus-square-o:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before{content:""}.fa-level-up:before{content:""}.fa-level-down:before{content:""}.fa-check-square:before{content:""}.fa-pencil-square:before{content:""}.fa-external-link-square:before{content:""}.fa-share-square:before{content:""}.fa-compass:before{content:""}.fa-caret-square-o-down:before,.fa-toggle-down:before{content:""}.fa-caret-square-o-up:before,.fa-toggle-up:before{content:""}.fa-caret-square-o-right:before,.fa-toggle-right:before{content:""}.fa-eur:before,.fa-euro:before{content:""}.fa-gbp:before{content:""}.fa-dollar:before,.fa-usd:before{content:""}.fa-inr:before,.fa-rupee:before{content:""}.fa-cny:before,.fa-jpy:before,.fa-rmb:before,.fa-yen:before{content:""}.fa-rouble:before,.fa-rub:before,.fa-ruble:before{content:""}.fa-krw:before,.fa-won:before{content:""}.fa-bitcoin:before,.fa-btc:before{content:""}.fa-file:before{content:""}.fa-file-text:before{content:""}.fa-sort-alpha-asc:before{content:""}.fa-sort-alpha-desc:before{content:""}.fa-sort-amount-asc:before{content:""}.fa-sort-amount-desc:before{content:""}.fa-sort-numeric-asc:before{content:""}.fa-sort-numeric-desc:before{content:""}.fa-thumbs-up:before{content:""}.fa-thumbs-down:before{content:""}.fa-youtube-square:before{content:""}.fa-youtube:before{content:""}.fa-xing:before{content:""}.fa-xing-square:before{content:""}.fa-youtube-play:before{content:""}.fa-dropbox:before{content:""}.fa-stack-overflow:before{content:""}.fa-instagram:before{content:""}.fa-flickr:before{content:""}.fa-adn:before{content:""}.fa-bitbucket:before,.icon-bitbucket:before{content:""}.fa-bitbucket-square:before{content:""}.fa-tumblr:before{content:""}.fa-tumblr-square:before{content:""}.fa-long-arrow-down:before{content:""}.fa-long-arrow-up:before{content:""}.fa-long-arrow-left:before{content:""}.fa-long-arrow-right:before{content:""}.fa-apple:before{content:""}.fa-windows:before{content:""}.fa-android:before{content:""}.fa-linux:before{content:""}.fa-dribbble:before{content:""}.fa-skype:before{content:""}.fa-foursquare:before{content:""}.fa-trello:before{content:""}.fa-female:before{content:""}.fa-male:before{content:""}.fa-gittip:before,.fa-gratipay:before{content:""}.fa-sun-o:before{content:""}.fa-moon-o:before{content:""}.fa-archive:before{content:""}.fa-bug:before{content:""}.fa-vk:before{content:""}.fa-weibo:before{content:""}.fa-renren:before{content:""}.fa-pagelines:before{content:""}.fa-stack-exchange:before{content:""}.fa-arrow-circle-o-right:before{content:""}.fa-arrow-circle-o-left:before{content:""}.fa-caret-square-o-left:before,.fa-toggle-left:before{content:""}.fa-dot-circle-o:before{content:""}.fa-wheelchair:before{content:""}.fa-vimeo-square:before{content:""}.fa-try:before,.fa-turkish-lira:before{content:""}.fa-plus-square-o:before,.wy-menu-vertical li span.toctree-expand:before{content:""}.fa-space-shuttle:before{content:""}.fa-slack:before{content:""}.fa-envelope-square:before{content:""}.fa-wordpress:before{content:""}.fa-openid:before{content:""}.fa-bank:before,.fa-institution:before,.fa-university:before{content:""}.fa-graduation-cap:before,.fa-mortar-board:before{content:""}.fa-yahoo:before{content:""}.fa-google:before{content:""}.fa-reddit:before{content:""}.fa-reddit-square:before{content:""}.fa-stumbleupon-circle:before{content:""}.fa-stumbleupon:before{content:""}.fa-delicious:before{content:""}.fa-digg:before{content:""}.fa-pied-piper-pp:before{content:""}.fa-pied-piper-alt:before{content:""}.fa-drupal:before{content:""}.fa-joomla:before{content:""}.fa-language:before{content:""}.fa-fax:before{content:""}.fa-building:before{content:""}.fa-child:before{content:""}.fa-paw:before{content:""}.fa-spoon:before{content:""}.fa-cube:before{content:""}.fa-cubes:before{content:""}.fa-behance:before{content:""}.fa-behance-square:before{content:""}.fa-steam:before{content:""}.fa-steam-square:before{content:""}.fa-recycle:before{content:""}.fa-automobile:before,.fa-car:before{content:""}.fa-cab:before,.fa-taxi:before{content:""}.fa-tree:before{content:""}.fa-spotify:before{content:""}.fa-deviantart:before{content:""}.fa-soundcloud:before{content:""}.fa-database:before{content:""}.fa-file-pdf-o:before{content:""}.fa-file-word-o:before{content:""}.fa-file-excel-o:before{content:""}.fa-file-powerpoint-o:before{content:""}.fa-file-image-o:before,.fa-file-photo-o:before,.fa-file-picture-o:before{content:""}.fa-file-archive-o:before,.fa-file-zip-o:before{content:""}.fa-file-audio-o:before,.fa-file-sound-o:before{content:""}.fa-file-movie-o:before,.fa-file-video-o:before{content:""}.fa-file-code-o:before{content:""}.fa-vine:before{content:""}.fa-codepen:before{content:""}.fa-jsfiddle:before{content:""}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-ring:before,.fa-life-saver:before,.fa-support:before{content:""}.fa-circle-o-notch:before{content:""}.fa-ra:before,.fa-rebel:before,.fa-resistance:before{content:""}.fa-empire:before,.fa-ge:before{content:""}.fa-git-square:before{content:""}.fa-git:before{content:""}.fa-hacker-news:before,.fa-y-combinator-square:before,.fa-yc-square:before{content:""}.fa-tencent-weibo:before{content:""}.fa-qq:before{content:""}.fa-wechat:before,.fa-weixin:before{content:""}.fa-paper-plane:before,.fa-send:before{content:""}.fa-paper-plane-o:before,.fa-send-o:before{content:""}.fa-history:before{content:""}.fa-circle-thin:before{content:""}.fa-header:before{content:""}.fa-paragraph:before{content:""}.fa-sliders:before{content:""}.fa-share-alt:before{content:""}.fa-share-alt-square:before{content:""}.fa-bomb:before{content:""}.fa-futbol-o:before,.fa-soccer-ball-o:before{content:""}.fa-tty:before{content:""}.fa-binoculars:before{content:""}.fa-plug:before{content:""}.fa-slideshare:before{content:""}.fa-twitch:before{content:""}.fa-yelp:before{content:""}.fa-newspaper-o:before{content:""}.fa-wifi:before{content:""}.fa-calculator:before{content:""}.fa-paypal:before{content:""}.fa-google-wallet:before{content:""}.fa-cc-visa:before{content:""}.fa-cc-mastercard:before{content:""}.fa-cc-discover:before{content:""}.fa-cc-amex:before{content:""}.fa-cc-paypal:before{content:""}.fa-cc-stripe:before{content:""}.fa-bell-slash:before{content:""}.fa-bell-slash-o:before{content:""}.fa-trash:before{content:""}.fa-copyright:before{content:""}.fa-at:before{content:""}.fa-eyedropper:before{content:""}.fa-paint-brush:before{content:""}.fa-birthday-cake:before{content:""}.fa-area-chart:before{content:""}.fa-pie-chart:before{content:""}.fa-line-chart:before{content:""}.fa-lastfm:before{content:""}.fa-lastfm-square:before{content:""}.fa-toggle-off:before{content:""}.fa-toggle-on:before{content:""}.fa-bicycle:before{content:""}.fa-bus:before{content:""}.fa-ioxhost:before{content:""}.fa-angellist:before{content:""}.fa-cc:before{content:""}.fa-ils:before,.fa-shekel:before,.fa-sheqel:before{content:""}.fa-meanpath:before{content:""}.fa-buysellads:before{content:""}.fa-connectdevelop:before{content:""}.fa-dashcube:before{content:""}.fa-forumbee:before{content:""}.fa-leanpub:before{content:""}.fa-sellsy:before{content:""}.fa-shirtsinbulk:before{content:""}.fa-simplybuilt:before{content:""}.fa-skyatlas:before{content:""}.fa-cart-plus:before{content:""}.fa-cart-arrow-down:before{content:""}.fa-diamond:before{content:""}.fa-ship:before{content:""}.fa-user-secret:before{content:""}.fa-motorcycle:before{content:""}.fa-street-view:before{content:""}.fa-heartbeat:before{content:""}.fa-venus:before{content:""}.fa-mars:before{content:""}.fa-mercury:before{content:""}.fa-intersex:before,.fa-transgender:before{content:""}.fa-transgender-alt:before{content:""}.fa-venus-double:before{content:""}.fa-mars-double:before{content:""}.fa-venus-mars:before{content:""}.fa-mars-stroke:before{content:""}.fa-mars-stroke-v:before{content:""}.fa-mars-stroke-h:before{content:""}.fa-neuter:before{content:""}.fa-genderless:before{content:""}.fa-facebook-official:before{content:""}.fa-pinterest-p:before{content:""}.fa-whatsapp:before{content:""}.fa-server:before{content:""}.fa-user-plus:before{content:""}.fa-user-times:before{content:""}.fa-bed:before,.fa-hotel:before{content:""}.fa-viacoin:before{content:""}.fa-train:before{content:""}.fa-subway:before{content:""}.fa-medium:before{content:""}.fa-y-combinator:before,.fa-yc:before{content:""}.fa-optin-monster:before{content:""}.fa-opencart:before{content:""}.fa-expeditedssl:before{content:""}.fa-battery-4:before,.fa-battery-full:before,.fa-battery:before{content:""}.fa-battery-3:before,.fa-battery-three-quarters:before{content:""}.fa-battery-2:before,.fa-battery-half:before{content:""}.fa-battery-1:before,.fa-battery-quarter:before{content:""}.fa-battery-0:before,.fa-battery-empty:before{content:""}.fa-mouse-pointer:before{content:""}.fa-i-cursor:before{content:""}.fa-object-group:before{content:""}.fa-object-ungroup:before{content:""}.fa-sticky-note:before{content:""}.fa-sticky-note-o:before{content:""}.fa-cc-jcb:before{content:""}.fa-cc-diners-club:before{content:""}.fa-clone:before{content:""}.fa-balance-scale:before{content:""}.fa-hourglass-o:before{content:""}.fa-hourglass-1:before,.fa-hourglass-start:before{content:""}.fa-hourglass-2:before,.fa-hourglass-half:before{content:""}.fa-hourglass-3:before,.fa-hourglass-end:before{content:""}.fa-hourglass:before{content:""}.fa-hand-grab-o:before,.fa-hand-rock-o:before{content:""}.fa-hand-paper-o:before,.fa-hand-stop-o:before{content:""}.fa-hand-scissors-o:before{content:""}.fa-hand-lizard-o:before{content:""}.fa-hand-spock-o:before{content:""}.fa-hand-pointer-o:before{content:""}.fa-hand-peace-o:before{content:""}.fa-trademark:before{content:""}.fa-registered:before{content:""}.fa-creative-commons:before{content:""}.fa-gg:before{content:""}.fa-gg-circle:before{content:""}.fa-tripadvisor:before{content:""}.fa-odnoklassniki:before{content:""}.fa-odnoklassniki-square:before{content:""}.fa-get-pocket:before{content:""}.fa-wikipedia-w:before{content:""}.fa-safari:before{content:""}.fa-chrome:before{content:""}.fa-firefox:before{content:""}.fa-opera:before{content:""}.fa-internet-explorer:before{content:""}.fa-television:before,.fa-tv:before{content:""}.fa-contao:before{content:""}.fa-500px:before{content:""}.fa-amazon:before{content:""}.fa-calendar-plus-o:before{content:""}.fa-calendar-minus-o:before{content:""}.fa-calendar-times-o:before{content:""}.fa-calendar-check-o:before{content:""}.fa-industry:before{content:""}.fa-map-pin:before{content:""}.fa-map-signs:before{content:""}.fa-map-o:before{content:""}.fa-map:before{content:""}.fa-commenting:before{content:""}.fa-commenting-o:before{content:""}.fa-houzz:before{content:""}.fa-vimeo:before{content:""}.fa-black-tie:before{content:""}.fa-fonticons:before{content:""}.fa-reddit-alien:before{content:""}.fa-edge:before{content:""}.fa-credit-card-alt:before{content:""}.fa-codiepie:before{content:""}.fa-modx:before{content:""}.fa-fort-awesome:before{content:""}.fa-usb:before{content:""}.fa-product-hunt:before{content:""}.fa-mixcloud:before{content:""}.fa-scribd:before{content:""}.fa-pause-circle:before{content:""}.fa-pause-circle-o:before{content:""}.fa-stop-circle:before{content:""}.fa-stop-circle-o:before{content:""}.fa-shopping-bag:before{content:""}.fa-shopping-basket:before{content:""}.fa-hashtag:before{content:""}.fa-bluetooth:before{content:""}.fa-bluetooth-b:before{content:""}.fa-percent:before{content:""}.fa-gitlab:before,.icon-gitlab:before{content:""}.fa-wpbeginner:before{content:""}.fa-wpforms:before{content:""}.fa-envira:before{content:""}.fa-universal-access:before{content:""}.fa-wheelchair-alt:before{content:""}.fa-question-circle-o:before{content:""}.fa-blind:before{content:""}.fa-audio-description:before{content:""}.fa-volume-control-phone:before{content:""}.fa-braille:before{content:""}.fa-assistive-listening-systems:before{content:""}.fa-american-sign-language-interpreting:before,.fa-asl-interpreting:before{content:""}.fa-deaf:before,.fa-deafness:before,.fa-hard-of-hearing:before{content:""}.fa-glide:before{content:""}.fa-glide-g:before{content:""}.fa-sign-language:before,.fa-signing:before{content:""}.fa-low-vision:before{content:""}.fa-viadeo:before{content:""}.fa-viadeo-square:before{content:""}.fa-snapchat:before{content:""}.fa-snapchat-ghost:before{content:""}.fa-snapchat-square:before{content:""}.fa-pied-piper:before{content:""}.fa-first-order:before{content:""}.fa-yoast:before{content:""}.fa-themeisle:before{content:""}.fa-google-plus-circle:before,.fa-google-plus-official:before{content:""}.fa-fa:before,.fa-font-awesome:before{content:""}.fa-handshake-o:before{content:""}.fa-envelope-open:before{content:""}.fa-envelope-open-o:before{content:""}.fa-linode:before{content:""}.fa-address-book:before{content:""}.fa-address-book-o:before{content:""}.fa-address-card:before,.fa-vcard:before{content:""}.fa-address-card-o:before,.fa-vcard-o:before{content:""}.fa-user-circle:before{content:""}.fa-user-circle-o:before{content:""}.fa-user-o:before{content:""}.fa-id-badge:before{content:""}.fa-drivers-license:before,.fa-id-card:before{content:""}.fa-drivers-license-o:before,.fa-id-card-o:before{content:""}.fa-quora:before{content:""}.fa-free-code-camp:before{content:""}.fa-telegram:before{content:""}.fa-thermometer-4:before,.fa-thermometer-full:before,.fa-thermometer:before{content:""}.fa-thermometer-3:before,.fa-thermometer-three-quarters:before{content:""}.fa-thermometer-2:before,.fa-thermometer-half:before{content:""}.fa-thermometer-1:before,.fa-thermometer-quarter:before{content:""}.fa-thermometer-0:before,.fa-thermometer-empty:before{content:""}.fa-shower:before{content:""}.fa-bath:before,.fa-bathtub:before,.fa-s15:before{content:""}.fa-podcast:before{content:""}.fa-window-maximize:before{content:""}.fa-window-minimize:before{content:""}.fa-window-restore:before{content:""}.fa-times-rectangle:before,.fa-window-close:before{content:""}.fa-times-rectangle-o:before,.fa-window-close-o:before{content:""}.fa-bandcamp:before{content:""}.fa-grav:before{content:""}.fa-etsy:before{content:""}.fa-imdb:before{content:""}.fa-ravelry:before{content:""}.fa-eercast:before{content:""}.fa-microchip:before{content:""}.fa-snowflake-o:before{content:""}.fa-superpowers:before{content:""}.fa-wpexplorer:before{content:""}.fa-meetup:before{content:""}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-dropdown .caret,.wy-inline-validate.wy-inline-validate-danger .wy-input-context,.wy-inline-validate.wy-inline-validate-info .wy-input-context,.wy-inline-validate.wy-inline-validate-success .wy-input-context,.wy-inline-validate.wy-inline-validate-warning .wy-input-context,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{font-family:inherit}.fa:before,.icon:before,.rst-content .admonition-title:before,.rst-content .code-block-caption .headerlink:before,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before{font-family:FontAwesome;display:inline-block;font-style:normal;font-weight:400;line-height:1;text-decoration:inherit}.rst-content .code-block-caption a .headerlink,.rst-content a .admonition-title,.rst-content code.download a span:first-child,.rst-content dl dt a .headerlink,.rst-content h1 a .headerlink,.rst-content h2 a .headerlink,.rst-content h3 a .headerlink,.rst-content h4 a .headerlink,.rst-content h5 a .headerlink,.rst-content h6 a .headerlink,.rst-content p.caption a .headerlink,.rst-content table>caption a .headerlink,.rst-content tt.download a span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li a span.toctree-expand,a .fa,a .icon,a .rst-content .admonition-title,a .rst-content .code-block-caption .headerlink,a .rst-content code.download span:first-child,a .rst-content dl dt .headerlink,a .rst-content h1 .headerlink,a .rst-content h2 .headerlink,a .rst-content h3 .headerlink,a .rst-content h4 .headerlink,a .rst-content h5 .headerlink,a .rst-content h6 .headerlink,a .rst-content p.caption .headerlink,a .rst-content table>caption .headerlink,a .rst-content tt.download span:first-child,a .wy-menu-vertical li span.toctree-expand{display:inline-block;text-decoration:inherit}.btn .fa,.btn .icon,.btn .rst-content .admonition-title,.btn .rst-content .code-block-caption .headerlink,.btn .rst-content code.download span:first-child,.btn .rst-content dl dt .headerlink,.btn .rst-content h1 .headerlink,.btn .rst-content h2 .headerlink,.btn .rst-content h3 .headerlink,.btn .rst-content h4 .headerlink,.btn .rst-content h5 .headerlink,.btn .rst-content h6 .headerlink,.btn .rst-content p.caption .headerlink,.btn .rst-content table>caption .headerlink,.btn .rst-content tt.download span:first-child,.btn .wy-menu-vertical li.current>a span.toctree-expand,.btn .wy-menu-vertical li.on a span.toctree-expand,.btn .wy-menu-vertical li span.toctree-expand,.nav .fa,.nav .icon,.nav .rst-content .admonition-title,.nav .rst-content .code-block-caption .headerlink,.nav .rst-content code.download span:first-child,.nav .rst-content dl dt .headerlink,.nav .rst-content h1 .headerlink,.nav .rst-content h2 .headerlink,.nav .rst-content h3 .headerlink,.nav .rst-content h4 .headerlink,.nav .rst-content h5 .headerlink,.nav .rst-content h6 .headerlink,.nav .rst-content p.caption .headerlink,.nav .rst-content table>caption .headerlink,.nav .rst-content tt.download span:first-child,.nav .wy-menu-vertical li.current>a span.toctree-expand,.nav .wy-menu-vertical li.on a span.toctree-expand,.nav .wy-menu-vertical li span.toctree-expand,.rst-content .btn .admonition-title,.rst-content .code-block-caption .btn .headerlink,.rst-content .code-block-caption .nav .headerlink,.rst-content .nav .admonition-title,.rst-content code.download .btn span:first-child,.rst-content code.download .nav span:first-child,.rst-content dl dt .btn .headerlink,.rst-content dl dt .nav .headerlink,.rst-content h1 .btn .headerlink,.rst-content h1 .nav .headerlink,.rst-content h2 .btn .headerlink,.rst-content h2 .nav .headerlink,.rst-content h3 .btn .headerlink,.rst-content h3 .nav .headerlink,.rst-content h4 .btn .headerlink,.rst-content h4 .nav .headerlink,.rst-content h5 .btn .headerlink,.rst-content h5 .nav .headerlink,.rst-content h6 .btn .headerlink,.rst-content h6 .nav .headerlink,.rst-content p.caption .btn .headerlink,.rst-content p.caption .nav .headerlink,.rst-content table>caption .btn .headerlink,.rst-content table>caption .nav .headerlink,.rst-content tt.download .btn span:first-child,.rst-content tt.download .nav span:first-child,.wy-menu-vertical li .btn span.toctree-expand,.wy-menu-vertical li.current>a .btn span.toctree-expand,.wy-menu-vertical li.current>a .nav span.toctree-expand,.wy-menu-vertical li .nav span.toctree-expand,.wy-menu-vertical li.on a .btn span.toctree-expand,.wy-menu-vertical li.on a .nav span.toctree-expand{display:inline}.btn .fa-large.icon,.btn .fa.fa-large,.btn .rst-content .code-block-caption .fa-large.headerlink,.btn .rst-content .fa-large.admonition-title,.btn .rst-content code.download span.fa-large:first-child,.btn .rst-content dl dt .fa-large.headerlink,.btn .rst-content h1 .fa-large.headerlink,.btn .rst-content h2 .fa-large.headerlink,.btn .rst-content h3 .fa-large.headerlink,.btn .rst-content h4 .fa-large.headerlink,.btn .rst-content h5 .fa-large.headerlink,.btn .rst-content h6 .fa-large.headerlink,.btn .rst-content p.caption .fa-large.headerlink,.btn .rst-content table>caption .fa-large.headerlink,.btn .rst-content tt.download span.fa-large:first-child,.btn .wy-menu-vertical li span.fa-large.toctree-expand,.nav .fa-large.icon,.nav .fa.fa-large,.nav .rst-content .code-block-caption .fa-large.headerlink,.nav .rst-content .fa-large.admonition-title,.nav .rst-content code.download span.fa-large:first-child,.nav .rst-content dl dt .fa-large.headerlink,.nav .rst-content h1 .fa-large.headerlink,.nav .rst-content h2 .fa-large.headerlink,.nav .rst-content h3 .fa-large.headerlink,.nav .rst-content h4 .fa-large.headerlink,.nav .rst-content h5 .fa-large.headerlink,.nav .rst-content h6 .fa-large.headerlink,.nav .rst-content p.caption .fa-large.headerlink,.nav .rst-content table>caption .fa-large.headerlink,.nav .rst-content tt.download span.fa-large:first-child,.nav .wy-menu-vertical li span.fa-large.toctree-expand,.rst-content .btn .fa-large.admonition-title,.rst-content .code-block-caption .btn .fa-large.headerlink,.rst-content .code-block-caption .nav .fa-large.headerlink,.rst-content .nav .fa-large.admonition-title,.rst-content code.download .btn span.fa-large:first-child,.rst-content code.download .nav span.fa-large:first-child,.rst-content dl dt .btn .fa-large.headerlink,.rst-content dl dt .nav .fa-large.headerlink,.rst-content h1 .btn .fa-large.headerlink,.rst-content h1 .nav .fa-large.headerlink,.rst-content h2 .btn .fa-large.headerlink,.rst-content h2 .nav .fa-large.headerlink,.rst-content h3 .btn .fa-large.headerlink,.rst-content h3 .nav .fa-large.headerlink,.rst-content h4 .btn .fa-large.headerlink,.rst-content h4 .nav .fa-large.headerlink,.rst-content h5 .btn .fa-large.headerlink,.rst-content h5 .nav .fa-large.headerlink,.rst-content h6 .btn .fa-large.headerlink,.rst-content h6 .nav .fa-large.headerlink,.rst-content p.caption .btn .fa-large.headerlink,.rst-content p.caption .nav .fa-large.headerlink,.rst-content table>caption .btn .fa-large.headerlink,.rst-content table>caption .nav .fa-large.headerlink,.rst-content tt.download .btn span.fa-large:first-child,.rst-content tt.download .nav span.fa-large:first-child,.wy-menu-vertical li .btn span.fa-large.toctree-expand,.wy-menu-vertical li .nav span.fa-large.toctree-expand{line-height:.9em}.btn .fa-spin.icon,.btn .fa.fa-spin,.btn .rst-content .code-block-caption .fa-spin.headerlink,.btn .rst-content .fa-spin.admonition-title,.btn .rst-content code.download span.fa-spin:first-child,.btn .rst-content dl dt .fa-spin.headerlink,.btn .rst-content h1 .fa-spin.headerlink,.btn .rst-content h2 .fa-spin.headerlink,.btn .rst-content h3 .fa-spin.headerlink,.btn .rst-content h4 .fa-spin.headerlink,.btn .rst-content h5 .fa-spin.headerlink,.btn .rst-content h6 .fa-spin.headerlink,.btn .rst-content p.caption .fa-spin.headerlink,.btn .rst-content table>caption .fa-spin.headerlink,.btn .rst-content tt.download span.fa-spin:first-child,.btn .wy-menu-vertical li span.fa-spin.toctree-expand,.nav .fa-spin.icon,.nav .fa.fa-spin,.nav .rst-content .code-block-caption .fa-spin.headerlink,.nav .rst-content .fa-spin.admonition-title,.nav .rst-content code.download span.fa-spin:first-child,.nav .rst-content dl dt .fa-spin.headerlink,.nav .rst-content h1 .fa-spin.headerlink,.nav .rst-content h2 .fa-spin.headerlink,.nav .rst-content h3 .fa-spin.headerlink,.nav .rst-content h4 .fa-spin.headerlink,.nav .rst-content h5 .fa-spin.headerlink,.nav .rst-content h6 .fa-spin.headerlink,.nav .rst-content p.caption .fa-spin.headerlink,.nav .rst-content table>caption .fa-spin.headerlink,.nav .rst-content tt.download span.fa-spin:first-child,.nav .wy-menu-vertical li span.fa-spin.toctree-expand,.rst-content .btn .fa-spin.admonition-title,.rst-content .code-block-caption .btn .fa-spin.headerlink,.rst-content .code-block-caption .nav .fa-spin.headerlink,.rst-content .nav .fa-spin.admonition-title,.rst-content code.download .btn span.fa-spin:first-child,.rst-content code.download .nav span.fa-spin:first-child,.rst-content dl dt .btn .fa-spin.headerlink,.rst-content dl dt .nav .fa-spin.headerlink,.rst-content h1 .btn .fa-spin.headerlink,.rst-content h1 .nav .fa-spin.headerlink,.rst-content h2 .btn .fa-spin.headerlink,.rst-content h2 .nav .fa-spin.headerlink,.rst-content h3 .btn .fa-spin.headerlink,.rst-content h3 .nav .fa-spin.headerlink,.rst-content h4 .btn .fa-spin.headerlink,.rst-content h4 .nav .fa-spin.headerlink,.rst-content h5 .btn .fa-spin.headerlink,.rst-content h5 .nav .fa-spin.headerlink,.rst-content h6 .btn .fa-spin.headerlink,.rst-content h6 .nav .fa-spin.headerlink,.rst-content p.caption .btn .fa-spin.headerlink,.rst-content p.caption .nav .fa-spin.headerlink,.rst-content table>caption .btn .fa-spin.headerlink,.rst-content table>caption .nav .fa-spin.headerlink,.rst-content tt.download .btn span.fa-spin:first-child,.rst-content tt.download .nav span.fa-spin:first-child,.wy-menu-vertical li .btn span.fa-spin.toctree-expand,.wy-menu-vertical li .nav span.fa-spin.toctree-expand{display:inline-block}.btn.fa:before,.btn.icon:before,.rst-content .btn.admonition-title:before,.rst-content .code-block-caption .btn.headerlink:before,.rst-content code.download span.btn:first-child:before,.rst-content dl dt .btn.headerlink:before,.rst-content h1 .btn.headerlink:before,.rst-content h2 .btn.headerlink:before,.rst-content h3 .btn.headerlink:before,.rst-content h4 .btn.headerlink:before,.rst-content h5 .btn.headerlink:before,.rst-content h6 .btn.headerlink:before,.rst-content p.caption .btn.headerlink:before,.rst-content table>caption .btn.headerlink:before,.rst-content tt.download span.btn:first-child:before,.wy-menu-vertical li span.btn.toctree-expand:before{opacity:.5;-webkit-transition:opacity .05s ease-in;-moz-transition:opacity .05s ease-in;transition:opacity .05s ease-in}.btn.fa:hover:before,.btn.icon:hover:before,.rst-content .btn.admonition-title:hover:before,.rst-content .code-block-caption .btn.headerlink:hover:before,.rst-content code.download span.btn:first-child:hover:before,.rst-content dl dt .btn.headerlink:hover:before,.rst-content h1 .btn.headerlink:hover:before,.rst-content h2 .btn.headerlink:hover:before,.rst-content h3 .btn.headerlink:hover:before,.rst-content h4 .btn.headerlink:hover:before,.rst-content h5 .btn.headerlink:hover:before,.rst-content h6 .btn.headerlink:hover:before,.rst-content p.caption .btn.headerlink:hover:before,.rst-content table>caption .btn.headerlink:hover:before,.rst-content tt.download span.btn:first-child:hover:before,.wy-menu-vertical li span.btn.toctree-expand:hover:before{opacity:1}.btn-mini .fa:before,.btn-mini .icon:before,.btn-mini .rst-content .admonition-title:before,.btn-mini .rst-content .code-block-caption .headerlink:before,.btn-mini .rst-content code.download span:first-child:before,.btn-mini .rst-content dl dt .headerlink:before,.btn-mini .rst-content h1 .headerlink:before,.btn-mini .rst-content h2 .headerlink:before,.btn-mini .rst-content h3 .headerlink:before,.btn-mini .rst-content h4 .headerlink:before,.btn-mini .rst-content h5 .headerlink:before,.btn-mini .rst-content h6 .headerlink:before,.btn-mini .rst-content p.caption .headerlink:before,.btn-mini .rst-content table>caption .headerlink:before,.btn-mini .rst-content tt.download span:first-child:before,.btn-mini .wy-menu-vertical li span.toctree-expand:before,.rst-content .btn-mini .admonition-title:before,.rst-content .code-block-caption .btn-mini .headerlink:before,.rst-content code.download .btn-mini span:first-child:before,.rst-content dl dt .btn-mini .headerlink:before,.rst-content h1 .btn-mini .headerlink:before,.rst-content h2 .btn-mini .headerlink:before,.rst-content h3 .btn-mini .headerlink:before,.rst-content h4 .btn-mini .headerlink:before,.rst-content h5 .btn-mini .headerlink:before,.rst-content h6 .btn-mini .headerlink:before,.rst-content p.caption .btn-mini .headerlink:before,.rst-content table>caption .btn-mini .headerlink:before,.rst-content tt.download .btn-mini span:first-child:before,.wy-menu-vertical li .btn-mini span.toctree-expand:before{font-size:14px;vertical-align:-15%}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.wy-alert{padding:12px;line-height:24px;margin-bottom:24px;background:#e7f2fa}.rst-content .admonition-title,.wy-alert-title{font-weight:700;display:block;color:#fff;background:#6ab0de;padding:6px 12px;margin:-12px -12px 12px}.rst-content .danger,.rst-content .error,.rst-content .wy-alert-danger.admonition,.rst-content .wy-alert-danger.admonition-todo,.rst-content .wy-alert-danger.attention,.rst-content .wy-alert-danger.caution,.rst-content .wy-alert-danger.hint,.rst-content .wy-alert-danger.important,.rst-content .wy-alert-danger.note,.rst-content .wy-alert-danger.seealso,.rst-content .wy-alert-danger.tip,.rst-content .wy-alert-danger.warning,.wy-alert.wy-alert-danger{background:#fdf3f2}.rst-content .danger .admonition-title,.rst-content .danger .wy-alert-title,.rst-content .error .admonition-title,.rst-content .error .wy-alert-title,.rst-content .wy-alert-danger.admonition-todo .admonition-title,.rst-content .wy-alert-danger.admonition-todo .wy-alert-title,.rst-content .wy-alert-danger.admonition .admonition-title,.rst-content .wy-alert-danger.admonition .wy-alert-title,.rst-content .wy-alert-danger.attention .admonition-title,.rst-content .wy-alert-danger.attention .wy-alert-title,.rst-content .wy-alert-danger.caution .admonition-title,.rst-content .wy-alert-danger.caution .wy-alert-title,.rst-content .wy-alert-danger.hint .admonition-title,.rst-content .wy-alert-danger.hint .wy-alert-title,.rst-content .wy-alert-danger.important .admonition-title,.rst-content .wy-alert-danger.important .wy-alert-title,.rst-content .wy-alert-danger.note .admonition-title,.rst-content .wy-alert-danger.note .wy-alert-title,.rst-content .wy-alert-danger.seealso .admonition-title,.rst-content .wy-alert-danger.seealso .wy-alert-title,.rst-content .wy-alert-danger.tip .admonition-title,.rst-content .wy-alert-danger.tip .wy-alert-title,.rst-content .wy-alert-danger.warning .admonition-title,.rst-content .wy-alert-danger.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-danger .admonition-title,.wy-alert.wy-alert-danger .rst-content .admonition-title,.wy-alert.wy-alert-danger .wy-alert-title{background:#f29f97}.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .warning,.rst-content .wy-alert-warning.admonition,.rst-content .wy-alert-warning.danger,.rst-content .wy-alert-warning.error,.rst-content .wy-alert-warning.hint,.rst-content .wy-alert-warning.important,.rst-content .wy-alert-warning.note,.rst-content .wy-alert-warning.seealso,.rst-content .wy-alert-warning.tip,.wy-alert.wy-alert-warning{background:#ffedcc}.rst-content .admonition-todo .admonition-title,.rst-content .admonition-todo .wy-alert-title,.rst-content .attention .admonition-title,.rst-content .attention .wy-alert-title,.rst-content .caution .admonition-title,.rst-content .caution .wy-alert-title,.rst-content .warning .admonition-title,.rst-content .warning .wy-alert-title,.rst-content .wy-alert-warning.admonition .admonition-title,.rst-content .wy-alert-warning.admonition .wy-alert-title,.rst-content .wy-alert-warning.danger .admonition-title,.rst-content .wy-alert-warning.danger .wy-alert-title,.rst-content .wy-alert-warning.error .admonition-title,.rst-content .wy-alert-warning.error .wy-alert-title,.rst-content .wy-alert-warning.hint .admonition-title,.rst-content .wy-alert-warning.hint .wy-alert-title,.rst-content .wy-alert-warning.important .admonition-title,.rst-content .wy-alert-warning.important .wy-alert-title,.rst-content .wy-alert-warning.note .admonition-title,.rst-content .wy-alert-warning.note .wy-alert-title,.rst-content .wy-alert-warning.seealso .admonition-title,.rst-content .wy-alert-warning.seealso .wy-alert-title,.rst-content .wy-alert-warning.tip .admonition-title,.rst-content .wy-alert-warning.tip .wy-alert-title,.rst-content .wy-alert.wy-alert-warning .admonition-title,.wy-alert.wy-alert-warning .rst-content .admonition-title,.wy-alert.wy-alert-warning .wy-alert-title{background:#f0b37e}.rst-content .note,.rst-content .seealso,.rst-content .wy-alert-info.admonition,.rst-content .wy-alert-info.admonition-todo,.rst-content .wy-alert-info.attention,.rst-content .wy-alert-info.caution,.rst-content .wy-alert-info.danger,.rst-content .wy-alert-info.error,.rst-content .wy-alert-info.hint,.rst-content .wy-alert-info.important,.rst-content .wy-alert-info.tip,.rst-content .wy-alert-info.warning,.wy-alert.wy-alert-info{background:#e7f2fa}.rst-content .note .admonition-title,.rst-content .note .wy-alert-title,.rst-content .seealso .admonition-title,.rst-content .seealso .wy-alert-title,.rst-content .wy-alert-info.admonition-todo .admonition-title,.rst-content .wy-alert-info.admonition-todo .wy-alert-title,.rst-content .wy-alert-info.admonition .admonition-title,.rst-content .wy-alert-info.admonition .wy-alert-title,.rst-content .wy-alert-info.attention .admonition-title,.rst-content .wy-alert-info.attention .wy-alert-title,.rst-content .wy-alert-info.caution .admonition-title,.rst-content .wy-alert-info.caution .wy-alert-title,.rst-content .wy-alert-info.danger .admonition-title,.rst-content .wy-alert-info.danger .wy-alert-title,.rst-content .wy-alert-info.error .admonition-title,.rst-content .wy-alert-info.error .wy-alert-title,.rst-content .wy-alert-info.hint .admonition-title,.rst-content .wy-alert-info.hint .wy-alert-title,.rst-content .wy-alert-info.important .admonition-title,.rst-content .wy-alert-info.important .wy-alert-title,.rst-content .wy-alert-info.tip .admonition-title,.rst-content .wy-alert-info.tip .wy-alert-title,.rst-content .wy-alert-info.warning .admonition-title,.rst-content .wy-alert-info.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-info .admonition-title,.wy-alert.wy-alert-info .rst-content .admonition-title,.wy-alert.wy-alert-info .wy-alert-title{background:#6ab0de}.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .wy-alert-success.admonition,.rst-content .wy-alert-success.admonition-todo,.rst-content .wy-alert-success.attention,.rst-content .wy-alert-success.caution,.rst-content .wy-alert-success.danger,.rst-content .wy-alert-success.error,.rst-content .wy-alert-success.note,.rst-content .wy-alert-success.seealso,.rst-content .wy-alert-success.warning,.wy-alert.wy-alert-success{background:#dbfaf4}.rst-content .hint .admonition-title,.rst-content .hint .wy-alert-title,.rst-content .important .admonition-title,.rst-content .important .wy-alert-title,.rst-content .tip .admonition-title,.rst-content .tip .wy-alert-title,.rst-content .wy-alert-success.admonition-todo .admonition-title,.rst-content .wy-alert-success.admonition-todo .wy-alert-title,.rst-content .wy-alert-success.admonition .admonition-title,.rst-content .wy-alert-success.admonition .wy-alert-title,.rst-content .wy-alert-success.attention .admonition-title,.rst-content .wy-alert-success.attention .wy-alert-title,.rst-content .wy-alert-success.caution .admonition-title,.rst-content .wy-alert-success.caution .wy-alert-title,.rst-content .wy-alert-success.danger .admonition-title,.rst-content .wy-alert-success.danger .wy-alert-title,.rst-content .wy-alert-success.error .admonition-title,.rst-content .wy-alert-success.error .wy-alert-title,.rst-content .wy-alert-success.note .admonition-title,.rst-content .wy-alert-success.note .wy-alert-title,.rst-content .wy-alert-success.seealso .admonition-title,.rst-content .wy-alert-success.seealso .wy-alert-title,.rst-content .wy-alert-success.warning .admonition-title,.rst-content .wy-alert-success.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-success .admonition-title,.wy-alert.wy-alert-success .rst-content .admonition-title,.wy-alert.wy-alert-success .wy-alert-title{background:#1abc9c}.rst-content .wy-alert-neutral.admonition,.rst-content .wy-alert-neutral.admonition-todo,.rst-content .wy-alert-neutral.attention,.rst-content .wy-alert-neutral.caution,.rst-content .wy-alert-neutral.danger,.rst-content .wy-alert-neutral.error,.rst-content .wy-alert-neutral.hint,.rst-content .wy-alert-neutral.important,.rst-content .wy-alert-neutral.note,.rst-content .wy-alert-neutral.seealso,.rst-content .wy-alert-neutral.tip,.rst-content .wy-alert-neutral.warning,.wy-alert.wy-alert-neutral{background:#f3f6f6}.rst-content .wy-alert-neutral.admonition-todo .admonition-title,.rst-content .wy-alert-neutral.admonition-todo .wy-alert-title,.rst-content .wy-alert-neutral.admonition .admonition-title,.rst-content .wy-alert-neutral.admonition .wy-alert-title,.rst-content .wy-alert-neutral.attention .admonition-title,.rst-content .wy-alert-neutral.attention .wy-alert-title,.rst-content .wy-alert-neutral.caution .admonition-title,.rst-content .wy-alert-neutral.caution .wy-alert-title,.rst-content .wy-alert-neutral.danger .admonition-title,.rst-content .wy-alert-neutral.danger .wy-alert-title,.rst-content .wy-alert-neutral.error .admonition-title,.rst-content .wy-alert-neutral.error .wy-alert-title,.rst-content .wy-alert-neutral.hint .admonition-title,.rst-content .wy-alert-neutral.hint .wy-alert-title,.rst-content .wy-alert-neutral.important .admonition-title,.rst-content .wy-alert-neutral.important .wy-alert-title,.rst-content .wy-alert-neutral.note .admonition-title,.rst-content .wy-alert-neutral.note .wy-alert-title,.rst-content .wy-alert-neutral.seealso .admonition-title,.rst-content .wy-alert-neutral.seealso .wy-alert-title,.rst-content .wy-alert-neutral.tip .admonition-title,.rst-content .wy-alert-neutral.tip .wy-alert-title,.rst-content .wy-alert-neutral.warning .admonition-title,.rst-content .wy-alert-neutral.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-neutral .admonition-title,.wy-alert.wy-alert-neutral .rst-content .admonition-title,.wy-alert.wy-alert-neutral .wy-alert-title{color:#404040;background:#e1e4e5}.rst-content .wy-alert-neutral.admonition-todo a,.rst-content .wy-alert-neutral.admonition a,.rst-content .wy-alert-neutral.attention a,.rst-content .wy-alert-neutral.caution a,.rst-content .wy-alert-neutral.danger a,.rst-content .wy-alert-neutral.error a,.rst-content .wy-alert-neutral.hint a,.rst-content .wy-alert-neutral.important a,.rst-content .wy-alert-neutral.note a,.rst-content .wy-alert-neutral.seealso a,.rst-content .wy-alert-neutral.tip a,.rst-content .wy-alert-neutral.warning a,.wy-alert.wy-alert-neutral a{color:#2980b9}.rst-content .admonition-todo p:last-child,.rst-content .admonition p:last-child,.rst-content .attention p:last-child,.rst-content .caution p:last-child,.rst-content .danger p:last-child,.rst-content .error p:last-child,.rst-content .hint p:last-child,.rst-content .important p:last-child,.rst-content .note p:last-child,.rst-content .seealso p:last-child,.rst-content .tip p:last-child,.rst-content .warning p:last-child,.wy-alert p:last-child{margin-bottom:0}.wy-tray-container{position:fixed;bottom:0;left:0;z-index:600}.wy-tray-container li{display:block;width:300px;background:transparent;color:#fff;text-align:center;box-shadow:0 5px 5px 0 rgba(0,0,0,.1);padding:0 24px;min-width:20%;opacity:0;height:0;line-height:56px;overflow:hidden;-webkit-transition:all .3s ease-in;-moz-transition:all .3s ease-in;transition:all .3s ease-in}.wy-tray-container li.wy-tray-item-success{background:#27ae60}.wy-tray-container li.wy-tray-item-info{background:#2980b9}.wy-tray-container li.wy-tray-item-warning{background:#e67e22}.wy-tray-container li.wy-tray-item-danger{background:#e74c3c}.wy-tray-container li.on{opacity:1;height:56px}@media screen and (max-width:768px){.wy-tray-container{bottom:auto;top:0;width:100%}.wy-tray-container li{width:100%}}button{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle;cursor:pointer;line-height:normal;-webkit-appearance:button;*overflow:visible}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}button[disabled]{cursor:default}.btn{display:inline-block;border-radius:2px;line-height:normal;white-space:nowrap;text-align:center;cursor:pointer;font-size:100%;padding:6px 12px 8px;color:#fff;border:1px solid rgba(0,0,0,.1);background-color:#27ae60;text-decoration:none;font-weight:400;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 2px -1px hsla(0,0%,100%,.5),inset 0 -2px 0 0 rgba(0,0,0,.1);outline-none:false;vertical-align:middle;*display:inline;zoom:1;-webkit-user-drag:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transition:all .1s linear;-moz-transition:all .1s linear;transition:all .1s linear}.btn-hover{background:#2e8ece;color:#fff}.btn:hover{background:#2cc36b;color:#fff}.btn:focus{background:#2cc36b;outline:0}.btn:active{box-shadow:inset 0 -1px 0 0 rgba(0,0,0,.05),inset 0 2px 0 0 rgba(0,0,0,.1);padding:8px 12px 6px}.btn:visited{color:#fff}.btn-disabled,.btn-disabled:active,.btn-disabled:focus,.btn-disabled:hover,.btn:disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:.4;cursor:not-allowed;box-shadow:none}.btn::-moz-focus-inner{padding:0;border:0}.btn-small{font-size:80%}.btn-info{background-color:#2980b9!important}.btn-info:hover{background-color:#2e8ece!important}.btn-neutral{background-color:#f3f6f6!important;color:#404040!important}.btn-neutral:hover{background-color:#e5ebeb!important;color:#404040}.btn-neutral:visited{color:#404040!important}.btn-success{background-color:#27ae60!important}.btn-success:hover{background-color:#295!important}.btn-danger{background-color:#e74c3c!important}.btn-danger:hover{background-color:#ea6153!important}.btn-warning{background-color:#e67e22!important}.btn-warning:hover{background-color:#e98b39!important}.btn-invert{background-color:#222}.btn-invert:hover{background-color:#2f2f2f!important}.btn-link{background-color:transparent!important;color:#2980b9;box-shadow:none;border-color:transparent!important}.btn-link:active,.btn-link:hover{background-color:transparent!important;color:#409ad5!important;box-shadow:none}.btn-link:visited{color:#9b59b6}.wy-btn-group .btn,.wy-control .btn{vertical-align:middle}.wy-btn-group{margin-bottom:24px;*zoom:1}.wy-btn-group:after,.wy-btn-group:before{display:table;content:""}.wy-btn-group:after{clear:both}.wy-dropdown{position:relative;display:inline-block}.wy-dropdown-active .wy-dropdown-menu{display:block}.wy-dropdown-menu{position:absolute;left:0;display:none;float:left;top:100%;min-width:100%;background:#fcfcfc;z-index:100;border:1px solid #cfd7dd;box-shadow:0 2px 2px 0 rgba(0,0,0,.1);padding:12px}.wy-dropdown-menu>dd>a{display:block;clear:both;color:#404040;white-space:nowrap;font-size:90%;padding:0 12px;cursor:pointer}.wy-dropdown-menu>dd>a:hover{background:#2980b9;color:#fff}.wy-dropdown-menu>dd.divider{border-top:1px solid #cfd7dd;margin:6px 0}.wy-dropdown-menu>dd.search{padding-bottom:12px}.wy-dropdown-menu>dd.search input[type=search]{width:100%}.wy-dropdown-menu>dd.call-to-action{background:#e3e3e3;text-transform:uppercase;font-weight:500;font-size:80%}.wy-dropdown-menu>dd.call-to-action:hover{background:#e3e3e3}.wy-dropdown-menu>dd.call-to-action .btn{color:#fff}.wy-dropdown.wy-dropdown-up .wy-dropdown-menu{bottom:100%;top:auto;left:auto;right:0}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu{background:#fcfcfc;margin-top:2px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a{padding:6px 12px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a:hover{background:#2980b9;color:#fff}.wy-dropdown.wy-dropdown-left .wy-dropdown-menu{right:0;left:auto;text-align:right}.wy-dropdown-arrow:before{content:" ";border-bottom:5px solid #f5f5f5;border-left:5px solid transparent;border-right:5px solid transparent;position:absolute;display:block;top:-4px;left:50%;margin-left:-3px}.wy-dropdown-arrow.wy-dropdown-arrow-left:before{left:11px}.wy-form-stacked select{display:block}.wy-form-aligned .wy-help-inline,.wy-form-aligned input,.wy-form-aligned label,.wy-form-aligned select,.wy-form-aligned textarea{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-form-aligned .wy-control-group>label{display:inline-block;vertical-align:middle;width:10em;margin:6px 12px 0 0;float:left}.wy-form-aligned .wy-control{float:left}.wy-form-aligned .wy-control label{display:block}.wy-form-aligned .wy-control select{margin-top:6px}fieldset{margin:0}fieldset,legend{border:0;padding:0}legend{width:100%;white-space:normal;margin-bottom:24px;font-size:150%;*margin-left:-7px}label,legend{display:block}label{margin:0 0 .3125em;color:#333;font-size:90%}input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}.wy-control-group{margin-bottom:24px;max-width:1200px;margin-left:auto;margin-right:auto;*zoom:1}.wy-control-group:after,.wy-control-group:before{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group.wy-control-group-required>label:after{content:" *";color:#e74c3c}.wy-control-group .wy-form-full,.wy-control-group .wy-form-halves,.wy-control-group .wy-form-thirds{padding-bottom:12px}.wy-control-group .wy-form-full input[type=color],.wy-control-group .wy-form-full input[type=date],.wy-control-group .wy-form-full input[type=datetime-local],.wy-control-group .wy-form-full input[type=datetime],.wy-control-group .wy-form-full input[type=email],.wy-control-group .wy-form-full input[type=month],.wy-control-group .wy-form-full input[type=number],.wy-control-group .wy-form-full input[type=password],.wy-control-group .wy-form-full input[type=search],.wy-control-group .wy-form-full input[type=tel],.wy-control-group .wy-form-full input[type=text],.wy-control-group .wy-form-full input[type=time],.wy-control-group .wy-form-full input[type=url],.wy-control-group .wy-form-full input[type=week],.wy-control-group .wy-form-full select,.wy-control-group .wy-form-halves input[type=color],.wy-control-group .wy-form-halves input[type=date],.wy-control-group .wy-form-halves input[type=datetime-local],.wy-control-group .wy-form-halves input[type=datetime],.wy-control-group .wy-form-halves input[type=email],.wy-control-group .wy-form-halves input[type=month],.wy-control-group .wy-form-halves input[type=number],.wy-control-group .wy-form-halves input[type=password],.wy-control-group .wy-form-halves input[type=search],.wy-control-group .wy-form-halves input[type=tel],.wy-control-group .wy-form-halves input[type=text],.wy-control-group .wy-form-halves input[type=time],.wy-control-group .wy-form-halves input[type=url],.wy-control-group .wy-form-halves input[type=week],.wy-control-group .wy-form-halves select,.wy-control-group .wy-form-thirds input[type=color],.wy-control-group .wy-form-thirds input[type=date],.wy-control-group .wy-form-thirds input[type=datetime-local],.wy-control-group .wy-form-thirds input[type=datetime],.wy-control-group .wy-form-thirds input[type=email],.wy-control-group .wy-form-thirds input[type=month],.wy-control-group .wy-form-thirds input[type=number],.wy-control-group .wy-form-thirds input[type=password],.wy-control-group .wy-form-thirds input[type=search],.wy-control-group .wy-form-thirds input[type=tel],.wy-control-group .wy-form-thirds input[type=text],.wy-control-group .wy-form-thirds input[type=time],.wy-control-group .wy-form-thirds input[type=url],.wy-control-group .wy-form-thirds input[type=week],.wy-control-group .wy-form-thirds select{width:100%}.wy-control-group .wy-form-full{float:left;display:block;width:100%;margin-right:0}.wy-control-group .wy-form-full:last-child{margin-right:0}.wy-control-group .wy-form-halves{float:left;display:block;margin-right:2.35765%;width:48.82117%}.wy-control-group .wy-form-halves:last-child,.wy-control-group .wy-form-halves:nth-of-type(2n){margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(odd){clear:left}.wy-control-group .wy-form-thirds{float:left;display:block;margin-right:2.35765%;width:31.76157%}.wy-control-group .wy-form-thirds:last-child,.wy-control-group .wy-form-thirds:nth-of-type(3n){margin-right:0}.wy-control-group .wy-form-thirds:nth-of-type(3n+1){clear:left}.wy-control-group.wy-control-group-no-input .wy-control,.wy-control-no-input{margin:6px 0 0;font-size:90%}.wy-control-no-input{display:inline-block}.wy-control-group.fluid-input input[type=color],.wy-control-group.fluid-input input[type=date],.wy-control-group.fluid-input input[type=datetime-local],.wy-control-group.fluid-input input[type=datetime],.wy-control-group.fluid-input input[type=email],.wy-control-group.fluid-input input[type=month],.wy-control-group.fluid-input input[type=number],.wy-control-group.fluid-input input[type=password],.wy-control-group.fluid-input input[type=search],.wy-control-group.fluid-input input[type=tel],.wy-control-group.fluid-input input[type=text],.wy-control-group.fluid-input input[type=time],.wy-control-group.fluid-input input[type=url],.wy-control-group.fluid-input input[type=week]{width:100%}.wy-form-message-inline{padding-left:.3em;color:#666;font-size:90%}.wy-form-message{display:block;color:#999;font-size:70%;margin-top:.3125em;font-style:italic}.wy-form-message p{font-size:inherit;font-style:italic;margin-bottom:6px}.wy-form-message p:last-child{margin-bottom:0}input{line-height:normal}input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;*overflow:visible}input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week]{-webkit-appearance:none;padding:6px;display:inline-block;border:1px solid #ccc;font-size:80%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 3px #ddd;border-radius:0;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}input[type=datetime-local]{padding:.34375em .625em}input[disabled]{cursor:default}input[type=checkbox],input[type=radio]{padding:0;margin-right:.3125em;*height:13px;*width:13px}input[type=checkbox],input[type=radio],input[type=search]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}input[type=search]::-webkit-search-cancel-button,input[type=search]::-webkit-search-decoration{-webkit-appearance:none}input[type=color]:focus,input[type=date]:focus,input[type=datetime-local]:focus,input[type=datetime]:focus,input[type=email]:focus,input[type=month]:focus,input[type=number]:focus,input[type=password]:focus,input[type=search]:focus,input[type=tel]:focus,input[type=text]:focus,input[type=time]:focus,input[type=url]:focus,input[type=week]:focus{outline:0;outline:thin dotted\9;border-color:#333}input.no-focus:focus{border-color:#ccc!important}input[type=checkbox]:focus,input[type=file]:focus,input[type=radio]:focus{outline:thin dotted #333;outline:1px auto #129fea}input[type=color][disabled],input[type=date][disabled],input[type=datetime-local][disabled],input[type=datetime][disabled],input[type=email][disabled],input[type=month][disabled],input[type=number][disabled],input[type=password][disabled],input[type=search][disabled],input[type=tel][disabled],input[type=text][disabled],input[type=time][disabled],input[type=url][disabled],input[type=week][disabled]{cursor:not-allowed;background-color:#fafafa}input:focus:invalid,select:focus:invalid,textarea:focus:invalid{color:#e74c3c;border:1px solid #e74c3c}input:focus:invalid:focus,select:focus:invalid:focus,textarea:focus:invalid:focus{border-color:#e74c3c}input[type=checkbox]:focus:invalid:focus,input[type=file]:focus:invalid:focus,input[type=radio]:focus:invalid:focus{outline-color:#e74c3c}input.wy-input-large{padding:12px;font-size:100%}textarea{overflow:auto;vertical-align:top;width:100%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif}select,textarea{padding:.5em .625em;display:inline-block;border:1px solid #ccc;font-size:80%;box-shadow:inset 0 1px 3px #ddd;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}select{border:1px solid #ccc;background-color:#fff}select[multiple]{height:auto}select:focus,textarea:focus{outline:0}input[readonly],select[disabled],select[readonly],textarea[disabled],textarea[readonly]{cursor:not-allowed;background-color:#fafafa}input[type=checkbox][disabled],input[type=radio][disabled]{cursor:not-allowed}.wy-checkbox,.wy-radio{margin:6px 0;color:#404040;display:block}.wy-checkbox input,.wy-radio input{vertical-align:baseline}.wy-form-message-inline{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-input-prefix,.wy-input-suffix{white-space:nowrap;padding:6px}.wy-input-prefix .wy-input-context,.wy-input-suffix .wy-input-context{line-height:27px;padding:0 8px;display:inline-block;font-size:80%;background-color:#f3f6f6;border:1px solid #ccc;color:#999}.wy-input-suffix .wy-input-context{border-left:0}.wy-input-prefix .wy-input-context{border-right:0}.wy-switch{position:relative;display:block;height:24px;margin-top:12px;cursor:pointer}.wy-switch:before{left:0;top:0;width:36px;height:12px;background:#ccc}.wy-switch:after,.wy-switch:before{position:absolute;content:"";display:block;border-radius:4px;-webkit-transition:all .2s ease-in-out;-moz-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.wy-switch:after{width:18px;height:18px;background:#999;left:-3px;top:-3px}.wy-switch span{position:absolute;left:48px;display:block;font-size:12px;color:#ccc;line-height:1}.wy-switch.active:before{background:#1e8449}.wy-switch.active:after{left:24px;background:#27ae60}.wy-switch.disabled{cursor:not-allowed;opacity:.8}.wy-control-group.wy-control-group-error .wy-form-message,.wy-control-group.wy-control-group-error>label{color:#e74c3c}.wy-control-group.wy-control-group-error input[type=color],.wy-control-group.wy-control-group-error input[type=date],.wy-control-group.wy-control-group-error input[type=datetime-local],.wy-control-group.wy-control-group-error input[type=datetime],.wy-control-group.wy-control-group-error input[type=email],.wy-control-group.wy-control-group-error input[type=month],.wy-control-group.wy-control-group-error input[type=number],.wy-control-group.wy-control-group-error input[type=password],.wy-control-group.wy-control-group-error input[type=search],.wy-control-group.wy-control-group-error input[type=tel],.wy-control-group.wy-control-group-error input[type=text],.wy-control-group.wy-control-group-error input[type=time],.wy-control-group.wy-control-group-error input[type=url],.wy-control-group.wy-control-group-error input[type=week],.wy-control-group.wy-control-group-error textarea{border:1px solid #e74c3c}.wy-inline-validate{white-space:nowrap}.wy-inline-validate .wy-input-context{padding:.5em .625em;display:inline-block;font-size:80%}.wy-inline-validate.wy-inline-validate-success .wy-input-context{color:#27ae60}.wy-inline-validate.wy-inline-validate-danger .wy-input-context{color:#e74c3c}.wy-inline-validate.wy-inline-validate-warning .wy-input-context{color:#e67e22}.wy-inline-validate.wy-inline-validate-info .wy-input-context{color:#2980b9}.rotate-90{-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.rotate-180{-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.rotate-270{-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.mirror{-webkit-transform:scaleX(-1);-moz-transform:scaleX(-1);-ms-transform:scaleX(-1);-o-transform:scaleX(-1);transform:scaleX(-1)}.mirror.rotate-90{-webkit-transform:scaleX(-1) rotate(90deg);-moz-transform:scaleX(-1) rotate(90deg);-ms-transform:scaleX(-1) rotate(90deg);-o-transform:scaleX(-1) rotate(90deg);transform:scaleX(-1) rotate(90deg)}.mirror.rotate-180{-webkit-transform:scaleX(-1) rotate(180deg);-moz-transform:scaleX(-1) rotate(180deg);-ms-transform:scaleX(-1) rotate(180deg);-o-transform:scaleX(-1) rotate(180deg);transform:scaleX(-1) rotate(180deg)}.mirror.rotate-270{-webkit-transform:scaleX(-1) rotate(270deg);-moz-transform:scaleX(-1) rotate(270deg);-ms-transform:scaleX(-1) rotate(270deg);-o-transform:scaleX(-1) rotate(270deg);transform:scaleX(-1) rotate(270deg)}@media only screen and (max-width:480px){.wy-form button[type=submit]{margin:.7em 0 0}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=text],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week],.wy-form label{margin-bottom:.3em;display:block}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week]{margin-bottom:0}.wy-form-aligned .wy-control-group label{margin-bottom:.3em;text-align:left;display:block;width:100%}.wy-form-aligned .wy-control{margin:1.5em 0 0}.wy-form-message,.wy-form-message-inline,.wy-form .wy-help-inline{display:block;font-size:80%;padding:6px 0}}@media screen and (max-width:768px){.tablet-hide{display:none}}@media screen and (max-width:480px){.mobile-hide{display:none}}.float-left{float:left}.float-right{float:right}.full-width{width:100%}.rst-content table.docutils,.rst-content table.field-list,.wy-table{border-collapse:collapse;border-spacing:0;empty-cells:show;margin-bottom:24px}.rst-content table.docutils caption,.rst-content table.field-list caption,.wy-table caption{color:#000;font:italic 85%/1 arial,sans-serif;padding:1em 0;text-align:center}.rst-content table.docutils td,.rst-content table.docutils th,.rst-content table.field-list td,.rst-content table.field-list th,.wy-table td,.wy-table th{font-size:90%;margin:0;overflow:visible;padding:8px 16px}.rst-content table.docutils td:first-child,.rst-content table.docutils th:first-child,.rst-content table.field-list td:first-child,.rst-content table.field-list th:first-child,.wy-table td:first-child,.wy-table th:first-child{border-left-width:0}.rst-content table.docutils thead,.rst-content table.field-list thead,.wy-table thead{color:#000;text-align:left;vertical-align:bottom;white-space:nowrap}.rst-content table.docutils thead th,.rst-content table.field-list thead th,.wy-table thead th{font-weight:700;border-bottom:2px solid #e1e4e5}.rst-content table.docutils td,.rst-content table.field-list td,.wy-table td{background-color:transparent;vertical-align:middle}.rst-content table.docutils td p,.rst-content table.field-list td p,.wy-table td p{line-height:18px}.rst-content table.docutils td p:last-child,.rst-content table.field-list td p:last-child,.wy-table td p:last-child{margin-bottom:0}.rst-content table.docutils .wy-table-cell-min,.rst-content table.field-list .wy-table-cell-min,.wy-table .wy-table-cell-min{width:1%;padding-right:0}.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-content table.field-list .wy-table-cell-min input[type=checkbox],.wy-table .wy-table-cell-min input[type=checkbox]{margin:0}.wy-table-secondary{color:grey;font-size:90%}.wy-table-tertiary{color:grey;font-size:80%}.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td,.wy-table-backed,.wy-table-odd td,.wy-table-striped tr:nth-child(2n-1) td{background-color:#f3f6f6}.rst-content table.docutils,.wy-table-bordered-all{border:1px solid #e1e4e5}.rst-content table.docutils td,.wy-table-bordered-all td{border-bottom:1px solid #e1e4e5;border-left:1px solid #e1e4e5}.rst-content table.docutils tbody>tr:last-child td,.wy-table-bordered-all tbody>tr:last-child td{border-bottom-width:0}.wy-table-bordered{border:1px solid #e1e4e5}.wy-table-bordered-rows td{border-bottom:1px solid #e1e4e5}.wy-table-bordered-rows tbody>tr:last-child td{border-bottom-width:0}.wy-table-horizontal td,.wy-table-horizontal th{border-width:0 0 1px;border-bottom:1px solid #e1e4e5}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-responsive{margin-bottom:24px;max-width:100%;overflow:auto}.wy-table-responsive table{margin-bottom:0!important}.wy-table-responsive table td,.wy-table-responsive table th{white-space:nowrap}a{color:#2980b9;text-decoration:none;cursor:pointer}a:hover{color:#3091d1}a:visited{color:#9b59b6}html{height:100%}body,html{overflow-x:hidden}body{font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;font-weight:400;color:#404040;min-height:100%;background:#edf0f2}.wy-text-left{text-align:left}.wy-text-center{text-align:center}.wy-text-right{text-align:right}.wy-text-large{font-size:120%}.wy-text-normal{font-size:100%}.wy-text-small,small{font-size:80%}.wy-text-strike{text-decoration:line-through}.wy-text-warning{color:#e67e22!important}a.wy-text-warning:hover{color:#eb9950!important}.wy-text-info{color:#2980b9!important}a.wy-text-info:hover{color:#409ad5!important}.wy-text-success{color:#27ae60!important}a.wy-text-success:hover{color:#36d278!important}.wy-text-danger{color:#e74c3c!important}a.wy-text-danger:hover{color:#ed7669!important}.wy-text-neutral{color:#404040!important}a.wy-text-neutral:hover{color:#595959!important}.rst-content .toctree-wrapper>p.caption,h1,h2,h3,h4,h5,h6,legend{margin-top:0;font-weight:700;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif}p{line-height:24px;font-size:16px;margin:0 0 24px}h1{font-size:175%}.rst-content .toctree-wrapper>p.caption,h2{font-size:150%}h3{font-size:125%}h4{font-size:115%}h5{font-size:110%}h6{font-size:100%}hr{display:block;height:1px;border:0;border-top:1px solid #e1e4e5;margin:24px 0;padding:0}.rst-content code,.rst-content tt,code{white-space:nowrap;max-width:100%;background:#fff;border:1px solid #e1e4e5;font-size:75%;padding:0 5px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#e74c3c;overflow-x:auto}.rst-content tt.code-large,code.code-large{font-size:90%}.rst-content .section ul,.rst-content .toctree-wrapper ul,.wy-plain-list-disc,article ul{list-style:disc;line-height:24px;margin-bottom:24px}.rst-content .section ul li,.rst-content .toctree-wrapper ul li,.wy-plain-list-disc li,article ul li{list-style:disc;margin-left:24px}.rst-content .section ul li p:last-child,.rst-content .section ul li ul,.rst-content .toctree-wrapper ul li p:last-child,.rst-content .toctree-wrapper ul li ul,.wy-plain-list-disc li p:last-child,.wy-plain-list-disc li ul,article ul li p:last-child,article ul li ul{margin-bottom:0}.rst-content .section ul li li,.rst-content .toctree-wrapper ul li li,.wy-plain-list-disc li li,article ul li li{list-style:circle}.rst-content .section ul li li li,.rst-content .toctree-wrapper ul li li li,.wy-plain-list-disc li li li,article ul li li li{list-style:square}.rst-content .section ul li ol li,.rst-content .toctree-wrapper ul li ol li,.wy-plain-list-disc li ol li,article ul li ol li{list-style:decimal}.rst-content .section ol,.rst-content ol.arabic,.wy-plain-list-decimal,article ol{list-style:decimal;line-height:24px;margin-bottom:24px}.rst-content .section ol li,.rst-content ol.arabic li,.wy-plain-list-decimal li,article ol li{list-style:decimal;margin-left:24px}.rst-content .section ol li p:last-child,.rst-content .section ol li ul,.rst-content ol.arabic li p:last-child,.rst-content ol.arabic li ul,.wy-plain-list-decimal li p:last-child,.wy-plain-list-decimal li ul,article ol li p:last-child,article ol li ul{margin-bottom:0}.rst-content .section ol li ul li,.rst-content ol.arabic li ul li,.wy-plain-list-decimal li ul li,article ol li ul li{list-style:disc}.wy-breadcrumbs{*zoom:1}.wy-breadcrumbs:after,.wy-breadcrumbs:before{display:table;content:""}.wy-breadcrumbs:after{clear:both}.wy-breadcrumbs li{display:inline-block}.wy-breadcrumbs li.wy-breadcrumbs-aside{float:right}.wy-breadcrumbs li a{display:inline-block;padding:5px}.wy-breadcrumbs li a:first-child{padding-left:0}.rst-content .wy-breadcrumbs li tt,.wy-breadcrumbs li .rst-content tt,.wy-breadcrumbs li code{padding:5px;border:none;background:none}.rst-content .wy-breadcrumbs li tt.literal,.wy-breadcrumbs li .rst-content tt.literal,.wy-breadcrumbs li code.literal{color:#404040}.wy-breadcrumbs-extra{margin-bottom:0;color:#b3b3b3;font-size:80%;display:inline-block}@media screen and (max-width:480px){.wy-breadcrumbs-extra,.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}@media print{.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}html{font-size:16px}.wy-affix{position:fixed;top:1.618em}.wy-menu a:hover{text-decoration:none}.wy-menu-horiz{*zoom:1}.wy-menu-horiz:after,.wy-menu-horiz:before{display:table;content:""}.wy-menu-horiz:after{clear:both}.wy-menu-horiz li,.wy-menu-horiz ul{display:inline-block}.wy-menu-horiz li:hover{background:hsla(0,0%,100%,.1)}.wy-menu-horiz li.divide-left{border-left:1px solid #404040}.wy-menu-horiz li.divide-right{border-right:1px solid #404040}.wy-menu-horiz a{height:32px;display:inline-block;line-height:32px;padding:0 16px}.wy-menu-vertical{width:300px}.wy-menu-vertical header,.wy-menu-vertical p.caption{color:#55a5d9;height:32px;line-height:32px;padding:0 1.618em;margin:12px 0 0;display:block;font-weight:700;text-transform:uppercase;font-size:85%;white-space:nowrap}.wy-menu-vertical ul{margin-bottom:0}.wy-menu-vertical li.divide-top{border-top:1px solid #404040}.wy-menu-vertical li.divide-bottom{border-bottom:1px solid #404040}.wy-menu-vertical li.current{background:#e3e3e3}.wy-menu-vertical li.current a{color:grey;border-right:1px solid #c9c9c9;padding:.4045em 2.427em}.wy-menu-vertical li.current a:hover{background:#d6d6d6}.rst-content .wy-menu-vertical li tt,.wy-menu-vertical li .rst-content tt,.wy-menu-vertical li code{border:none;background:inherit;color:inherit;padding-left:0;padding-right:0}.wy-menu-vertical li span.toctree-expand{display:block;float:left;margin-left:-1.2em;font-size:.8em;line-height:1.6em;color:#4d4d4d}.wy-menu-vertical li.current>a,.wy-menu-vertical li.on a{color:#404040;font-weight:700;position:relative;background:#fcfcfc;border:none;padding:.4045em 1.618em}.wy-menu-vertical li.current>a:hover,.wy-menu-vertical li.on a:hover{background:#fcfcfc}.wy-menu-vertical li.current>a:hover span.toctree-expand,.wy-menu-vertical li.on a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand{display:block;font-size:.8em;line-height:1.6em;color:#333}.wy-menu-vertical li.toctree-l1.current>a{border-bottom:1px solid #c9c9c9;border-top:1px solid #c9c9c9}.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .toctree-l11>ul{display:none}.wy-menu-vertical .toctree-l1.current .current.toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .current.toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .current.toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .current.toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .current.toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .current.toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .current.toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .current.toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .current.toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .current.toctree-l11>ul{display:block}.wy-menu-vertical li.toctree-l3,.wy-menu-vertical li.toctree-l4{font-size:.9em}.wy-menu-vertical li.toctree-l2 a,.wy-menu-vertical li.toctree-l3 a,.wy-menu-vertical li.toctree-l4 a,.wy-menu-vertical li.toctree-l5 a,.wy-menu-vertical li.toctree-l6 a,.wy-menu-vertical li.toctree-l7 a,.wy-menu-vertical li.toctree-l8 a,.wy-menu-vertical li.toctree-l9 a,.wy-menu-vertical li.toctree-l10 a{color:#404040}.wy-menu-vertical li.toctree-l2 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l3 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l4 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l5 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l6 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l7 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l8 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l9 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l10 a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{display:block}.wy-menu-vertical li.toctree-l2.current>a{padding:.4045em 2.427em}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current>a{padding:.4045em 4.045em}.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current>a{padding:.4045em 5.663em}.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current>a{padding:.4045em 7.281em}.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current>a{padding:.4045em 8.899em}.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current>a{padding:.4045em 10.517em}.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current>a{padding:.4045em 12.135em}.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current>a{padding:.4045em 13.753em}.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current>a{padding:.4045em 15.371em}.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{padding:.4045em 16.989em}.wy-menu-vertical li.toctree-l2.current>a,.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{background:#c9c9c9}.wy-menu-vertical li.toctree-l2 span.toctree-expand{color:#a3a3a3}.wy-menu-vertical li.toctree-l3.current>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{background:#bdbdbd}.wy-menu-vertical li.toctree-l3 span.toctree-expand{color:#969696}.wy-menu-vertical li.current ul{display:block}.wy-menu-vertical li ul{margin-bottom:0;display:none}.wy-menu-vertical li ul li a{margin-bottom:0;color:#d9d9d9;font-weight:400}.wy-menu-vertical a{line-height:18px;padding:.4045em 1.618em;display:block;position:relative;font-size:90%;color:#d9d9d9}.wy-menu-vertical a:hover{background-color:#4e4a4a;cursor:pointer}.wy-menu-vertical a:hover span.toctree-expand{color:#d9d9d9}.wy-menu-vertical a:active{background-color:#2980b9;cursor:pointer;color:#fff}.wy-menu-vertical a:active span.toctree-expand{color:#fff}.wy-side-nav-search{display:block;width:300px;padding:.809em;margin-bottom:.809em;z-index:200;background-color:#2980b9;text-align:center;color:#fcfcfc}.wy-side-nav-search input[type=text]{width:100%;border-radius:50px;padding:6px 12px;border-color:#2472a4}.wy-side-nav-search img{display:block;margin:auto auto .809em;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a{color:#fcfcfc;font-size:100%;font-weight:700;display:inline-block;padding:4px 6px;margin-bottom:.809em}.wy-side-nav-search .wy-dropdown>a:hover,.wy-side-nav-search>a:hover{background:hsla(0,0%,100%,.1)}.wy-side-nav-search .wy-dropdown>a img.logo,.wy-side-nav-search>a img.logo{display:block;margin:0 auto;height:auto;width:auto;border-radius:0;max-width:100%;background:transparent}.wy-side-nav-search .wy-dropdown>a.icon img.logo,.wy-side-nav-search>a.icon img.logo{margin-top:.85em}.wy-side-nav-search>div.version{margin-top:-.4045em;margin-bottom:.809em;font-weight:400;color:hsla(0,0%,100%,.3)}.wy-nav .wy-menu-vertical header{color:#2980b9}.wy-nav .wy-menu-vertical a{color:#b3b3b3}.wy-nav .wy-menu-vertical a:hover{background-color:#2980b9;color:#fff}[data-menu-wrap]{-webkit-transition:all .2s ease-in;-moz-transition:all .2s ease-in;transition:all .2s ease-in;position:absolute;opacity:1;width:100%;opacity:0}[data-menu-wrap].move-center{left:0;right:auto;opacity:1}[data-menu-wrap].move-left{right:auto;left:-100%;opacity:0}[data-menu-wrap].move-right{right:-100%;left:auto;opacity:0}.wy-body-for-nav{background:#fcfcfc}.wy-grid-for-nav{position:absolute;width:100%;height:100%}.wy-nav-side{position:fixed;top:0;bottom:0;left:0;padding-bottom:2em;width:300px;overflow-x:hidden;overflow-y:hidden;min-height:100%;color:#9b9b9b;background:#343131;z-index:200}.wy-side-scroll{width:320px;position:relative;overflow-x:hidden;overflow-y:scroll;height:100%}.wy-nav-top{display:none;background:#2980b9;color:#fff;padding:.4045em .809em;position:relative;line-height:50px;text-align:center;font-size:100%;*zoom:1}.wy-nav-top:after,.wy-nav-top:before{display:table;content:""}.wy-nav-top:after{clear:both}.wy-nav-top a{color:#fff;font-weight:700}.wy-nav-top img{margin-right:12px;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-nav-top i{font-size:30px;float:left;cursor:pointer;padding-top:inherit}.wy-nav-content-wrap{margin-left:300px;background:#fcfcfc;min-height:100%}.wy-nav-content{padding:1.618em 3.236em;height:100%;max-width:800px;margin:auto}.wy-body-mask{position:fixed;width:100%;height:100%;background:rgba(0,0,0,.2);display:none;z-index:499}.wy-body-mask.on{display:block}footer{color:grey}footer p{margin-bottom:12px}.rst-content footer span.commit tt,footer span.commit .rst-content tt,footer span.commit code{padding:0;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:1em;background:none;border:none;color:grey}.rst-footer-buttons{*zoom:1}.rst-footer-buttons:after,.rst-footer-buttons:before{width:100%;display:table;content:""}.rst-footer-buttons:after{clear:both}.rst-breadcrumbs-buttons{margin-top:12px;*zoom:1}.rst-breadcrumbs-buttons:after,.rst-breadcrumbs-buttons:before{display:table;content:""}.rst-breadcrumbs-buttons:after{clear:both}#search-results .search li{margin-bottom:24px;border-bottom:1px solid #e1e4e5;padding-bottom:24px}#search-results .search li:first-child{border-top:1px solid #e1e4e5;padding-top:24px}#search-results .search li a{font-size:120%;margin-bottom:12px;display:inline-block}#search-results .context{color:grey;font-size:90%}.genindextable li>ul{margin-left:24px}@media screen and (max-width:768px){.wy-body-for-nav{background:#fcfcfc}.wy-nav-top{display:block}.wy-nav-side{left:-300px}.wy-nav-side.shift{width:85%;left:0}.wy-menu.wy-menu-vertical,.wy-side-nav-search,.wy-side-scroll{width:auto}.wy-nav-content-wrap{margin-left:0}.wy-nav-content-wrap .wy-nav-content{padding:1.618em}.wy-nav-content-wrap.shift{position:fixed;min-width:100%;left:85%;top:0;height:100%;overflow:hidden}}@media screen and (min-width:1100px){.wy-nav-content-wrap{background:rgba(0,0,0,.05)}.wy-nav-content{margin:0;background:#fcfcfc}}@media print{.rst-versions,.wy-nav-side,footer{display:none}.wy-nav-content-wrap{margin-left:0}}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60;*zoom:1}.rst-versions .rst-current-version:after,.rst-versions .rst-current-version:before{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-content .code-block-caption .rst-versions .rst-current-version .headerlink,.rst-content .rst-versions .rst-current-version .admonition-title,.rst-content code.download .rst-versions .rst-current-version span:first-child,.rst-content dl dt .rst-versions .rst-current-version .headerlink,.rst-content h1 .rst-versions .rst-current-version .headerlink,.rst-content h2 .rst-versions .rst-current-version .headerlink,.rst-content h3 .rst-versions .rst-current-version .headerlink,.rst-content h4 .rst-versions .rst-current-version .headerlink,.rst-content h5 .rst-versions .rst-current-version .headerlink,.rst-content h6 .rst-versions .rst-current-version .headerlink,.rst-content p.caption .rst-versions .rst-current-version .headerlink,.rst-content table>caption .rst-versions .rst-current-version .headerlink,.rst-content tt.download .rst-versions .rst-current-version span:first-child,.rst-versions .rst-current-version .fa,.rst-versions .rst-current-version .icon,.rst-versions .rst-current-version .rst-content .admonition-title,.rst-versions .rst-current-version .rst-content .code-block-caption .headerlink,.rst-versions .rst-current-version .rst-content code.download span:first-child,.rst-versions .rst-current-version .rst-content dl dt .headerlink,.rst-versions .rst-current-version .rst-content h1 .headerlink,.rst-versions .rst-current-version .rst-content h2 .headerlink,.rst-versions .rst-current-version .rst-content h3 .headerlink,.rst-versions .rst-current-version .rst-content h4 .headerlink,.rst-versions .rst-current-version .rst-content h5 .headerlink,.rst-versions .rst-current-version .rst-content h6 .headerlink,.rst-versions .rst-current-version .rst-content p.caption .headerlink,.rst-versions .rst-current-version .rst-content table>caption .headerlink,.rst-versions .rst-current-version .rst-content tt.download span:first-child,.rst-versions .rst-current-version .wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li .rst-versions .rst-current-version span.toctree-expand{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}}.rst-content img{max-width:100%;height:auto}.rst-content div.figure{margin-bottom:24px}.rst-content div.figure p.caption{font-style:italic}.rst-content div.figure p:last-child.caption{margin-bottom:0}.rst-content div.figure.align-center{text-align:center}.rst-content .section>a>img,.rst-content .section>img{margin-bottom:24px}.rst-content abbr[title]{text-decoration:none}.rst-content.style-external-links a.reference.external:after{font-family:FontAwesome;content:"\f08e";color:#b3b3b3;vertical-align:super;font-size:60%;margin:0 .2em}.rst-content blockquote{margin-left:24px;line-height:24px;margin-bottom:24px}.rst-content pre.literal-block{white-space:pre;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;display:block;overflow:auto}.rst-content div[class^=highlight],.rst-content pre.literal-block{border:1px solid #e1e4e5;overflow-x:auto;margin:1px 0 24px}.rst-content div[class^=highlight] div[class^=highlight],.rst-content pre.literal-block div[class^=highlight]{padding:0;border:none;margin:0}.rst-content div[class^=highlight] td.code{width:100%}.rst-content .linenodiv pre{border-right:1px solid #e6e9ea;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;user-select:none;pointer-events:none}.rst-content div[class^=highlight] pre{white-space:pre;margin:0;padding:12px;display:block;overflow:auto}.rst-content div[class^=highlight] pre .hll{display:block;margin:0 -12px;padding:0 12px}.rst-content .linenodiv pre,.rst-content div[class^=highlight] pre,.rst-content pre.literal-block{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:12px;line-height:1.4}.rst-content div.highlight .gp{user-select:none;pointer-events:none}.rst-content .code-block-caption{font-style:italic;font-size:85%;line-height:1;padding:1em 0;text-align:center}@media print{.rst-content .codeblock,.rst-content div[class^=highlight],.rst-content div[class^=highlight] pre{white-space:pre-wrap}}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning{clear:both}.rst-content .admonition-todo .last,.rst-content .admonition-todo>:last-child,.rst-content .admonition .last,.rst-content .admonition>:last-child,.rst-content .attention .last,.rst-content .attention>:last-child,.rst-content .caution .last,.rst-content .caution>:last-child,.rst-content .danger .last,.rst-content .danger>:last-child,.rst-content .error .last,.rst-content .error>:last-child,.rst-content .hint .last,.rst-content .hint>:last-child,.rst-content .important .last,.rst-content .important>:last-child,.rst-content .note .last,.rst-content .note>:last-child,.rst-content .seealso .last,.rst-content .seealso>:last-child,.rst-content .tip .last,.rst-content .tip>:last-child,.rst-content .warning .last,.rst-content .warning>:last-child{margin-bottom:0}.rst-content .admonition-title:before{margin-right:4px}.rst-content .admonition table{border-color:rgba(0,0,0,.1)}.rst-content .admonition table td,.rst-content .admonition table th{background:transparent!important;border-color:rgba(0,0,0,.1)!important}.rst-content .section ol.loweralpha,.rst-content .section ol.loweralpha>li{list-style:lower-alpha}.rst-content .section ol.upperalpha,.rst-content .section ol.upperalpha>li{list-style:upper-alpha}.rst-content .section ol li>*,.rst-content .section ul li>*{margin-top:12px;margin-bottom:12px}.rst-content .section ol li>:first-child,.rst-content .section ul li>:first-child{margin-top:0}.rst-content .section ol li>p,.rst-content .section ol li>p:last-child,.rst-content .section ul li>p,.rst-content .section ul li>p:last-child{margin-bottom:12px}.rst-content .section ol li>p:only-child,.rst-content .section ol li>p:only-child:last-child,.rst-content .section ul li>p:only-child,.rst-content .section ul li>p:only-child:last-child{margin-bottom:0}.rst-content .section ol li>ol,.rst-content .section ol li>ul,.rst-content .section ul li>ol,.rst-content .section ul li>ul{margin-bottom:12px}.rst-content .section ol.simple li>*,.rst-content .section ol.simple li ol,.rst-content .section ol.simple li ul,.rst-content .section ul.simple li>*,.rst-content .section ul.simple li ol,.rst-content .section ul.simple li ul{margin-top:0;margin-bottom:0}.rst-content .line-block{margin-left:0;margin-bottom:24px;line-height:24px}.rst-content .line-block .line-block{margin-left:24px;margin-bottom:0}.rst-content .topic-title{font-weight:700;margin-bottom:12px}.rst-content .toc-backref{color:#404040}.rst-content .align-right{float:right;margin:0 0 24px 24px}.rst-content .align-left{float:left;margin:0 24px 24px 0}.rst-content .align-center{margin:auto}.rst-content .align-center:not(table){display:block}.rst-content .code-block-caption .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink{visibility:hidden;font-size:14px}.rst-content .code-block-caption .headerlink:after,.rst-content .toctree-wrapper>p.caption .headerlink:after,.rst-content dl dt .headerlink:after,.rst-content h1 .headerlink:after,.rst-content h2 .headerlink:after,.rst-content h3 .headerlink:after,.rst-content h4 .headerlink:after,.rst-content h5 .headerlink:after,.rst-content h6 .headerlink:after,.rst-content p.caption .headerlink:after,.rst-content table>caption .headerlink:after{content:"\f0c1";font-family:FontAwesome}.rst-content .code-block-caption:hover .headerlink:after,.rst-content .toctree-wrapper>p.caption:hover .headerlink:after,.rst-content dl dt:hover .headerlink:after,.rst-content h1:hover .headerlink:after,.rst-content h2:hover .headerlink:after,.rst-content h3:hover .headerlink:after,.rst-content h4:hover .headerlink:after,.rst-content h5:hover .headerlink:after,.rst-content h6:hover .headerlink:after,.rst-content p.caption:hover .headerlink:after,.rst-content table>caption:hover .headerlink:after{visibility:visible}.rst-content table>caption .headerlink:after{font-size:12px}.rst-content .centered{text-align:center}.rst-content .sidebar{float:right;width:40%;display:block;margin:0 0 24px 24px;padding:24px;background:#f3f6f6;border:1px solid #e1e4e5}.rst-content .sidebar dl,.rst-content .sidebar p,.rst-content .sidebar ul{font-size:90%}.rst-content .sidebar .last,.rst-content .sidebar>:last-child{margin-bottom:0}.rst-content .sidebar .sidebar-title{display:block;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif;font-weight:700;background:#e1e4e5;padding:6px 12px;margin:-24px -24px 24px;font-size:100%}.rst-content .highlighted{background:#f1c40f;box-shadow:0 0 0 2px #f1c40f;display:inline;font-weight:700}.rst-content .citation-reference,.rst-content .footnote-reference{vertical-align:baseline;position:relative;top:-.4em;line-height:0;font-size:90%}.rst-content .hlist{width:100%}html.writer-html4 .rst-content table.docutils.citation,html.writer-html4 .rst-content table.docutils.footnote{background:none;border:none}html.writer-html4 .rst-content table.docutils.citation td,html.writer-html4 .rst-content table.docutils.citation tr,html.writer-html4 .rst-content table.docutils.footnote td,html.writer-html4 .rst-content table.docutils.footnote tr{border:none;background-color:transparent!important;white-space:normal}html.writer-html4 .rst-content table.docutils.citation td.label,html.writer-html4 .rst-content table.docutils.footnote td.label{padding-left:0;padding-right:0;vertical-align:top}html.writer-html5 .rst-content dl dt span.classifier:before{content:" : "}html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{display:grid;grid-template-columns:max-content auto}html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{padding-left:1rem}html.writer-html5 .rst-content dl.field-list>dt:after,html.writer-html5 .rst-content dl.footnote>dt:after{content:":"}html.writer-html5 .rst-content dl.field-list>dd,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dd,html.writer-html5 .rst-content dl.footnote>dt{margin-bottom:0}html.writer-html5 .rst-content dl.footnote{font-size:.9rem}html.writer-html5 .rst-content dl.footnote>dt{margin:0 .5rem .5rem 0;line-height:1.2rem;word-break:break-all;font-weight:400}html.writer-html5 .rst-content dl.footnote>dt>span.brackets{margin-right:.5rem}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:before{content:"["}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:after{content:"]"}html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref{font-style:italic}html.writer-html5 .rst-content dl.footnote>dd{margin:0 0 .5rem;line-height:1.2rem}html.writer-html5 .rst-content dl.footnote>dd p,html.writer-html5 .rst-content dl.option-list kbd{font-size:.9rem}.rst-content table.docutils.footnote,html.writer-html4 .rst-content table.docutils.citation,html.writer-html5 .rst-content dl.footnote{color:grey}.rst-content table.docutils.footnote code,.rst-content table.docutils.footnote tt,html.writer-html4 .rst-content table.docutils.citation code,html.writer-html4 .rst-content table.docutils.citation tt,html.writer-html5 .rst-content dl.footnote code,html.writer-html5 .rst-content dl.footnote tt{color:#555}.rst-content .wy-table-responsive.citation,.rst-content .wy-table-responsive.footnote{margin-bottom:0}.rst-content .wy-table-responsive.citation+:not(.citation),.rst-content .wy-table-responsive.footnote+:not(.footnote){margin-top:24px}.rst-content .wy-table-responsive.citation:last-child,.rst-content .wy-table-responsive.footnote:last-child{margin-bottom:24px}.rst-content table.docutils th{border-color:#e1e4e5}html.writer-html5 .rst-content table.docutils th{border:1px solid #e1e4e5}html.writer-html5 .rst-content table.docutils td>p,html.writer-html5 .rst-content table.docutils th>p{line-height:1rem;margin-bottom:0;font-size:.9rem}.rst-content table.docutils td .last,.rst-content table.docutils td .last>:last-child{margin-bottom:0}.rst-content table.field-list,.rst-content table.field-list td{border:none}.rst-content table.field-list td p{font-size:inherit;line-height:inherit}.rst-content table.field-list td>strong{display:inline-block}.rst-content table.field-list .field-name{padding-right:10px;text-align:left;white-space:nowrap}.rst-content table.field-list .field-body{text-align:left}.rst-content code,.rst-content tt{color:#000;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;padding:2px 5px}.rst-content code big,.rst-content code em,.rst-content tt big,.rst-content tt em{font-size:100%!important;line-height:normal}.rst-content code.literal,.rst-content tt.literal{color:#e74c3c}.rst-content code.xref,.rst-content tt.xref,a .rst-content code,a .rst-content tt{font-weight:700;color:#404040}.rst-content kbd,.rst-content pre,.rst-content samp{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace}.rst-content a code,.rst-content a tt{color:#2980b9}.rst-content dl{margin-bottom:24px}.rst-content dl dt{font-weight:700;margin-bottom:12px}.rst-content dl ol,.rst-content dl p,.rst-content dl table,.rst-content dl ul{margin-bottom:12px}.rst-content dl dd{margin:0 0 12px 24px;line-height:24px}html.writer-html4 .rst-content dl:not(.docutils),html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple){margin-bottom:24px}html.writer-html4 .rst-content dl:not(.docutils)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt{display:table;margin:6px 0;font-size:90%;line-height:normal;background:#e7f2fa;color:#2980b9;border-top:3px solid #6ab0de;padding:6px;position:relative}html.writer-html4 .rst-content dl:not(.docutils)>dt:before,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:before{color:#6ab0de}html.writer-html4 .rst-content dl:not(.docutils)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt{margin-bottom:6px;border:none;border-left:3px solid #ccc;background:#f0f0f0;color:#555}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils)>dt:first-child,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:first-child{margin-top:0}html.writer-html4 .rst-content dl:not(.docutils) code,html.writer-html4 .rst-content dl:not(.docutils) tt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) code.descclassname,html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descclassname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{background-color:transparent;border:none;padding:0;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .optional,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .optional{display:inline-block;padding:0 4px;color:#000;font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .property,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .property{display:inline-block;padding-right:8px}.rst-content .viewcode-back,.rst-content .viewcode-link{display:inline-block;color:#27ae60;font-size:80%;padding-left:24px}.rst-content .viewcode-back{display:block;float:right}.rst-content p.rubric{margin-bottom:12px;font-weight:700}.rst-content code.download,.rst-content tt.download{background:inherit;padding:inherit;font-weight:400;font-family:inherit;font-size:inherit;color:inherit;border:inherit;white-space:inherit}.rst-content code.download span:first-child,.rst-content tt.download span:first-child{-webkit-font-smoothing:subpixel-antialiased}.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{margin-right:4px}.rst-content .guilabel{border:1px solid #7fbbe3;background:#e7f2fa;font-size:80%;font-weight:700;border-radius:4px;padding:2.4px 6px;margin:auto 2px}.rst-content .versionmodified{font-style:italic}@media screen and (max-width:480px){.rst-content .sidebar{width:100%}}span[id*=MathJax-Span]{color:#404040}.math{text-align:center}@font-face{font-family:Lato;src:url(fonts/lato-normal.woff2?bd03a2cc277bbbc338d464e679fe9942) format("woff2"),url(fonts/lato-normal.woff?27bd77b9162d388cb8d4c4217c7c5e2a) format("woff");font-weight:400;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold.woff2?cccb897485813c7c256901dbca54ecf2) format("woff2"),url(fonts/lato-bold.woff?d878b6c29b10beca227e9eef4246111b) format("woff");font-weight:700;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold-italic.woff2?0b6bb6725576b072c5d0b02ecdd1900d) format("woff2"),url(fonts/lato-bold-italic.woff?9c7e4e9eb485b4a121c760e61bc3707c) format("woff");font-weight:700;font-style:italic;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-normal-italic.woff2?4eb103b4d12be57cb1d040ed5e162e9d) format("woff2"),url(fonts/lato-normal-italic.woff?f28f2d6482446544ef1ea1ccc6dd5892) format("woff");font-weight:400;font-style:italic;font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:400;src:url(fonts/Roboto-Slab-Regular.woff2?7abf5b8d04d26a2cafea937019bca958) format("woff2"),url(fonts/Roboto-Slab-Regular.woff?c1be9284088d487c5e3ff0a10a92e58c) format("woff");font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:700;src:url(fonts/Roboto-Slab-Bold.woff2?9984f4a9bda09be08e83f2506954adbe) format("woff2"),url(fonts/Roboto-Slab-Bold.woff?bed5564a116b05148e3b3bea6fb1162a) format("woff");font-display:block} \ No newline at end of file diff --git a/en/3.0/_static/doctools.js b/en/3.0/_static/doctools.js new file mode 100644 index 0000000000..527b876ca6 --- /dev/null +++ b/en/3.0/_static/doctools.js @@ -0,0 +1,156 @@ +/* + * doctools.js + * ~~~~~~~~~~~ + * + * Base JavaScript utilities for all Sphinx HTML documentation. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ +"use strict"; + +const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([ + "TEXTAREA", + "INPUT", + "SELECT", + "BUTTON", +]); + +const _ready = (callback) => { + if (document.readyState !== "loading") { + callback(); + } else { + document.addEventListener("DOMContentLoaded", callback); + } +}; + +/** + * Small JavaScript module for the documentation. + */ +const Documentation = { + init: () => { + Documentation.initDomainIndexTable(); + Documentation.initOnKeyListeners(); + }, + + /** + * i18n support + */ + TRANSLATIONS: {}, + PLURAL_EXPR: (n) => (n === 1 ? 0 : 1), + LOCALE: "unknown", + + // gettext and ngettext don't access this so that the functions + // can safely bound to a different name (_ = Documentation.gettext) + gettext: (string) => { + const translated = Documentation.TRANSLATIONS[string]; + switch (typeof translated) { + case "undefined": + return string; // no translation + case "string": + return translated; // translation exists + default: + return translated[0]; // (singular, plural) translation tuple exists + } + }, + + ngettext: (singular, plural, n) => { + const translated = Documentation.TRANSLATIONS[singular]; + if (typeof translated !== "undefined") + return translated[Documentation.PLURAL_EXPR(n)]; + return n === 1 ? singular : plural; + }, + + addTranslations: (catalog) => { + Object.assign(Documentation.TRANSLATIONS, catalog.messages); + Documentation.PLURAL_EXPR = new Function( + "n", + `return (${catalog.plural_expr})` + ); + Documentation.LOCALE = catalog.locale; + }, + + /** + * helper function to focus on search bar + */ + focusSearchBar: () => { + document.querySelectorAll("input[name=q]")[0]?.focus(); + }, + + /** + * Initialise the domain index toggle buttons + */ + initDomainIndexTable: () => { + const toggler = (el) => { + const idNumber = el.id.substr(7); + const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`); + if (el.src.substr(-9) === "minus.png") { + el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`; + toggledRows.forEach((el) => (el.style.display = "none")); + } else { + el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`; + toggledRows.forEach((el) => (el.style.display = "")); + } + }; + + const togglerElements = document.querySelectorAll("img.toggler"); + togglerElements.forEach((el) => + el.addEventListener("click", (event) => toggler(event.currentTarget)) + ); + togglerElements.forEach((el) => (el.style.display = "")); + if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler); + }, + + initOnKeyListeners: () => { + // only install a listener if it is really needed + if ( + !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS && + !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS + ) + return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.altKey || event.ctrlKey || event.metaKey) return; + + if (!event.shiftKey) { + switch (event.key) { + case "ArrowLeft": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const prevLink = document.querySelector('link[rel="prev"]'); + if (prevLink && prevLink.href) { + window.location.href = prevLink.href; + event.preventDefault(); + } + break; + case "ArrowRight": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const nextLink = document.querySelector('link[rel="next"]'); + if (nextLink && nextLink.href) { + window.location.href = nextLink.href; + event.preventDefault(); + } + break; + } + } + + // some keyboard layouts may need Shift to get / + switch (event.key) { + case "/": + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break; + Documentation.focusSearchBar(); + event.preventDefault(); + } + }); + }, +}; + +// quick alias for translations +const _ = Documentation.gettext; + +_ready(Documentation.init); diff --git a/en/3.0/_static/documentation_options.js b/en/3.0/_static/documentation_options.js new file mode 100644 index 0000000000..f9961581ec --- /dev/null +++ b/en/3.0/_static/documentation_options.js @@ -0,0 +1,14 @@ +var DOCUMENTATION_OPTIONS = { + URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), + VERSION: 'Netris v3.0', + LANGUAGE: 'en', + COLLAPSE_INDEX: false, + BUILDER: 'html', + FILE_SUFFIX: '.html', + LINK_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt', + NAVIGATION_WITH_KEYS: false, + SHOW_SEARCH_SUMMARY: true, + ENABLE_SEARCH_SHORTCUTS: true, +}; \ No newline at end of file diff --git a/en/3.0/_static/file.png b/en/3.0/_static/file.png new file mode 100644 index 0000000000..a858a410e4 Binary files /dev/null and b/en/3.0/_static/file.png differ diff --git a/en/3.0/_static/jquery-3.6.0.js b/en/3.0/_static/jquery-3.6.0.js new file mode 100644 index 0000000000..fc6c299b73 --- /dev/null +++ b/en/3.0/_static/jquery-3.6.0.js @@ -0,0 +1,10881 @@ +/*! + * jQuery JavaScript Library v3.6.0 + * https://jquery.com/ + * + * Includes Sizzle.js + * https://sizzlejs.com/ + * + * Copyright OpenJS Foundation and other contributors + * Released under the MIT license + * https://jquery.org/license + * + * Date: 2021-03-02T17:08Z + */ +( function( global, factory ) { + + "use strict"; + + if ( typeof module === "object" && typeof module.exports === "object" ) { + + // For CommonJS and CommonJS-like environments where a proper `window` + // is present, execute the factory and get jQuery. + // For environments that do not have a `window` with a `document` + // (such as Node.js), expose a factory as module.exports. + // This accentuates the need for the creation of a real `window`. + // e.g. var jQuery = require("jquery")(window); + // See ticket #14549 for more info. + module.exports = global.document ? + factory( global, true ) : + function( w ) { + if ( !w.document ) { + throw new Error( "jQuery requires a window with a document" ); + } + return factory( w ); + }; + } else { + factory( global ); + } + +// Pass this if window is not defined yet +} )( typeof window !== "undefined" ? window : this, function( window, noGlobal ) { + +// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 - 9.1 +// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict mode +// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict mode should be common +// enough that all such attempts are guarded in a try block. +"use strict"; + +var arr = []; + +var getProto = Object.getPrototypeOf; + +var slice = arr.slice; + +var flat = arr.flat ? function( array ) { + return arr.flat.call( array ); +} : function( array ) { + return arr.concat.apply( [], array ); +}; + + +var push = arr.push; + +var indexOf = arr.indexOf; + +var class2type = {}; + +var toString = class2type.toString; + +var hasOwn = class2type.hasOwnProperty; + +var fnToString = hasOwn.toString; + +var ObjectFunctionString = fnToString.call( Object ); + +var support = {}; + +var isFunction = function isFunction( obj ) { + + // Support: Chrome <=57, Firefox <=52 + // In some browsers, typeof returns "function" for HTML elements + // (i.e., `typeof document.createElement( "object" ) === "function"`). + // We don't want to classify *any* DOM node as a function. + // Support: QtWeb <=3.8.5, WebKit <=534.34, wkhtmltopdf tool <=0.12.5 + // Plus for old WebKit, typeof returns "function" for HTML collections + // (e.g., `typeof document.getElementsByTagName("div") === "function"`). (gh-4756) + return typeof obj === "function" && typeof obj.nodeType !== "number" && + typeof obj.item !== "function"; + }; + + +var isWindow = function isWindow( obj ) { + return obj != null && obj === obj.window; + }; + + +var document = window.document; + + + + var preservedScriptAttributes = { + type: true, + src: true, + nonce: true, + noModule: true + }; + + function DOMEval( code, node, doc ) { + doc = doc || document; + + var i, val, + script = doc.createElement( "script" ); + + script.text = code; + if ( node ) { + for ( i in preservedScriptAttributes ) { + + // Support: Firefox 64+, Edge 18+ + // Some browsers don't support the "nonce" property on scripts. + // On the other hand, just using `getAttribute` is not enough as + // the `nonce` attribute is reset to an empty string whenever it + // becomes browsing-context connected. + // See https://github.com/whatwg/html/issues/2369 + // See https://html.spec.whatwg.org/#nonce-attributes + // The `node.getAttribute` check was added for the sake of + // `jQuery.globalEval` so that it can fake a nonce-containing node + // via an object. + val = node[ i ] || node.getAttribute && node.getAttribute( i ); + if ( val ) { + script.setAttribute( i, val ); + } + } + } + doc.head.appendChild( script ).parentNode.removeChild( script ); + } + + +function toType( obj ) { + if ( obj == null ) { + return obj + ""; + } + + // Support: Android <=2.3 only (functionish RegExp) + return typeof obj === "object" || typeof obj === "function" ? + class2type[ toString.call( obj ) ] || "object" : + typeof obj; +} +/* global Symbol */ +// Defining this global in .eslintrc.json would create a danger of using the global +// unguarded in another place, it seems safer to define global only for this module + + + +var + version = "3.6.0", + + // Define a local copy of jQuery + jQuery = function( selector, context ) { + + // The jQuery object is actually just the init constructor 'enhanced' + // Need init if jQuery is called (just allow error to be thrown if not included) + return new jQuery.fn.init( selector, context ); + }; + +jQuery.fn = jQuery.prototype = { + + // The current version of jQuery being used + jquery: version, + + constructor: jQuery, + + // The default length of a jQuery object is 0 + length: 0, + + toArray: function() { + return slice.call( this ); + }, + + // Get the Nth element in the matched element set OR + // Get the whole matched element set as a clean array + get: function( num ) { + + // Return all the elements in a clean array + if ( num == null ) { + return slice.call( this ); + } + + // Return just the one element from the set + return num < 0 ? this[ num + this.length ] : this[ num ]; + }, + + // Take an array of elements and push it onto the stack + // (returning the new matched element set) + pushStack: function( elems ) { + + // Build a new jQuery matched element set + var ret = jQuery.merge( this.constructor(), elems ); + + // Add the old object onto the stack (as a reference) + ret.prevObject = this; + + // Return the newly-formed element set + return ret; + }, + + // Execute a callback for every element in the matched set. + each: function( callback ) { + return jQuery.each( this, callback ); + }, + + map: function( callback ) { + return this.pushStack( jQuery.map( this, function( elem, i ) { + return callback.call( elem, i, elem ); + } ) ); + }, + + slice: function() { + return this.pushStack( slice.apply( this, arguments ) ); + }, + + first: function() { + return this.eq( 0 ); + }, + + last: function() { + return this.eq( -1 ); + }, + + even: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return ( i + 1 ) % 2; + } ) ); + }, + + odd: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return i % 2; + } ) ); + }, + + eq: function( i ) { + var len = this.length, + j = +i + ( i < 0 ? len : 0 ); + return this.pushStack( j >= 0 && j < len ? [ this[ j ] ] : [] ); + }, + + end: function() { + return this.prevObject || this.constructor(); + }, + + // For internal use only. + // Behaves like an Array's method, not like a jQuery method. + push: push, + sort: arr.sort, + splice: arr.splice +}; + +jQuery.extend = jQuery.fn.extend = function() { + var options, name, src, copy, copyIsArray, clone, + target = arguments[ 0 ] || {}, + i = 1, + length = arguments.length, + deep = false; + + // Handle a deep copy situation + if ( typeof target === "boolean" ) { + deep = target; + + // Skip the boolean and the target + target = arguments[ i ] || {}; + i++; + } + + // Handle case when target is a string or something (possible in deep copy) + if ( typeof target !== "object" && !isFunction( target ) ) { + target = {}; + } + + // Extend jQuery itself if only one argument is passed + if ( i === length ) { + target = this; + i--; + } + + for ( ; i < length; i++ ) { + + // Only deal with non-null/undefined values + if ( ( options = arguments[ i ] ) != null ) { + + // Extend the base object + for ( name in options ) { + copy = options[ name ]; + + // Prevent Object.prototype pollution + // Prevent never-ending loop + if ( name === "__proto__" || target === copy ) { + continue; + } + + // Recurse if we're merging plain objects or arrays + if ( deep && copy && ( jQuery.isPlainObject( copy ) || + ( copyIsArray = Array.isArray( copy ) ) ) ) { + src = target[ name ]; + + // Ensure proper type for the source value + if ( copyIsArray && !Array.isArray( src ) ) { + clone = []; + } else if ( !copyIsArray && !jQuery.isPlainObject( src ) ) { + clone = {}; + } else { + clone = src; + } + copyIsArray = false; + + // Never move original objects, clone them + target[ name ] = jQuery.extend( deep, clone, copy ); + + // Don't bring in undefined values + } else if ( copy !== undefined ) { + target[ name ] = copy; + } + } + } + } + + // Return the modified object + return target; +}; + +jQuery.extend( { + + // Unique for each copy of jQuery on the page + expando: "jQuery" + ( version + Math.random() ).replace( /\D/g, "" ), + + // Assume jQuery is ready without the ready module + isReady: true, + + error: function( msg ) { + throw new Error( msg ); + }, + + noop: function() {}, + + isPlainObject: function( obj ) { + var proto, Ctor; + + // Detect obvious negatives + // Use toString instead of jQuery.type to catch host objects + if ( !obj || toString.call( obj ) !== "[object Object]" ) { + return false; + } + + proto = getProto( obj ); + + // Objects with no prototype (e.g., `Object.create( null )`) are plain + if ( !proto ) { + return true; + } + + // Objects with prototype are plain iff they were constructed by a global Object function + Ctor = hasOwn.call( proto, "constructor" ) && proto.constructor; + return typeof Ctor === "function" && fnToString.call( Ctor ) === ObjectFunctionString; + }, + + isEmptyObject: function( obj ) { + var name; + + for ( name in obj ) { + return false; + } + return true; + }, + + // Evaluates a script in a provided context; falls back to the global one + // if not specified. + globalEval: function( code, options, doc ) { + DOMEval( code, { nonce: options && options.nonce }, doc ); + }, + + each: function( obj, callback ) { + var length, i = 0; + + if ( isArrayLike( obj ) ) { + length = obj.length; + for ( ; i < length; i++ ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } else { + for ( i in obj ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } + + return obj; + }, + + // results is for internal usage only + makeArray: function( arr, results ) { + var ret = results || []; + + if ( arr != null ) { + if ( isArrayLike( Object( arr ) ) ) { + jQuery.merge( ret, + typeof arr === "string" ? + [ arr ] : arr + ); + } else { + push.call( ret, arr ); + } + } + + return ret; + }, + + inArray: function( elem, arr, i ) { + return arr == null ? -1 : indexOf.call( arr, elem, i ); + }, + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + merge: function( first, second ) { + var len = +second.length, + j = 0, + i = first.length; + + for ( ; j < len; j++ ) { + first[ i++ ] = second[ j ]; + } + + first.length = i; + + return first; + }, + + grep: function( elems, callback, invert ) { + var callbackInverse, + matches = [], + i = 0, + length = elems.length, + callbackExpect = !invert; + + // Go through the array, only saving the items + // that pass the validator function + for ( ; i < length; i++ ) { + callbackInverse = !callback( elems[ i ], i ); + if ( callbackInverse !== callbackExpect ) { + matches.push( elems[ i ] ); + } + } + + return matches; + }, + + // arg is for internal usage only + map: function( elems, callback, arg ) { + var length, value, + i = 0, + ret = []; + + // Go through the array, translating each of the items to their new values + if ( isArrayLike( elems ) ) { + length = elems.length; + for ( ; i < length; i++ ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + + // Go through every key on the object, + } else { + for ( i in elems ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + } + + // Flatten any nested arrays + return flat( ret ); + }, + + // A global GUID counter for objects + guid: 1, + + // jQuery.support is not used in Core but other projects attach their + // properties to it so it needs to exist. + support: support +} ); + +if ( typeof Symbol === "function" ) { + jQuery.fn[ Symbol.iterator ] = arr[ Symbol.iterator ]; +} + +// Populate the class2type map +jQuery.each( "Boolean Number String Function Array Date RegExp Object Error Symbol".split( " " ), + function( _i, name ) { + class2type[ "[object " + name + "]" ] = name.toLowerCase(); + } ); + +function isArrayLike( obj ) { + + // Support: real iOS 8.2 only (not reproducible in simulator) + // `in` check used to prevent JIT error (gh-2145) + // hasOwn isn't used here due to false negatives + // regarding Nodelist length in IE + var length = !!obj && "length" in obj && obj.length, + type = toType( obj ); + + if ( isFunction( obj ) || isWindow( obj ) ) { + return false; + } + + return type === "array" || length === 0 || + typeof length === "number" && length > 0 && ( length - 1 ) in obj; +} +var Sizzle = +/*! + * Sizzle CSS Selector Engine v2.3.6 + * https://sizzlejs.com/ + * + * Copyright JS Foundation and other contributors + * Released under the MIT license + * https://js.foundation/ + * + * Date: 2021-02-16 + */ +( function( window ) { +var i, + support, + Expr, + getText, + isXML, + tokenize, + compile, + select, + outermostContext, + sortInput, + hasDuplicate, + + // Local document vars + setDocument, + document, + docElem, + documentIsHTML, + rbuggyQSA, + rbuggyMatches, + matches, + contains, + + // Instance-specific data + expando = "sizzle" + 1 * new Date(), + preferredDoc = window.document, + dirruns = 0, + done = 0, + classCache = createCache(), + tokenCache = createCache(), + compilerCache = createCache(), + nonnativeSelectorCache = createCache(), + sortOrder = function( a, b ) { + if ( a === b ) { + hasDuplicate = true; + } + return 0; + }, + + // Instance methods + hasOwn = ( {} ).hasOwnProperty, + arr = [], + pop = arr.pop, + pushNative = arr.push, + push = arr.push, + slice = arr.slice, + + // Use a stripped-down indexOf as it's faster than native + // https://jsperf.com/thor-indexof-vs-for/5 + indexOf = function( list, elem ) { + var i = 0, + len = list.length; + for ( ; i < len; i++ ) { + if ( list[ i ] === elem ) { + return i; + } + } + return -1; + }, + + booleans = "checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|" + + "ismap|loop|multiple|open|readonly|required|scoped", + + // Regular expressions + + // http://www.w3.org/TR/css3-selectors/#whitespace + whitespace = "[\\x20\\t\\r\\n\\f]", + + // https://www.w3.org/TR/css-syntax-3/#ident-token-diagram + identifier = "(?:\\\\[\\da-fA-F]{1,6}" + whitespace + + "?|\\\\[^\\r\\n\\f]|[\\w-]|[^\0-\\x7f])+", + + // Attribute selectors: http://www.w3.org/TR/selectors/#attribute-selectors + attributes = "\\[" + whitespace + "*(" + identifier + ")(?:" + whitespace + + + // Operator (capture 2) + "*([*^$|!~]?=)" + whitespace + + + // "Attribute values must be CSS identifiers [capture 5] + // or strings [capture 3 or capture 4]" + "*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|(" + identifier + "))|)" + + whitespace + "*\\]", + + pseudos = ":(" + identifier + ")(?:\\((" + + + // To reduce the number of selectors needing tokenize in the preFilter, prefer arguments: + // 1. quoted (capture 3; capture 4 or capture 5) + "('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|" + + + // 2. simple (capture 6) + "((?:\\\\.|[^\\\\()[\\]]|" + attributes + ")*)|" + + + // 3. anything else (capture 2) + ".*" + + ")\\)|)", + + // Leading and non-escaped trailing whitespace, capturing some non-whitespace characters preceding the latter + rwhitespace = new RegExp( whitespace + "+", "g" ), + rtrim = new RegExp( "^" + whitespace + "+|((?:^|[^\\\\])(?:\\\\.)*)" + + whitespace + "+$", "g" ), + + rcomma = new RegExp( "^" + whitespace + "*," + whitespace + "*" ), + rcombinators = new RegExp( "^" + whitespace + "*([>+~]|" + whitespace + ")" + whitespace + + "*" ), + rdescend = new RegExp( whitespace + "|>" ), + + rpseudo = new RegExp( pseudos ), + ridentifier = new RegExp( "^" + identifier + "$" ), + + matchExpr = { + "ID": new RegExp( "^#(" + identifier + ")" ), + "CLASS": new RegExp( "^\\.(" + identifier + ")" ), + "TAG": new RegExp( "^(" + identifier + "|[*])" ), + "ATTR": new RegExp( "^" + attributes ), + "PSEUDO": new RegExp( "^" + pseudos ), + "CHILD": new RegExp( "^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\(" + + whitespace + "*(even|odd|(([+-]|)(\\d*)n|)" + whitespace + "*(?:([+-]|)" + + whitespace + "*(\\d+)|))" + whitespace + "*\\)|)", "i" ), + "bool": new RegExp( "^(?:" + booleans + ")$", "i" ), + + // For use in libraries implementing .is() + // We use this for POS matching in `select` + "needsContext": new RegExp( "^" + whitespace + + "*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\(" + whitespace + + "*((?:-\\d)?\\d*)" + whitespace + "*\\)|)(?=[^-]|$)", "i" ) + }, + + rhtml = /HTML$/i, + rinputs = /^(?:input|select|textarea|button)$/i, + rheader = /^h\d$/i, + + rnative = /^[^{]+\{\s*\[native \w/, + + // Easily-parseable/retrievable ID or TAG or CLASS selectors + rquickExpr = /^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/, + + rsibling = /[+~]/, + + // CSS escapes + // http://www.w3.org/TR/CSS21/syndata.html#escaped-characters + runescape = new RegExp( "\\\\[\\da-fA-F]{1,6}" + whitespace + "?|\\\\([^\\r\\n\\f])", "g" ), + funescape = function( escape, nonHex ) { + var high = "0x" + escape.slice( 1 ) - 0x10000; + + return nonHex ? + + // Strip the backslash prefix from a non-hex escape sequence + nonHex : + + // Replace a hexadecimal escape sequence with the encoded Unicode code point + // Support: IE <=11+ + // For values outside the Basic Multilingual Plane (BMP), manually construct a + // surrogate pair + high < 0 ? + String.fromCharCode( high + 0x10000 ) : + String.fromCharCode( high >> 10 | 0xD800, high & 0x3FF | 0xDC00 ); + }, + + // CSS string/identifier serialization + // https://drafts.csswg.org/cssom/#common-serializing-idioms + rcssescape = /([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g, + fcssescape = function( ch, asCodePoint ) { + if ( asCodePoint ) { + + // U+0000 NULL becomes U+FFFD REPLACEMENT CHARACTER + if ( ch === "\0" ) { + return "\uFFFD"; + } + + // Control characters and (dependent upon position) numbers get escaped as code points + return ch.slice( 0, -1 ) + "\\" + + ch.charCodeAt( ch.length - 1 ).toString( 16 ) + " "; + } + + // Other potentially-special ASCII characters get backslash-escaped + return "\\" + ch; + }, + + // Used for iframes + // See setDocument() + // Removing the function wrapper causes a "Permission Denied" + // error in IE + unloadHandler = function() { + setDocument(); + }, + + inDisabledFieldset = addCombinator( + function( elem ) { + return elem.disabled === true && elem.nodeName.toLowerCase() === "fieldset"; + }, + { dir: "parentNode", next: "legend" } + ); + +// Optimize for push.apply( _, NodeList ) +try { + push.apply( + ( arr = slice.call( preferredDoc.childNodes ) ), + preferredDoc.childNodes + ); + + // Support: Android<4.0 + // Detect silently failing push.apply + // eslint-disable-next-line no-unused-expressions + arr[ preferredDoc.childNodes.length ].nodeType; +} catch ( e ) { + push = { apply: arr.length ? + + // Leverage slice if possible + function( target, els ) { + pushNative.apply( target, slice.call( els ) ); + } : + + // Support: IE<9 + // Otherwise append directly + function( target, els ) { + var j = target.length, + i = 0; + + // Can't trust NodeList.length + while ( ( target[ j++ ] = els[ i++ ] ) ) {} + target.length = j - 1; + } + }; +} + +function Sizzle( selector, context, results, seed ) { + var m, i, elem, nid, match, groups, newSelector, + newContext = context && context.ownerDocument, + + // nodeType defaults to 9, since context defaults to document + nodeType = context ? context.nodeType : 9; + + results = results || []; + + // Return early from calls with invalid selector or context + if ( typeof selector !== "string" || !selector || + nodeType !== 1 && nodeType !== 9 && nodeType !== 11 ) { + + return results; + } + + // Try to shortcut find operations (as opposed to filters) in HTML documents + if ( !seed ) { + setDocument( context ); + context = context || document; + + if ( documentIsHTML ) { + + // If the selector is sufficiently simple, try using a "get*By*" DOM method + // (excepting DocumentFragment context, where the methods don't exist) + if ( nodeType !== 11 && ( match = rquickExpr.exec( selector ) ) ) { + + // ID selector + if ( ( m = match[ 1 ] ) ) { + + // Document context + if ( nodeType === 9 ) { + if ( ( elem = context.getElementById( m ) ) ) { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( elem.id === m ) { + results.push( elem ); + return results; + } + } else { + return results; + } + + // Element context + } else { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( newContext && ( elem = newContext.getElementById( m ) ) && + contains( context, elem ) && + elem.id === m ) { + + results.push( elem ); + return results; + } + } + + // Type selector + } else if ( match[ 2 ] ) { + push.apply( results, context.getElementsByTagName( selector ) ); + return results; + + // Class selector + } else if ( ( m = match[ 3 ] ) && support.getElementsByClassName && + context.getElementsByClassName ) { + + push.apply( results, context.getElementsByClassName( m ) ); + return results; + } + } + + // Take advantage of querySelectorAll + if ( support.qsa && + !nonnativeSelectorCache[ selector + " " ] && + ( !rbuggyQSA || !rbuggyQSA.test( selector ) ) && + + // Support: IE 8 only + // Exclude object elements + ( nodeType !== 1 || context.nodeName.toLowerCase() !== "object" ) ) { + + newSelector = selector; + newContext = context; + + // qSA considers elements outside a scoping root when evaluating child or + // descendant combinators, which is not what we want. + // In such cases, we work around the behavior by prefixing every selector in the + // list with an ID selector referencing the scope context. + // The technique has to be used as well when a leading combinator is used + // as such selectors are not recognized by querySelectorAll. + // Thanks to Andrew Dupont for this technique. + if ( nodeType === 1 && + ( rdescend.test( selector ) || rcombinators.test( selector ) ) ) { + + // Expand context for sibling selectors + newContext = rsibling.test( selector ) && testContext( context.parentNode ) || + context; + + // We can use :scope instead of the ID hack if the browser + // supports it & if we're not changing the context. + if ( newContext !== context || !support.scope ) { + + // Capture the context ID, setting it first if necessary + if ( ( nid = context.getAttribute( "id" ) ) ) { + nid = nid.replace( rcssescape, fcssescape ); + } else { + context.setAttribute( "id", ( nid = expando ) ); + } + } + + // Prefix every selector in the list + groups = tokenize( selector ); + i = groups.length; + while ( i-- ) { + groups[ i ] = ( nid ? "#" + nid : ":scope" ) + " " + + toSelector( groups[ i ] ); + } + newSelector = groups.join( "," ); + } + + try { + push.apply( results, + newContext.querySelectorAll( newSelector ) + ); + return results; + } catch ( qsaError ) { + nonnativeSelectorCache( selector, true ); + } finally { + if ( nid === expando ) { + context.removeAttribute( "id" ); + } + } + } + } + } + + // All others + return select( selector.replace( rtrim, "$1" ), context, results, seed ); +} + +/** + * Create key-value caches of limited size + * @returns {function(string, object)} Returns the Object data after storing it on itself with + * property name the (space-suffixed) string and (if the cache is larger than Expr.cacheLength) + * deleting the oldest entry + */ +function createCache() { + var keys = []; + + function cache( key, value ) { + + // Use (key + " ") to avoid collision with native prototype properties (see Issue #157) + if ( keys.push( key + " " ) > Expr.cacheLength ) { + + // Only keep the most recent entries + delete cache[ keys.shift() ]; + } + return ( cache[ key + " " ] = value ); + } + return cache; +} + +/** + * Mark a function for special use by Sizzle + * @param {Function} fn The function to mark + */ +function markFunction( fn ) { + fn[ expando ] = true; + return fn; +} + +/** + * Support testing using an element + * @param {Function} fn Passed the created element and returns a boolean result + */ +function assert( fn ) { + var el = document.createElement( "fieldset" ); + + try { + return !!fn( el ); + } catch ( e ) { + return false; + } finally { + + // Remove from its parent by default + if ( el.parentNode ) { + el.parentNode.removeChild( el ); + } + + // release memory in IE + el = null; + } +} + +/** + * Adds the same handler for all of the specified attrs + * @param {String} attrs Pipe-separated list of attributes + * @param {Function} handler The method that will be applied + */ +function addHandle( attrs, handler ) { + var arr = attrs.split( "|" ), + i = arr.length; + + while ( i-- ) { + Expr.attrHandle[ arr[ i ] ] = handler; + } +} + +/** + * Checks document order of two siblings + * @param {Element} a + * @param {Element} b + * @returns {Number} Returns less than 0 if a precedes b, greater than 0 if a follows b + */ +function siblingCheck( a, b ) { + var cur = b && a, + diff = cur && a.nodeType === 1 && b.nodeType === 1 && + a.sourceIndex - b.sourceIndex; + + // Use IE sourceIndex if available on both nodes + if ( diff ) { + return diff; + } + + // Check if b follows a + if ( cur ) { + while ( ( cur = cur.nextSibling ) ) { + if ( cur === b ) { + return -1; + } + } + } + + return a ? 1 : -1; +} + +/** + * Returns a function to use in pseudos for input types + * @param {String} type + */ +function createInputPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for buttons + * @param {String} type + */ +function createButtonPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return ( name === "input" || name === "button" ) && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for :enabled/:disabled + * @param {Boolean} disabled true for :disabled; false for :enabled + */ +function createDisabledPseudo( disabled ) { + + // Known :disabled false positives: fieldset[disabled] > legend:nth-of-type(n+2) :can-disable + return function( elem ) { + + // Only certain elements can match :enabled or :disabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-enabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-disabled + if ( "form" in elem ) { + + // Check for inherited disabledness on relevant non-disabled elements: + // * listed form-associated elements in a disabled fieldset + // https://html.spec.whatwg.org/multipage/forms.html#category-listed + // https://html.spec.whatwg.org/multipage/forms.html#concept-fe-disabled + // * option elements in a disabled optgroup + // https://html.spec.whatwg.org/multipage/forms.html#concept-option-disabled + // All such elements have a "form" property. + if ( elem.parentNode && elem.disabled === false ) { + + // Option elements defer to a parent optgroup if present + if ( "label" in elem ) { + if ( "label" in elem.parentNode ) { + return elem.parentNode.disabled === disabled; + } else { + return elem.disabled === disabled; + } + } + + // Support: IE 6 - 11 + // Use the isDisabled shortcut property to check for disabled fieldset ancestors + return elem.isDisabled === disabled || + + // Where there is no isDisabled, check manually + /* jshint -W018 */ + elem.isDisabled !== !disabled && + inDisabledFieldset( elem ) === disabled; + } + + return elem.disabled === disabled; + + // Try to winnow out elements that can't be disabled before trusting the disabled property. + // Some victims get caught in our net (label, legend, menu, track), but it shouldn't + // even exist on them, let alone have a boolean value. + } else if ( "label" in elem ) { + return elem.disabled === disabled; + } + + // Remaining elements are neither :enabled nor :disabled + return false; + }; +} + +/** + * Returns a function to use in pseudos for positionals + * @param {Function} fn + */ +function createPositionalPseudo( fn ) { + return markFunction( function( argument ) { + argument = +argument; + return markFunction( function( seed, matches ) { + var j, + matchIndexes = fn( [], seed.length, argument ), + i = matchIndexes.length; + + // Match elements found at the specified indexes + while ( i-- ) { + if ( seed[ ( j = matchIndexes[ i ] ) ] ) { + seed[ j ] = !( matches[ j ] = seed[ j ] ); + } + } + } ); + } ); +} + +/** + * Checks a node for validity as a Sizzle context + * @param {Element|Object=} context + * @returns {Element|Object|Boolean} The input node if acceptable, otherwise a falsy value + */ +function testContext( context ) { + return context && typeof context.getElementsByTagName !== "undefined" && context; +} + +// Expose support vars for convenience +support = Sizzle.support = {}; + +/** + * Detects XML nodes + * @param {Element|Object} elem An element or a document + * @returns {Boolean} True iff elem is a non-HTML XML node + */ +isXML = Sizzle.isXML = function( elem ) { + var namespace = elem && elem.namespaceURI, + docElem = elem && ( elem.ownerDocument || elem ).documentElement; + + // Support: IE <=8 + // Assume HTML when documentElement doesn't yet exist, such as inside loading iframes + // https://bugs.jquery.com/ticket/4833 + return !rhtml.test( namespace || docElem && docElem.nodeName || "HTML" ); +}; + +/** + * Sets document-related variables once based on the current document + * @param {Element|Object} [doc] An element or document object to use to set the document + * @returns {Object} Returns the current document + */ +setDocument = Sizzle.setDocument = function( node ) { + var hasCompare, subWindow, + doc = node ? node.ownerDocument || node : preferredDoc; + + // Return early if doc is invalid or already selected + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( doc == document || doc.nodeType !== 9 || !doc.documentElement ) { + return document; + } + + // Update global variables + document = doc; + docElem = document.documentElement; + documentIsHTML = !isXML( document ); + + // Support: IE 9 - 11+, Edge 12 - 18+ + // Accessing iframe documents after unload throws "permission denied" errors (jQuery #13936) + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( preferredDoc != document && + ( subWindow = document.defaultView ) && subWindow.top !== subWindow ) { + + // Support: IE 11, Edge + if ( subWindow.addEventListener ) { + subWindow.addEventListener( "unload", unloadHandler, false ); + + // Support: IE 9 - 10 only + } else if ( subWindow.attachEvent ) { + subWindow.attachEvent( "onunload", unloadHandler ); + } + } + + // Support: IE 8 - 11+, Edge 12 - 18+, Chrome <=16 - 25 only, Firefox <=3.6 - 31 only, + // Safari 4 - 5 only, Opera <=11.6 - 12.x only + // IE/Edge & older browsers don't support the :scope pseudo-class. + // Support: Safari 6.0 only + // Safari 6.0 supports :scope but it's an alias of :root there. + support.scope = assert( function( el ) { + docElem.appendChild( el ).appendChild( document.createElement( "div" ) ); + return typeof el.querySelectorAll !== "undefined" && + !el.querySelectorAll( ":scope fieldset div" ).length; + } ); + + /* Attributes + ---------------------------------------------------------------------- */ + + // Support: IE<8 + // Verify that getAttribute really returns attributes and not properties + // (excepting IE8 booleans) + support.attributes = assert( function( el ) { + el.className = "i"; + return !el.getAttribute( "className" ); + } ); + + /* getElement(s)By* + ---------------------------------------------------------------------- */ + + // Check if getElementsByTagName("*") returns only elements + support.getElementsByTagName = assert( function( el ) { + el.appendChild( document.createComment( "" ) ); + return !el.getElementsByTagName( "*" ).length; + } ); + + // Support: IE<9 + support.getElementsByClassName = rnative.test( document.getElementsByClassName ); + + // Support: IE<10 + // Check if getElementById returns elements by name + // The broken getElementById methods don't pick up programmatically-set names, + // so use a roundabout getElementsByName test + support.getById = assert( function( el ) { + docElem.appendChild( el ).id = expando; + return !document.getElementsByName || !document.getElementsByName( expando ).length; + } ); + + // ID filter and find + if ( support.getById ) { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + return elem.getAttribute( "id" ) === attrId; + }; + }; + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var elem = context.getElementById( id ); + return elem ? [ elem ] : []; + } + }; + } else { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + var node = typeof elem.getAttributeNode !== "undefined" && + elem.getAttributeNode( "id" ); + return node && node.value === attrId; + }; + }; + + // Support: IE 6 - 7 only + // getElementById is not reliable as a find shortcut + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var node, i, elems, + elem = context.getElementById( id ); + + if ( elem ) { + + // Verify the id attribute + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + + // Fall back on getElementsByName + elems = context.getElementsByName( id ); + i = 0; + while ( ( elem = elems[ i++ ] ) ) { + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + } + } + + return []; + } + }; + } + + // Tag + Expr.find[ "TAG" ] = support.getElementsByTagName ? + function( tag, context ) { + if ( typeof context.getElementsByTagName !== "undefined" ) { + return context.getElementsByTagName( tag ); + + // DocumentFragment nodes don't have gEBTN + } else if ( support.qsa ) { + return context.querySelectorAll( tag ); + } + } : + + function( tag, context ) { + var elem, + tmp = [], + i = 0, + + // By happy coincidence, a (broken) gEBTN appears on DocumentFragment nodes too + results = context.getElementsByTagName( tag ); + + // Filter out possible comments + if ( tag === "*" ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem.nodeType === 1 ) { + tmp.push( elem ); + } + } + + return tmp; + } + return results; + }; + + // Class + Expr.find[ "CLASS" ] = support.getElementsByClassName && function( className, context ) { + if ( typeof context.getElementsByClassName !== "undefined" && documentIsHTML ) { + return context.getElementsByClassName( className ); + } + }; + + /* QSA/matchesSelector + ---------------------------------------------------------------------- */ + + // QSA and matchesSelector support + + // matchesSelector(:active) reports false when true (IE9/Opera 11.5) + rbuggyMatches = []; + + // qSa(:focus) reports false when true (Chrome 21) + // We allow this because of a bug in IE8/9 that throws an error + // whenever `document.activeElement` is accessed on an iframe + // So, we allow :focus to pass through QSA all the time to avoid the IE error + // See https://bugs.jquery.com/ticket/13378 + rbuggyQSA = []; + + if ( ( support.qsa = rnative.test( document.querySelectorAll ) ) ) { + + // Build QSA regex + // Regex strategy adopted from Diego Perini + assert( function( el ) { + + var input; + + // Select is set to empty string on purpose + // This is to test IE's treatment of not explicitly + // setting a boolean content attribute, + // since its presence should be enough + // https://bugs.jquery.com/ticket/12359 + docElem.appendChild( el ).innerHTML = "" + + ""; + + // Support: IE8, Opera 11-12.16 + // Nothing should be selected when empty strings follow ^= or $= or *= + // The test attribute must be unknown in Opera but "safe" for WinRT + // https://msdn.microsoft.com/en-us/library/ie/hh465388.aspx#attribute_section + if ( el.querySelectorAll( "[msallowcapture^='']" ).length ) { + rbuggyQSA.push( "[*^$]=" + whitespace + "*(?:''|\"\")" ); + } + + // Support: IE8 + // Boolean attributes and "value" are not treated correctly + if ( !el.querySelectorAll( "[selected]" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*(?:value|" + booleans + ")" ); + } + + // Support: Chrome<29, Android<4.4, Safari<7.0+, iOS<7.0+, PhantomJS<1.9.8+ + if ( !el.querySelectorAll( "[id~=" + expando + "-]" ).length ) { + rbuggyQSA.push( "~=" ); + } + + // Support: IE 11+, Edge 15 - 18+ + // IE 11/Edge don't find elements on a `[name='']` query in some cases. + // Adding a temporary attribute to the document before the selection works + // around the issue. + // Interestingly, IE 10 & older don't seem to have the issue. + input = document.createElement( "input" ); + input.setAttribute( "name", "" ); + el.appendChild( input ); + if ( !el.querySelectorAll( "[name='']" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*name" + whitespace + "*=" + + whitespace + "*(?:''|\"\")" ); + } + + // Webkit/Opera - :checked should return selected option elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + // IE8 throws error here and will not see later tests + if ( !el.querySelectorAll( ":checked" ).length ) { + rbuggyQSA.push( ":checked" ); + } + + // Support: Safari 8+, iOS 8+ + // https://bugs.webkit.org/show_bug.cgi?id=136851 + // In-page `selector#id sibling-combinator selector` fails + if ( !el.querySelectorAll( "a#" + expando + "+*" ).length ) { + rbuggyQSA.push( ".#.+[+~]" ); + } + + // Support: Firefox <=3.6 - 5 only + // Old Firefox doesn't throw on a badly-escaped identifier. + el.querySelectorAll( "\\\f" ); + rbuggyQSA.push( "[\\r\\n\\f]" ); + } ); + + assert( function( el ) { + el.innerHTML = "" + + ""; + + // Support: Windows 8 Native Apps + // The type and name attributes are restricted during .innerHTML assignment + var input = document.createElement( "input" ); + input.setAttribute( "type", "hidden" ); + el.appendChild( input ).setAttribute( "name", "D" ); + + // Support: IE8 + // Enforce case-sensitivity of name attribute + if ( el.querySelectorAll( "[name=d]" ).length ) { + rbuggyQSA.push( "name" + whitespace + "*[*^$|!~]?=" ); + } + + // FF 3.5 - :enabled/:disabled and hidden elements (hidden elements are still enabled) + // IE8 throws error here and will not see later tests + if ( el.querySelectorAll( ":enabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: IE9-11+ + // IE's :disabled selector does not pick up the children of disabled fieldsets + docElem.appendChild( el ).disabled = true; + if ( el.querySelectorAll( ":disabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: Opera 10 - 11 only + // Opera 10-11 does not throw on post-comma invalid pseudos + el.querySelectorAll( "*,:x" ); + rbuggyQSA.push( ",.*:" ); + } ); + } + + if ( ( support.matchesSelector = rnative.test( ( matches = docElem.matches || + docElem.webkitMatchesSelector || + docElem.mozMatchesSelector || + docElem.oMatchesSelector || + docElem.msMatchesSelector ) ) ) ) { + + assert( function( el ) { + + // Check to see if it's possible to do matchesSelector + // on a disconnected node (IE 9) + support.disconnectedMatch = matches.call( el, "*" ); + + // This should fail with an exception + // Gecko does not error, returns false instead + matches.call( el, "[s!='']:x" ); + rbuggyMatches.push( "!=", pseudos ); + } ); + } + + rbuggyQSA = rbuggyQSA.length && new RegExp( rbuggyQSA.join( "|" ) ); + rbuggyMatches = rbuggyMatches.length && new RegExp( rbuggyMatches.join( "|" ) ); + + /* Contains + ---------------------------------------------------------------------- */ + hasCompare = rnative.test( docElem.compareDocumentPosition ); + + // Element contains another + // Purposefully self-exclusive + // As in, an element does not contain itself + contains = hasCompare || rnative.test( docElem.contains ) ? + function( a, b ) { + var adown = a.nodeType === 9 ? a.documentElement : a, + bup = b && b.parentNode; + return a === bup || !!( bup && bup.nodeType === 1 && ( + adown.contains ? + adown.contains( bup ) : + a.compareDocumentPosition && a.compareDocumentPosition( bup ) & 16 + ) ); + } : + function( a, b ) { + if ( b ) { + while ( ( b = b.parentNode ) ) { + if ( b === a ) { + return true; + } + } + } + return false; + }; + + /* Sorting + ---------------------------------------------------------------------- */ + + // Document order sorting + sortOrder = hasCompare ? + function( a, b ) { + + // Flag for duplicate removal + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + // Sort on method existence if only one input has compareDocumentPosition + var compare = !a.compareDocumentPosition - !b.compareDocumentPosition; + if ( compare ) { + return compare; + } + + // Calculate position if both inputs belong to the same document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + compare = ( a.ownerDocument || a ) == ( b.ownerDocument || b ) ? + a.compareDocumentPosition( b ) : + + // Otherwise we know they are disconnected + 1; + + // Disconnected nodes + if ( compare & 1 || + ( !support.sortDetached && b.compareDocumentPosition( a ) === compare ) ) { + + // Choose the first element that is related to our preferred document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( a == document || a.ownerDocument == preferredDoc && + contains( preferredDoc, a ) ) { + return -1; + } + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( b == document || b.ownerDocument == preferredDoc && + contains( preferredDoc, b ) ) { + return 1; + } + + // Maintain original order + return sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + } + + return compare & 4 ? -1 : 1; + } : + function( a, b ) { + + // Exit early if the nodes are identical + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + var cur, + i = 0, + aup = a.parentNode, + bup = b.parentNode, + ap = [ a ], + bp = [ b ]; + + // Parentless nodes are either documents or disconnected + if ( !aup || !bup ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + return a == document ? -1 : + b == document ? 1 : + /* eslint-enable eqeqeq */ + aup ? -1 : + bup ? 1 : + sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + + // If the nodes are siblings, we can do a quick check + } else if ( aup === bup ) { + return siblingCheck( a, b ); + } + + // Otherwise we need full lists of their ancestors for comparison + cur = a; + while ( ( cur = cur.parentNode ) ) { + ap.unshift( cur ); + } + cur = b; + while ( ( cur = cur.parentNode ) ) { + bp.unshift( cur ); + } + + // Walk down the tree looking for a discrepancy + while ( ap[ i ] === bp[ i ] ) { + i++; + } + + return i ? + + // Do a sibling check if the nodes have a common ancestor + siblingCheck( ap[ i ], bp[ i ] ) : + + // Otherwise nodes in our document sort first + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + ap[ i ] == preferredDoc ? -1 : + bp[ i ] == preferredDoc ? 1 : + /* eslint-enable eqeqeq */ + 0; + }; + + return document; +}; + +Sizzle.matches = function( expr, elements ) { + return Sizzle( expr, null, null, elements ); +}; + +Sizzle.matchesSelector = function( elem, expr ) { + setDocument( elem ); + + if ( support.matchesSelector && documentIsHTML && + !nonnativeSelectorCache[ expr + " " ] && + ( !rbuggyMatches || !rbuggyMatches.test( expr ) ) && + ( !rbuggyQSA || !rbuggyQSA.test( expr ) ) ) { + + try { + var ret = matches.call( elem, expr ); + + // IE 9's matchesSelector returns false on disconnected nodes + if ( ret || support.disconnectedMatch || + + // As well, disconnected nodes are said to be in a document + // fragment in IE 9 + elem.document && elem.document.nodeType !== 11 ) { + return ret; + } + } catch ( e ) { + nonnativeSelectorCache( expr, true ); + } + } + + return Sizzle( expr, document, null, [ elem ] ).length > 0; +}; + +Sizzle.contains = function( context, elem ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( context.ownerDocument || context ) != document ) { + setDocument( context ); + } + return contains( context, elem ); +}; + +Sizzle.attr = function( elem, name ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( elem.ownerDocument || elem ) != document ) { + setDocument( elem ); + } + + var fn = Expr.attrHandle[ name.toLowerCase() ], + + // Don't get fooled by Object.prototype properties (jQuery #13807) + val = fn && hasOwn.call( Expr.attrHandle, name.toLowerCase() ) ? + fn( elem, name, !documentIsHTML ) : + undefined; + + return val !== undefined ? + val : + support.attributes || !documentIsHTML ? + elem.getAttribute( name ) : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; +}; + +Sizzle.escape = function( sel ) { + return ( sel + "" ).replace( rcssescape, fcssescape ); +}; + +Sizzle.error = function( msg ) { + throw new Error( "Syntax error, unrecognized expression: " + msg ); +}; + +/** + * Document sorting and removing duplicates + * @param {ArrayLike} results + */ +Sizzle.uniqueSort = function( results ) { + var elem, + duplicates = [], + j = 0, + i = 0; + + // Unless we *know* we can detect duplicates, assume their presence + hasDuplicate = !support.detectDuplicates; + sortInput = !support.sortStable && results.slice( 0 ); + results.sort( sortOrder ); + + if ( hasDuplicate ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem === results[ i ] ) { + j = duplicates.push( i ); + } + } + while ( j-- ) { + results.splice( duplicates[ j ], 1 ); + } + } + + // Clear input after sorting to release objects + // See https://github.com/jquery/sizzle/pull/225 + sortInput = null; + + return results; +}; + +/** + * Utility function for retrieving the text value of an array of DOM nodes + * @param {Array|Element} elem + */ +getText = Sizzle.getText = function( elem ) { + var node, + ret = "", + i = 0, + nodeType = elem.nodeType; + + if ( !nodeType ) { + + // If no nodeType, this is expected to be an array + while ( ( node = elem[ i++ ] ) ) { + + // Do not traverse comment nodes + ret += getText( node ); + } + } else if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) { + + // Use textContent for elements + // innerText usage removed for consistency of new lines (jQuery #11153) + if ( typeof elem.textContent === "string" ) { + return elem.textContent; + } else { + + // Traverse its children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + ret += getText( elem ); + } + } + } else if ( nodeType === 3 || nodeType === 4 ) { + return elem.nodeValue; + } + + // Do not include comment or processing instruction nodes + + return ret; +}; + +Expr = Sizzle.selectors = { + + // Can be adjusted by the user + cacheLength: 50, + + createPseudo: markFunction, + + match: matchExpr, + + attrHandle: {}, + + find: {}, + + relative: { + ">": { dir: "parentNode", first: true }, + " ": { dir: "parentNode" }, + "+": { dir: "previousSibling", first: true }, + "~": { dir: "previousSibling" } + }, + + preFilter: { + "ATTR": function( match ) { + match[ 1 ] = match[ 1 ].replace( runescape, funescape ); + + // Move the given value to match[3] whether quoted or unquoted + match[ 3 ] = ( match[ 3 ] || match[ 4 ] || + match[ 5 ] || "" ).replace( runescape, funescape ); + + if ( match[ 2 ] === "~=" ) { + match[ 3 ] = " " + match[ 3 ] + " "; + } + + return match.slice( 0, 4 ); + }, + + "CHILD": function( match ) { + + /* matches from matchExpr["CHILD"] + 1 type (only|nth|...) + 2 what (child|of-type) + 3 argument (even|odd|\d*|\d*n([+-]\d+)?|...) + 4 xn-component of xn+y argument ([+-]?\d*n|) + 5 sign of xn-component + 6 x of xn-component + 7 sign of y-component + 8 y of y-component + */ + match[ 1 ] = match[ 1 ].toLowerCase(); + + if ( match[ 1 ].slice( 0, 3 ) === "nth" ) { + + // nth-* requires argument + if ( !match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + // numeric x and y parameters for Expr.filter.CHILD + // remember that false/true cast respectively to 0/1 + match[ 4 ] = +( match[ 4 ] ? + match[ 5 ] + ( match[ 6 ] || 1 ) : + 2 * ( match[ 3 ] === "even" || match[ 3 ] === "odd" ) ); + match[ 5 ] = +( ( match[ 7 ] + match[ 8 ] ) || match[ 3 ] === "odd" ); + + // other types prohibit arguments + } else if ( match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + return match; + }, + + "PSEUDO": function( match ) { + var excess, + unquoted = !match[ 6 ] && match[ 2 ]; + + if ( matchExpr[ "CHILD" ].test( match[ 0 ] ) ) { + return null; + } + + // Accept quoted arguments as-is + if ( match[ 3 ] ) { + match[ 2 ] = match[ 4 ] || match[ 5 ] || ""; + + // Strip excess characters from unquoted arguments + } else if ( unquoted && rpseudo.test( unquoted ) && + + // Get excess from tokenize (recursively) + ( excess = tokenize( unquoted, true ) ) && + + // advance to the next closing parenthesis + ( excess = unquoted.indexOf( ")", unquoted.length - excess ) - unquoted.length ) ) { + + // excess is a negative index + match[ 0 ] = match[ 0 ].slice( 0, excess ); + match[ 2 ] = unquoted.slice( 0, excess ); + } + + // Return only captures needed by the pseudo filter method (type and argument) + return match.slice( 0, 3 ); + } + }, + + filter: { + + "TAG": function( nodeNameSelector ) { + var nodeName = nodeNameSelector.replace( runescape, funescape ).toLowerCase(); + return nodeNameSelector === "*" ? + function() { + return true; + } : + function( elem ) { + return elem.nodeName && elem.nodeName.toLowerCase() === nodeName; + }; + }, + + "CLASS": function( className ) { + var pattern = classCache[ className + " " ]; + + return pattern || + ( pattern = new RegExp( "(^|" + whitespace + + ")" + className + "(" + whitespace + "|$)" ) ) && classCache( + className, function( elem ) { + return pattern.test( + typeof elem.className === "string" && elem.className || + typeof elem.getAttribute !== "undefined" && + elem.getAttribute( "class" ) || + "" + ); + } ); + }, + + "ATTR": function( name, operator, check ) { + return function( elem ) { + var result = Sizzle.attr( elem, name ); + + if ( result == null ) { + return operator === "!="; + } + if ( !operator ) { + return true; + } + + result += ""; + + /* eslint-disable max-len */ + + return operator === "=" ? result === check : + operator === "!=" ? result !== check : + operator === "^=" ? check && result.indexOf( check ) === 0 : + operator === "*=" ? check && result.indexOf( check ) > -1 : + operator === "$=" ? check && result.slice( -check.length ) === check : + operator === "~=" ? ( " " + result.replace( rwhitespace, " " ) + " " ).indexOf( check ) > -1 : + operator === "|=" ? result === check || result.slice( 0, check.length + 1 ) === check + "-" : + false; + /* eslint-enable max-len */ + + }; + }, + + "CHILD": function( type, what, _argument, first, last ) { + var simple = type.slice( 0, 3 ) !== "nth", + forward = type.slice( -4 ) !== "last", + ofType = what === "of-type"; + + return first === 1 && last === 0 ? + + // Shortcut for :nth-*(n) + function( elem ) { + return !!elem.parentNode; + } : + + function( elem, _context, xml ) { + var cache, uniqueCache, outerCache, node, nodeIndex, start, + dir = simple !== forward ? "nextSibling" : "previousSibling", + parent = elem.parentNode, + name = ofType && elem.nodeName.toLowerCase(), + useCache = !xml && !ofType, + diff = false; + + if ( parent ) { + + // :(first|last|only)-(child|of-type) + if ( simple ) { + while ( dir ) { + node = elem; + while ( ( node = node[ dir ] ) ) { + if ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) { + + return false; + } + } + + // Reverse direction for :only-* (if we haven't yet done so) + start = dir = type === "only" && !start && "nextSibling"; + } + return true; + } + + start = [ forward ? parent.firstChild : parent.lastChild ]; + + // non-xml :nth-child(...) stores cache data on `parent` + if ( forward && useCache ) { + + // Seek `elem` from a previously-cached index + + // ...in a gzip-friendly way + node = parent; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex && cache[ 2 ]; + node = nodeIndex && parent.childNodes[ nodeIndex ]; + + while ( ( node = ++nodeIndex && node && node[ dir ] || + + // Fallback to seeking `elem` from the start + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + // When found, cache indexes on `parent` and break + if ( node.nodeType === 1 && ++diff && node === elem ) { + uniqueCache[ type ] = [ dirruns, nodeIndex, diff ]; + break; + } + } + + } else { + + // Use previously-cached element index if available + if ( useCache ) { + + // ...in a gzip-friendly way + node = elem; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex; + } + + // xml :nth-child(...) + // or :nth-last-child(...) or :nth(-last)?-of-type(...) + if ( diff === false ) { + + // Use the same loop as above to seek `elem` from the start + while ( ( node = ++nodeIndex && node && node[ dir ] || + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + if ( ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) && + ++diff ) { + + // Cache the index of each encountered element + if ( useCache ) { + outerCache = node[ expando ] || + ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + uniqueCache[ type ] = [ dirruns, diff ]; + } + + if ( node === elem ) { + break; + } + } + } + } + } + + // Incorporate the offset, then check against cycle size + diff -= last; + return diff === first || ( diff % first === 0 && diff / first >= 0 ); + } + }; + }, + + "PSEUDO": function( pseudo, argument ) { + + // pseudo-class names are case-insensitive + // http://www.w3.org/TR/selectors/#pseudo-classes + // Prioritize by case sensitivity in case custom pseudos are added with uppercase letters + // Remember that setFilters inherits from pseudos + var args, + fn = Expr.pseudos[ pseudo ] || Expr.setFilters[ pseudo.toLowerCase() ] || + Sizzle.error( "unsupported pseudo: " + pseudo ); + + // The user may use createPseudo to indicate that + // arguments are needed to create the filter function + // just as Sizzle does + if ( fn[ expando ] ) { + return fn( argument ); + } + + // But maintain support for old signatures + if ( fn.length > 1 ) { + args = [ pseudo, pseudo, "", argument ]; + return Expr.setFilters.hasOwnProperty( pseudo.toLowerCase() ) ? + markFunction( function( seed, matches ) { + var idx, + matched = fn( seed, argument ), + i = matched.length; + while ( i-- ) { + idx = indexOf( seed, matched[ i ] ); + seed[ idx ] = !( matches[ idx ] = matched[ i ] ); + } + } ) : + function( elem ) { + return fn( elem, 0, args ); + }; + } + + return fn; + } + }, + + pseudos: { + + // Potentially complex pseudos + "not": markFunction( function( selector ) { + + // Trim the selector passed to compile + // to avoid treating leading and trailing + // spaces as combinators + var input = [], + results = [], + matcher = compile( selector.replace( rtrim, "$1" ) ); + + return matcher[ expando ] ? + markFunction( function( seed, matches, _context, xml ) { + var elem, + unmatched = matcher( seed, null, xml, [] ), + i = seed.length; + + // Match elements unmatched by `matcher` + while ( i-- ) { + if ( ( elem = unmatched[ i ] ) ) { + seed[ i ] = !( matches[ i ] = elem ); + } + } + } ) : + function( elem, _context, xml ) { + input[ 0 ] = elem; + matcher( input, null, xml, results ); + + // Don't keep the element (issue #299) + input[ 0 ] = null; + return !results.pop(); + }; + } ), + + "has": markFunction( function( selector ) { + return function( elem ) { + return Sizzle( selector, elem ).length > 0; + }; + } ), + + "contains": markFunction( function( text ) { + text = text.replace( runescape, funescape ); + return function( elem ) { + return ( elem.textContent || getText( elem ) ).indexOf( text ) > -1; + }; + } ), + + // "Whether an element is represented by a :lang() selector + // is based solely on the element's language value + // being equal to the identifier C, + // or beginning with the identifier C immediately followed by "-". + // The matching of C against the element's language value is performed case-insensitively. + // The identifier C does not have to be a valid language name." + // http://www.w3.org/TR/selectors/#lang-pseudo + "lang": markFunction( function( lang ) { + + // lang value must be a valid identifier + if ( !ridentifier.test( lang || "" ) ) { + Sizzle.error( "unsupported lang: " + lang ); + } + lang = lang.replace( runescape, funescape ).toLowerCase(); + return function( elem ) { + var elemLang; + do { + if ( ( elemLang = documentIsHTML ? + elem.lang : + elem.getAttribute( "xml:lang" ) || elem.getAttribute( "lang" ) ) ) { + + elemLang = elemLang.toLowerCase(); + return elemLang === lang || elemLang.indexOf( lang + "-" ) === 0; + } + } while ( ( elem = elem.parentNode ) && elem.nodeType === 1 ); + return false; + }; + } ), + + // Miscellaneous + "target": function( elem ) { + var hash = window.location && window.location.hash; + return hash && hash.slice( 1 ) === elem.id; + }, + + "root": function( elem ) { + return elem === docElem; + }, + + "focus": function( elem ) { + return elem === document.activeElement && + ( !document.hasFocus || document.hasFocus() ) && + !!( elem.type || elem.href || ~elem.tabIndex ); + }, + + // Boolean properties + "enabled": createDisabledPseudo( false ), + "disabled": createDisabledPseudo( true ), + + "checked": function( elem ) { + + // In CSS3, :checked should return both checked and selected elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + var nodeName = elem.nodeName.toLowerCase(); + return ( nodeName === "input" && !!elem.checked ) || + ( nodeName === "option" && !!elem.selected ); + }, + + "selected": function( elem ) { + + // Accessing this property makes selected-by-default + // options in Safari work properly + if ( elem.parentNode ) { + // eslint-disable-next-line no-unused-expressions + elem.parentNode.selectedIndex; + } + + return elem.selected === true; + }, + + // Contents + "empty": function( elem ) { + + // http://www.w3.org/TR/selectors/#empty-pseudo + // :empty is negated by element (1) or content nodes (text: 3; cdata: 4; entity ref: 5), + // but not by others (comment: 8; processing instruction: 7; etc.) + // nodeType < 6 works because attributes (2) do not appear as children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + if ( elem.nodeType < 6 ) { + return false; + } + } + return true; + }, + + "parent": function( elem ) { + return !Expr.pseudos[ "empty" ]( elem ); + }, + + // Element/input types + "header": function( elem ) { + return rheader.test( elem.nodeName ); + }, + + "input": function( elem ) { + return rinputs.test( elem.nodeName ); + }, + + "button": function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === "button" || name === "button"; + }, + + "text": function( elem ) { + var attr; + return elem.nodeName.toLowerCase() === "input" && + elem.type === "text" && + + // Support: IE<8 + // New HTML5 attribute values (e.g., "search") appear with elem.type === "text" + ( ( attr = elem.getAttribute( "type" ) ) == null || + attr.toLowerCase() === "text" ); + }, + + // Position-in-collection + "first": createPositionalPseudo( function() { + return [ 0 ]; + } ), + + "last": createPositionalPseudo( function( _matchIndexes, length ) { + return [ length - 1 ]; + } ), + + "eq": createPositionalPseudo( function( _matchIndexes, length, argument ) { + return [ argument < 0 ? argument + length : argument ]; + } ), + + "even": createPositionalPseudo( function( matchIndexes, length ) { + var i = 0; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "odd": createPositionalPseudo( function( matchIndexes, length ) { + var i = 1; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "lt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? + argument + length : + argument > length ? + length : + argument; + for ( ; --i >= 0; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "gt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? argument + length : argument; + for ( ; ++i < length; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ) + } +}; + +Expr.pseudos[ "nth" ] = Expr.pseudos[ "eq" ]; + +// Add button/input type pseudos +for ( i in { radio: true, checkbox: true, file: true, password: true, image: true } ) { + Expr.pseudos[ i ] = createInputPseudo( i ); +} +for ( i in { submit: true, reset: true } ) { + Expr.pseudos[ i ] = createButtonPseudo( i ); +} + +// Easy API for creating new setFilters +function setFilters() {} +setFilters.prototype = Expr.filters = Expr.pseudos; +Expr.setFilters = new setFilters(); + +tokenize = Sizzle.tokenize = function( selector, parseOnly ) { + var matched, match, tokens, type, + soFar, groups, preFilters, + cached = tokenCache[ selector + " " ]; + + if ( cached ) { + return parseOnly ? 0 : cached.slice( 0 ); + } + + soFar = selector; + groups = []; + preFilters = Expr.preFilter; + + while ( soFar ) { + + // Comma and first run + if ( !matched || ( match = rcomma.exec( soFar ) ) ) { + if ( match ) { + + // Don't consume trailing commas as valid + soFar = soFar.slice( match[ 0 ].length ) || soFar; + } + groups.push( ( tokens = [] ) ); + } + + matched = false; + + // Combinators + if ( ( match = rcombinators.exec( soFar ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + + // Cast descendant combinators to space + type: match[ 0 ].replace( rtrim, " " ) + } ); + soFar = soFar.slice( matched.length ); + } + + // Filters + for ( type in Expr.filter ) { + if ( ( match = matchExpr[ type ].exec( soFar ) ) && ( !preFilters[ type ] || + ( match = preFilters[ type ]( match ) ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + type: type, + matches: match + } ); + soFar = soFar.slice( matched.length ); + } + } + + if ( !matched ) { + break; + } + } + + // Return the length of the invalid excess + // if we're just parsing + // Otherwise, throw an error or return tokens + return parseOnly ? + soFar.length : + soFar ? + Sizzle.error( selector ) : + + // Cache the tokens + tokenCache( selector, groups ).slice( 0 ); +}; + +function toSelector( tokens ) { + var i = 0, + len = tokens.length, + selector = ""; + for ( ; i < len; i++ ) { + selector += tokens[ i ].value; + } + return selector; +} + +function addCombinator( matcher, combinator, base ) { + var dir = combinator.dir, + skip = combinator.next, + key = skip || dir, + checkNonElements = base && key === "parentNode", + doneName = done++; + + return combinator.first ? + + // Check against closest ancestor/preceding element + function( elem, context, xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + return matcher( elem, context, xml ); + } + } + return false; + } : + + // Check against all ancestor/preceding elements + function( elem, context, xml ) { + var oldCache, uniqueCache, outerCache, + newCache = [ dirruns, doneName ]; + + // We can't set arbitrary data on XML nodes, so they don't benefit from combinator caching + if ( xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + if ( matcher( elem, context, xml ) ) { + return true; + } + } + } + } else { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + outerCache = elem[ expando ] || ( elem[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ elem.uniqueID ] || + ( outerCache[ elem.uniqueID ] = {} ); + + if ( skip && skip === elem.nodeName.toLowerCase() ) { + elem = elem[ dir ] || elem; + } else if ( ( oldCache = uniqueCache[ key ] ) && + oldCache[ 0 ] === dirruns && oldCache[ 1 ] === doneName ) { + + // Assign to newCache so results back-propagate to previous elements + return ( newCache[ 2 ] = oldCache[ 2 ] ); + } else { + + // Reuse newcache so results back-propagate to previous elements + uniqueCache[ key ] = newCache; + + // A match means we're done; a fail means we have to keep checking + if ( ( newCache[ 2 ] = matcher( elem, context, xml ) ) ) { + return true; + } + } + } + } + } + return false; + }; +} + +function elementMatcher( matchers ) { + return matchers.length > 1 ? + function( elem, context, xml ) { + var i = matchers.length; + while ( i-- ) { + if ( !matchers[ i ]( elem, context, xml ) ) { + return false; + } + } + return true; + } : + matchers[ 0 ]; +} + +function multipleContexts( selector, contexts, results ) { + var i = 0, + len = contexts.length; + for ( ; i < len; i++ ) { + Sizzle( selector, contexts[ i ], results ); + } + return results; +} + +function condense( unmatched, map, filter, context, xml ) { + var elem, + newUnmatched = [], + i = 0, + len = unmatched.length, + mapped = map != null; + + for ( ; i < len; i++ ) { + if ( ( elem = unmatched[ i ] ) ) { + if ( !filter || filter( elem, context, xml ) ) { + newUnmatched.push( elem ); + if ( mapped ) { + map.push( i ); + } + } + } + } + + return newUnmatched; +} + +function setMatcher( preFilter, selector, matcher, postFilter, postFinder, postSelector ) { + if ( postFilter && !postFilter[ expando ] ) { + postFilter = setMatcher( postFilter ); + } + if ( postFinder && !postFinder[ expando ] ) { + postFinder = setMatcher( postFinder, postSelector ); + } + return markFunction( function( seed, results, context, xml ) { + var temp, i, elem, + preMap = [], + postMap = [], + preexisting = results.length, + + // Get initial elements from seed or context + elems = seed || multipleContexts( + selector || "*", + context.nodeType ? [ context ] : context, + [] + ), + + // Prefilter to get matcher input, preserving a map for seed-results synchronization + matcherIn = preFilter && ( seed || !selector ) ? + condense( elems, preMap, preFilter, context, xml ) : + elems, + + matcherOut = matcher ? + + // If we have a postFinder, or filtered seed, or non-seed postFilter or preexisting results, + postFinder || ( seed ? preFilter : preexisting || postFilter ) ? + + // ...intermediate processing is necessary + [] : + + // ...otherwise use results directly + results : + matcherIn; + + // Find primary matches + if ( matcher ) { + matcher( matcherIn, matcherOut, context, xml ); + } + + // Apply postFilter + if ( postFilter ) { + temp = condense( matcherOut, postMap ); + postFilter( temp, [], context, xml ); + + // Un-match failing elements by moving them back to matcherIn + i = temp.length; + while ( i-- ) { + if ( ( elem = temp[ i ] ) ) { + matcherOut[ postMap[ i ] ] = !( matcherIn[ postMap[ i ] ] = elem ); + } + } + } + + if ( seed ) { + if ( postFinder || preFilter ) { + if ( postFinder ) { + + // Get the final matcherOut by condensing this intermediate into postFinder contexts + temp = []; + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) ) { + + // Restore matcherIn since elem is not yet a final match + temp.push( ( matcherIn[ i ] = elem ) ); + } + } + postFinder( null, ( matcherOut = [] ), temp, xml ); + } + + // Move matched elements from seed to results to keep them synchronized + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) && + ( temp = postFinder ? indexOf( seed, elem ) : preMap[ i ] ) > -1 ) { + + seed[ temp ] = !( results[ temp ] = elem ); + } + } + } + + // Add elements to results, through postFinder if defined + } else { + matcherOut = condense( + matcherOut === results ? + matcherOut.splice( preexisting, matcherOut.length ) : + matcherOut + ); + if ( postFinder ) { + postFinder( null, results, matcherOut, xml ); + } else { + push.apply( results, matcherOut ); + } + } + } ); +} + +function matcherFromTokens( tokens ) { + var checkContext, matcher, j, + len = tokens.length, + leadingRelative = Expr.relative[ tokens[ 0 ].type ], + implicitRelative = leadingRelative || Expr.relative[ " " ], + i = leadingRelative ? 1 : 0, + + // The foundational matcher ensures that elements are reachable from top-level context(s) + matchContext = addCombinator( function( elem ) { + return elem === checkContext; + }, implicitRelative, true ), + matchAnyContext = addCombinator( function( elem ) { + return indexOf( checkContext, elem ) > -1; + }, implicitRelative, true ), + matchers = [ function( elem, context, xml ) { + var ret = ( !leadingRelative && ( xml || context !== outermostContext ) ) || ( + ( checkContext = context ).nodeType ? + matchContext( elem, context, xml ) : + matchAnyContext( elem, context, xml ) ); + + // Avoid hanging onto element (issue #299) + checkContext = null; + return ret; + } ]; + + for ( ; i < len; i++ ) { + if ( ( matcher = Expr.relative[ tokens[ i ].type ] ) ) { + matchers = [ addCombinator( elementMatcher( matchers ), matcher ) ]; + } else { + matcher = Expr.filter[ tokens[ i ].type ].apply( null, tokens[ i ].matches ); + + // Return special upon seeing a positional matcher + if ( matcher[ expando ] ) { + + // Find the next relative operator (if any) for proper handling + j = ++i; + for ( ; j < len; j++ ) { + if ( Expr.relative[ tokens[ j ].type ] ) { + break; + } + } + return setMatcher( + i > 1 && elementMatcher( matchers ), + i > 1 && toSelector( + + // If the preceding token was a descendant combinator, insert an implicit any-element `*` + tokens + .slice( 0, i - 1 ) + .concat( { value: tokens[ i - 2 ].type === " " ? "*" : "" } ) + ).replace( rtrim, "$1" ), + matcher, + i < j && matcherFromTokens( tokens.slice( i, j ) ), + j < len && matcherFromTokens( ( tokens = tokens.slice( j ) ) ), + j < len && toSelector( tokens ) + ); + } + matchers.push( matcher ); + } + } + + return elementMatcher( matchers ); +} + +function matcherFromGroupMatchers( elementMatchers, setMatchers ) { + var bySet = setMatchers.length > 0, + byElement = elementMatchers.length > 0, + superMatcher = function( seed, context, xml, results, outermost ) { + var elem, j, matcher, + matchedCount = 0, + i = "0", + unmatched = seed && [], + setMatched = [], + contextBackup = outermostContext, + + // We must always have either seed elements or outermost context + elems = seed || byElement && Expr.find[ "TAG" ]( "*", outermost ), + + // Use integer dirruns iff this is the outermost matcher + dirrunsUnique = ( dirruns += contextBackup == null ? 1 : Math.random() || 0.1 ), + len = elems.length; + + if ( outermost ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + outermostContext = context == document || context || outermost; + } + + // Add elements passing elementMatchers directly to results + // Support: IE<9, Safari + // Tolerate NodeList properties (IE: "length"; Safari: ) matching elements by id + for ( ; i !== len && ( elem = elems[ i ] ) != null; i++ ) { + if ( byElement && elem ) { + j = 0; + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( !context && elem.ownerDocument != document ) { + setDocument( elem ); + xml = !documentIsHTML; + } + while ( ( matcher = elementMatchers[ j++ ] ) ) { + if ( matcher( elem, context || document, xml ) ) { + results.push( elem ); + break; + } + } + if ( outermost ) { + dirruns = dirrunsUnique; + } + } + + // Track unmatched elements for set filters + if ( bySet ) { + + // They will have gone through all possible matchers + if ( ( elem = !matcher && elem ) ) { + matchedCount--; + } + + // Lengthen the array for every element, matched or not + if ( seed ) { + unmatched.push( elem ); + } + } + } + + // `i` is now the count of elements visited above, and adding it to `matchedCount` + // makes the latter nonnegative. + matchedCount += i; + + // Apply set filters to unmatched elements + // NOTE: This can be skipped if there are no unmatched elements (i.e., `matchedCount` + // equals `i`), unless we didn't visit _any_ elements in the above loop because we have + // no element matchers and no seed. + // Incrementing an initially-string "0" `i` allows `i` to remain a string only in that + // case, which will result in a "00" `matchedCount` that differs from `i` but is also + // numerically zero. + if ( bySet && i !== matchedCount ) { + j = 0; + while ( ( matcher = setMatchers[ j++ ] ) ) { + matcher( unmatched, setMatched, context, xml ); + } + + if ( seed ) { + + // Reintegrate element matches to eliminate the need for sorting + if ( matchedCount > 0 ) { + while ( i-- ) { + if ( !( unmatched[ i ] || setMatched[ i ] ) ) { + setMatched[ i ] = pop.call( results ); + } + } + } + + // Discard index placeholder values to get only actual matches + setMatched = condense( setMatched ); + } + + // Add matches to results + push.apply( results, setMatched ); + + // Seedless set matches succeeding multiple successful matchers stipulate sorting + if ( outermost && !seed && setMatched.length > 0 && + ( matchedCount + setMatchers.length ) > 1 ) { + + Sizzle.uniqueSort( results ); + } + } + + // Override manipulation of globals by nested matchers + if ( outermost ) { + dirruns = dirrunsUnique; + outermostContext = contextBackup; + } + + return unmatched; + }; + + return bySet ? + markFunction( superMatcher ) : + superMatcher; +} + +compile = Sizzle.compile = function( selector, match /* Internal Use Only */ ) { + var i, + setMatchers = [], + elementMatchers = [], + cached = compilerCache[ selector + " " ]; + + if ( !cached ) { + + // Generate a function of recursive functions that can be used to check each element + if ( !match ) { + match = tokenize( selector ); + } + i = match.length; + while ( i-- ) { + cached = matcherFromTokens( match[ i ] ); + if ( cached[ expando ] ) { + setMatchers.push( cached ); + } else { + elementMatchers.push( cached ); + } + } + + // Cache the compiled function + cached = compilerCache( + selector, + matcherFromGroupMatchers( elementMatchers, setMatchers ) + ); + + // Save selector and tokenization + cached.selector = selector; + } + return cached; +}; + +/** + * A low-level selection function that works with Sizzle's compiled + * selector functions + * @param {String|Function} selector A selector or a pre-compiled + * selector function built with Sizzle.compile + * @param {Element} context + * @param {Array} [results] + * @param {Array} [seed] A set of elements to match against + */ +select = Sizzle.select = function( selector, context, results, seed ) { + var i, tokens, token, type, find, + compiled = typeof selector === "function" && selector, + match = !seed && tokenize( ( selector = compiled.selector || selector ) ); + + results = results || []; + + // Try to minimize operations if there is only one selector in the list and no seed + // (the latter of which guarantees us context) + if ( match.length === 1 ) { + + // Reduce context if the leading compound selector is an ID + tokens = match[ 0 ] = match[ 0 ].slice( 0 ); + if ( tokens.length > 2 && ( token = tokens[ 0 ] ).type === "ID" && + context.nodeType === 9 && documentIsHTML && Expr.relative[ tokens[ 1 ].type ] ) { + + context = ( Expr.find[ "ID" ]( token.matches[ 0 ] + .replace( runescape, funescape ), context ) || [] )[ 0 ]; + if ( !context ) { + return results; + + // Precompiled matchers will still verify ancestry, so step up a level + } else if ( compiled ) { + context = context.parentNode; + } + + selector = selector.slice( tokens.shift().value.length ); + } + + // Fetch a seed set for right-to-left matching + i = matchExpr[ "needsContext" ].test( selector ) ? 0 : tokens.length; + while ( i-- ) { + token = tokens[ i ]; + + // Abort if we hit a combinator + if ( Expr.relative[ ( type = token.type ) ] ) { + break; + } + if ( ( find = Expr.find[ type ] ) ) { + + // Search, expanding context for leading sibling combinators + if ( ( seed = find( + token.matches[ 0 ].replace( runescape, funescape ), + rsibling.test( tokens[ 0 ].type ) && testContext( context.parentNode ) || + context + ) ) ) { + + // If seed is empty or no tokens remain, we can return early + tokens.splice( i, 1 ); + selector = seed.length && toSelector( tokens ); + if ( !selector ) { + push.apply( results, seed ); + return results; + } + + break; + } + } + } + } + + // Compile and execute a filtering function if one is not provided + // Provide `match` to avoid retokenization if we modified the selector above + ( compiled || compile( selector, match ) )( + seed, + context, + !documentIsHTML, + results, + !context || rsibling.test( selector ) && testContext( context.parentNode ) || context + ); + return results; +}; + +// One-time assignments + +// Sort stability +support.sortStable = expando.split( "" ).sort( sortOrder ).join( "" ) === expando; + +// Support: Chrome 14-35+ +// Always assume duplicates if they aren't passed to the comparison function +support.detectDuplicates = !!hasDuplicate; + +// Initialize against the default document +setDocument(); + +// Support: Webkit<537.32 - Safari 6.0.3/Chrome 25 (fixed in Chrome 27) +// Detached nodes confoundingly follow *each other* +support.sortDetached = assert( function( el ) { + + // Should return 1, but returns 4 (following) + return el.compareDocumentPosition( document.createElement( "fieldset" ) ) & 1; +} ); + +// Support: IE<8 +// Prevent attribute/property "interpolation" +// https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx +if ( !assert( function( el ) { + el.innerHTML = ""; + return el.firstChild.getAttribute( "href" ) === "#"; +} ) ) { + addHandle( "type|href|height|width", function( elem, name, isXML ) { + if ( !isXML ) { + return elem.getAttribute( name, name.toLowerCase() === "type" ? 1 : 2 ); + } + } ); +} + +// Support: IE<9 +// Use defaultValue in place of getAttribute("value") +if ( !support.attributes || !assert( function( el ) { + el.innerHTML = ""; + el.firstChild.setAttribute( "value", "" ); + return el.firstChild.getAttribute( "value" ) === ""; +} ) ) { + addHandle( "value", function( elem, _name, isXML ) { + if ( !isXML && elem.nodeName.toLowerCase() === "input" ) { + return elem.defaultValue; + } + } ); +} + +// Support: IE<9 +// Use getAttributeNode to fetch booleans when getAttribute lies +if ( !assert( function( el ) { + return el.getAttribute( "disabled" ) == null; +} ) ) { + addHandle( booleans, function( elem, name, isXML ) { + var val; + if ( !isXML ) { + return elem[ name ] === true ? name.toLowerCase() : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; + } + } ); +} + +return Sizzle; + +} )( window ); + + + +jQuery.find = Sizzle; +jQuery.expr = Sizzle.selectors; + +// Deprecated +jQuery.expr[ ":" ] = jQuery.expr.pseudos; +jQuery.uniqueSort = jQuery.unique = Sizzle.uniqueSort; +jQuery.text = Sizzle.getText; +jQuery.isXMLDoc = Sizzle.isXML; +jQuery.contains = Sizzle.contains; +jQuery.escapeSelector = Sizzle.escape; + + + + +var dir = function( elem, dir, until ) { + var matched = [], + truncate = until !== undefined; + + while ( ( elem = elem[ dir ] ) && elem.nodeType !== 9 ) { + if ( elem.nodeType === 1 ) { + if ( truncate && jQuery( elem ).is( until ) ) { + break; + } + matched.push( elem ); + } + } + return matched; +}; + + +var siblings = function( n, elem ) { + var matched = []; + + for ( ; n; n = n.nextSibling ) { + if ( n.nodeType === 1 && n !== elem ) { + matched.push( n ); + } + } + + return matched; +}; + + +var rneedsContext = jQuery.expr.match.needsContext; + + + +function nodeName( elem, name ) { + + return elem.nodeName && elem.nodeName.toLowerCase() === name.toLowerCase(); + +} +var rsingleTag = ( /^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i ); + + + +// Implement the identical functionality for filter and not +function winnow( elements, qualifier, not ) { + if ( isFunction( qualifier ) ) { + return jQuery.grep( elements, function( elem, i ) { + return !!qualifier.call( elem, i, elem ) !== not; + } ); + } + + // Single element + if ( qualifier.nodeType ) { + return jQuery.grep( elements, function( elem ) { + return ( elem === qualifier ) !== not; + } ); + } + + // Arraylike of elements (jQuery, arguments, Array) + if ( typeof qualifier !== "string" ) { + return jQuery.grep( elements, function( elem ) { + return ( indexOf.call( qualifier, elem ) > -1 ) !== not; + } ); + } + + // Filtered directly for both simple and complex selectors + return jQuery.filter( qualifier, elements, not ); +} + +jQuery.filter = function( expr, elems, not ) { + var elem = elems[ 0 ]; + + if ( not ) { + expr = ":not(" + expr + ")"; + } + + if ( elems.length === 1 && elem.nodeType === 1 ) { + return jQuery.find.matchesSelector( elem, expr ) ? [ elem ] : []; + } + + return jQuery.find.matches( expr, jQuery.grep( elems, function( elem ) { + return elem.nodeType === 1; + } ) ); +}; + +jQuery.fn.extend( { + find: function( selector ) { + var i, ret, + len = this.length, + self = this; + + if ( typeof selector !== "string" ) { + return this.pushStack( jQuery( selector ).filter( function() { + for ( i = 0; i < len; i++ ) { + if ( jQuery.contains( self[ i ], this ) ) { + return true; + } + } + } ) ); + } + + ret = this.pushStack( [] ); + + for ( i = 0; i < len; i++ ) { + jQuery.find( selector, self[ i ], ret ); + } + + return len > 1 ? jQuery.uniqueSort( ret ) : ret; + }, + filter: function( selector ) { + return this.pushStack( winnow( this, selector || [], false ) ); + }, + not: function( selector ) { + return this.pushStack( winnow( this, selector || [], true ) ); + }, + is: function( selector ) { + return !!winnow( + this, + + // If this is a positional/relative selector, check membership in the returned set + // so $("p:first").is("p:last") won't return true for a doc with two "p". + typeof selector === "string" && rneedsContext.test( selector ) ? + jQuery( selector ) : + selector || [], + false + ).length; + } +} ); + + +// Initialize a jQuery object + + +// A central reference to the root jQuery(document) +var rootjQuery, + + // A simple way to check for HTML strings + // Prioritize #id over to avoid XSS via location.hash (#9521) + // Strict HTML recognition (#11290: must start with <) + // Shortcut simple #id case for speed + rquickExpr = /^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/, + + init = jQuery.fn.init = function( selector, context, root ) { + var match, elem; + + // HANDLE: $(""), $(null), $(undefined), $(false) + if ( !selector ) { + return this; + } + + // Method init() accepts an alternate rootjQuery + // so migrate can support jQuery.sub (gh-2101) + root = root || rootjQuery; + + // Handle HTML strings + if ( typeof selector === "string" ) { + if ( selector[ 0 ] === "<" && + selector[ selector.length - 1 ] === ">" && + selector.length >= 3 ) { + + // Assume that strings that start and end with <> are HTML and skip the regex check + match = [ null, selector, null ]; + + } else { + match = rquickExpr.exec( selector ); + } + + // Match html or make sure no context is specified for #id + if ( match && ( match[ 1 ] || !context ) ) { + + // HANDLE: $(html) -> $(array) + if ( match[ 1 ] ) { + context = context instanceof jQuery ? context[ 0 ] : context; + + // Option to run scripts is true for back-compat + // Intentionally let the error be thrown if parseHTML is not present + jQuery.merge( this, jQuery.parseHTML( + match[ 1 ], + context && context.nodeType ? context.ownerDocument || context : document, + true + ) ); + + // HANDLE: $(html, props) + if ( rsingleTag.test( match[ 1 ] ) && jQuery.isPlainObject( context ) ) { + for ( match in context ) { + + // Properties of context are called as methods if possible + if ( isFunction( this[ match ] ) ) { + this[ match ]( context[ match ] ); + + // ...and otherwise set as attributes + } else { + this.attr( match, context[ match ] ); + } + } + } + + return this; + + // HANDLE: $(#id) + } else { + elem = document.getElementById( match[ 2 ] ); + + if ( elem ) { + + // Inject the element directly into the jQuery object + this[ 0 ] = elem; + this.length = 1; + } + return this; + } + + // HANDLE: $(expr, $(...)) + } else if ( !context || context.jquery ) { + return ( context || root ).find( selector ); + + // HANDLE: $(expr, context) + // (which is just equivalent to: $(context).find(expr) + } else { + return this.constructor( context ).find( selector ); + } + + // HANDLE: $(DOMElement) + } else if ( selector.nodeType ) { + this[ 0 ] = selector; + this.length = 1; + return this; + + // HANDLE: $(function) + // Shortcut for document ready + } else if ( isFunction( selector ) ) { + return root.ready !== undefined ? + root.ready( selector ) : + + // Execute immediately if ready is not present + selector( jQuery ); + } + + return jQuery.makeArray( selector, this ); + }; + +// Give the init function the jQuery prototype for later instantiation +init.prototype = jQuery.fn; + +// Initialize central reference +rootjQuery = jQuery( document ); + + +var rparentsprev = /^(?:parents|prev(?:Until|All))/, + + // Methods guaranteed to produce a unique set when starting from a unique set + guaranteedUnique = { + children: true, + contents: true, + next: true, + prev: true + }; + +jQuery.fn.extend( { + has: function( target ) { + var targets = jQuery( target, this ), + l = targets.length; + + return this.filter( function() { + var i = 0; + for ( ; i < l; i++ ) { + if ( jQuery.contains( this, targets[ i ] ) ) { + return true; + } + } + } ); + }, + + closest: function( selectors, context ) { + var cur, + i = 0, + l = this.length, + matched = [], + targets = typeof selectors !== "string" && jQuery( selectors ); + + // Positional selectors never match, since there's no _selection_ context + if ( !rneedsContext.test( selectors ) ) { + for ( ; i < l; i++ ) { + for ( cur = this[ i ]; cur && cur !== context; cur = cur.parentNode ) { + + // Always skip document fragments + if ( cur.nodeType < 11 && ( targets ? + targets.index( cur ) > -1 : + + // Don't pass non-elements to Sizzle + cur.nodeType === 1 && + jQuery.find.matchesSelector( cur, selectors ) ) ) { + + matched.push( cur ); + break; + } + } + } + } + + return this.pushStack( matched.length > 1 ? jQuery.uniqueSort( matched ) : matched ); + }, + + // Determine the position of an element within the set + index: function( elem ) { + + // No argument, return index in parent + if ( !elem ) { + return ( this[ 0 ] && this[ 0 ].parentNode ) ? this.first().prevAll().length : -1; + } + + // Index in selector + if ( typeof elem === "string" ) { + return indexOf.call( jQuery( elem ), this[ 0 ] ); + } + + // Locate the position of the desired element + return indexOf.call( this, + + // If it receives a jQuery object, the first element is used + elem.jquery ? elem[ 0 ] : elem + ); + }, + + add: function( selector, context ) { + return this.pushStack( + jQuery.uniqueSort( + jQuery.merge( this.get(), jQuery( selector, context ) ) + ) + ); + }, + + addBack: function( selector ) { + return this.add( selector == null ? + this.prevObject : this.prevObject.filter( selector ) + ); + } +} ); + +function sibling( cur, dir ) { + while ( ( cur = cur[ dir ] ) && cur.nodeType !== 1 ) {} + return cur; +} + +jQuery.each( { + parent: function( elem ) { + var parent = elem.parentNode; + return parent && parent.nodeType !== 11 ? parent : null; + }, + parents: function( elem ) { + return dir( elem, "parentNode" ); + }, + parentsUntil: function( elem, _i, until ) { + return dir( elem, "parentNode", until ); + }, + next: function( elem ) { + return sibling( elem, "nextSibling" ); + }, + prev: function( elem ) { + return sibling( elem, "previousSibling" ); + }, + nextAll: function( elem ) { + return dir( elem, "nextSibling" ); + }, + prevAll: function( elem ) { + return dir( elem, "previousSibling" ); + }, + nextUntil: function( elem, _i, until ) { + return dir( elem, "nextSibling", until ); + }, + prevUntil: function( elem, _i, until ) { + return dir( elem, "previousSibling", until ); + }, + siblings: function( elem ) { + return siblings( ( elem.parentNode || {} ).firstChild, elem ); + }, + children: function( elem ) { + return siblings( elem.firstChild ); + }, + contents: function( elem ) { + if ( elem.contentDocument != null && + + // Support: IE 11+ + // elements with no `data` attribute has an object + // `contentDocument` with a `null` prototype. + getProto( elem.contentDocument ) ) { + + return elem.contentDocument; + } + + // Support: IE 9 - 11 only, iOS 7 only, Android Browser <=4.3 only + // Treat the template element as a regular one in browsers that + // don't support it. + if ( nodeName( elem, "template" ) ) { + elem = elem.content || elem; + } + + return jQuery.merge( [], elem.childNodes ); + } +}, function( name, fn ) { + jQuery.fn[ name ] = function( until, selector ) { + var matched = jQuery.map( this, fn, until ); + + if ( name.slice( -5 ) !== "Until" ) { + selector = until; + } + + if ( selector && typeof selector === "string" ) { + matched = jQuery.filter( selector, matched ); + } + + if ( this.length > 1 ) { + + // Remove duplicates + if ( !guaranteedUnique[ name ] ) { + jQuery.uniqueSort( matched ); + } + + // Reverse order for parents* and prev-derivatives + if ( rparentsprev.test( name ) ) { + matched.reverse(); + } + } + + return this.pushStack( matched ); + }; +} ); +var rnothtmlwhite = ( /[^\x20\t\r\n\f]+/g ); + + + +// Convert String-formatted options into Object-formatted ones +function createOptions( options ) { + var object = {}; + jQuery.each( options.match( rnothtmlwhite ) || [], function( _, flag ) { + object[ flag ] = true; + } ); + return object; +} + +/* + * Create a callback list using the following parameters: + * + * options: an optional list of space-separated options that will change how + * the callback list behaves or a more traditional option object + * + * By default a callback list will act like an event callback list and can be + * "fired" multiple times. + * + * Possible options: + * + * once: will ensure the callback list can only be fired once (like a Deferred) + * + * memory: will keep track of previous values and will call any callback added + * after the list has been fired right away with the latest "memorized" + * values (like a Deferred) + * + * unique: will ensure a callback can only be added once (no duplicate in the list) + * + * stopOnFalse: interrupt callings when a callback returns false + * + */ +jQuery.Callbacks = function( options ) { + + // Convert options from String-formatted to Object-formatted if needed + // (we check in cache first) + options = typeof options === "string" ? + createOptions( options ) : + jQuery.extend( {}, options ); + + var // Flag to know if list is currently firing + firing, + + // Last fire value for non-forgettable lists + memory, + + // Flag to know if list was already fired + fired, + + // Flag to prevent firing + locked, + + // Actual callback list + list = [], + + // Queue of execution data for repeatable lists + queue = [], + + // Index of currently firing callback (modified by add/remove as needed) + firingIndex = -1, + + // Fire callbacks + fire = function() { + + // Enforce single-firing + locked = locked || options.once; + + // Execute callbacks for all pending executions, + // respecting firingIndex overrides and runtime changes + fired = firing = true; + for ( ; queue.length; firingIndex = -1 ) { + memory = queue.shift(); + while ( ++firingIndex < list.length ) { + + // Run callback and check for early termination + if ( list[ firingIndex ].apply( memory[ 0 ], memory[ 1 ] ) === false && + options.stopOnFalse ) { + + // Jump to end and forget the data so .add doesn't re-fire + firingIndex = list.length; + memory = false; + } + } + } + + // Forget the data if we're done with it + if ( !options.memory ) { + memory = false; + } + + firing = false; + + // Clean up if we're done firing for good + if ( locked ) { + + // Keep an empty list if we have data for future add calls + if ( memory ) { + list = []; + + // Otherwise, this object is spent + } else { + list = ""; + } + } + }, + + // Actual Callbacks object + self = { + + // Add a callback or a collection of callbacks to the list + add: function() { + if ( list ) { + + // If we have memory from a past run, we should fire after adding + if ( memory && !firing ) { + firingIndex = list.length - 1; + queue.push( memory ); + } + + ( function add( args ) { + jQuery.each( args, function( _, arg ) { + if ( isFunction( arg ) ) { + if ( !options.unique || !self.has( arg ) ) { + list.push( arg ); + } + } else if ( arg && arg.length && toType( arg ) !== "string" ) { + + // Inspect recursively + add( arg ); + } + } ); + } )( arguments ); + + if ( memory && !firing ) { + fire(); + } + } + return this; + }, + + // Remove a callback from the list + remove: function() { + jQuery.each( arguments, function( _, arg ) { + var index; + while ( ( index = jQuery.inArray( arg, list, index ) ) > -1 ) { + list.splice( index, 1 ); + + // Handle firing indexes + if ( index <= firingIndex ) { + firingIndex--; + } + } + } ); + return this; + }, + + // Check if a given callback is in the list. + // If no argument is given, return whether or not list has callbacks attached. + has: function( fn ) { + return fn ? + jQuery.inArray( fn, list ) > -1 : + list.length > 0; + }, + + // Remove all callbacks from the list + empty: function() { + if ( list ) { + list = []; + } + return this; + }, + + // Disable .fire and .add + // Abort any current/pending executions + // Clear all callbacks and values + disable: function() { + locked = queue = []; + list = memory = ""; + return this; + }, + disabled: function() { + return !list; + }, + + // Disable .fire + // Also disable .add unless we have memory (since it would have no effect) + // Abort any pending executions + lock: function() { + locked = queue = []; + if ( !memory && !firing ) { + list = memory = ""; + } + return this; + }, + locked: function() { + return !!locked; + }, + + // Call all callbacks with the given context and arguments + fireWith: function( context, args ) { + if ( !locked ) { + args = args || []; + args = [ context, args.slice ? args.slice() : args ]; + queue.push( args ); + if ( !firing ) { + fire(); + } + } + return this; + }, + + // Call all the callbacks with the given arguments + fire: function() { + self.fireWith( this, arguments ); + return this; + }, + + // To know if the callbacks have already been called at least once + fired: function() { + return !!fired; + } + }; + + return self; +}; + + +function Identity( v ) { + return v; +} +function Thrower( ex ) { + throw ex; +} + +function adoptValue( value, resolve, reject, noValue ) { + var method; + + try { + + // Check for promise aspect first to privilege synchronous behavior + if ( value && isFunction( ( method = value.promise ) ) ) { + method.call( value ).done( resolve ).fail( reject ); + + // Other thenables + } else if ( value && isFunction( ( method = value.then ) ) ) { + method.call( value, resolve, reject ); + + // Other non-thenables + } else { + + // Control `resolve` arguments by letting Array#slice cast boolean `noValue` to integer: + // * false: [ value ].slice( 0 ) => resolve( value ) + // * true: [ value ].slice( 1 ) => resolve() + resolve.apply( undefined, [ value ].slice( noValue ) ); + } + + // For Promises/A+, convert exceptions into rejections + // Since jQuery.when doesn't unwrap thenables, we can skip the extra checks appearing in + // Deferred#then to conditionally suppress rejection. + } catch ( value ) { + + // Support: Android 4.0 only + // Strict mode functions invoked without .call/.apply get global-object context + reject.apply( undefined, [ value ] ); + } +} + +jQuery.extend( { + + Deferred: function( func ) { + var tuples = [ + + // action, add listener, callbacks, + // ... .then handlers, argument index, [final state] + [ "notify", "progress", jQuery.Callbacks( "memory" ), + jQuery.Callbacks( "memory" ), 2 ], + [ "resolve", "done", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 0, "resolved" ], + [ "reject", "fail", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 1, "rejected" ] + ], + state = "pending", + promise = { + state: function() { + return state; + }, + always: function() { + deferred.done( arguments ).fail( arguments ); + return this; + }, + "catch": function( fn ) { + return promise.then( null, fn ); + }, + + // Keep pipe for back-compat + pipe: function( /* fnDone, fnFail, fnProgress */ ) { + var fns = arguments; + + return jQuery.Deferred( function( newDefer ) { + jQuery.each( tuples, function( _i, tuple ) { + + // Map tuples (progress, done, fail) to arguments (done, fail, progress) + var fn = isFunction( fns[ tuple[ 4 ] ] ) && fns[ tuple[ 4 ] ]; + + // deferred.progress(function() { bind to newDefer or newDefer.notify }) + // deferred.done(function() { bind to newDefer or newDefer.resolve }) + // deferred.fail(function() { bind to newDefer or newDefer.reject }) + deferred[ tuple[ 1 ] ]( function() { + var returned = fn && fn.apply( this, arguments ); + if ( returned && isFunction( returned.promise ) ) { + returned.promise() + .progress( newDefer.notify ) + .done( newDefer.resolve ) + .fail( newDefer.reject ); + } else { + newDefer[ tuple[ 0 ] + "With" ]( + this, + fn ? [ returned ] : arguments + ); + } + } ); + } ); + fns = null; + } ).promise(); + }, + then: function( onFulfilled, onRejected, onProgress ) { + var maxDepth = 0; + function resolve( depth, deferred, handler, special ) { + return function() { + var that = this, + args = arguments, + mightThrow = function() { + var returned, then; + + // Support: Promises/A+ section 2.3.3.3.3 + // https://promisesaplus.com/#point-59 + // Ignore double-resolution attempts + if ( depth < maxDepth ) { + return; + } + + returned = handler.apply( that, args ); + + // Support: Promises/A+ section 2.3.1 + // https://promisesaplus.com/#point-48 + if ( returned === deferred.promise() ) { + throw new TypeError( "Thenable self-resolution" ); + } + + // Support: Promises/A+ sections 2.3.3.1, 3.5 + // https://promisesaplus.com/#point-54 + // https://promisesaplus.com/#point-75 + // Retrieve `then` only once + then = returned && + + // Support: Promises/A+ section 2.3.4 + // https://promisesaplus.com/#point-64 + // Only check objects and functions for thenability + ( typeof returned === "object" || + typeof returned === "function" ) && + returned.then; + + // Handle a returned thenable + if ( isFunction( then ) ) { + + // Special processors (notify) just wait for resolution + if ( special ) { + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ) + ); + + // Normal processors (resolve) also hook into progress + } else { + + // ...and disregard older resolution values + maxDepth++; + + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ), + resolve( maxDepth, deferred, Identity, + deferred.notifyWith ) + ); + } + + // Handle all other returned values + } else { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Identity ) { + that = undefined; + args = [ returned ]; + } + + // Process the value(s) + // Default process is resolve + ( special || deferred.resolveWith )( that, args ); + } + }, + + // Only normal processors (resolve) catch and reject exceptions + process = special ? + mightThrow : + function() { + try { + mightThrow(); + } catch ( e ) { + + if ( jQuery.Deferred.exceptionHook ) { + jQuery.Deferred.exceptionHook( e, + process.stackTrace ); + } + + // Support: Promises/A+ section 2.3.3.3.4.1 + // https://promisesaplus.com/#point-61 + // Ignore post-resolution exceptions + if ( depth + 1 >= maxDepth ) { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Thrower ) { + that = undefined; + args = [ e ]; + } + + deferred.rejectWith( that, args ); + } + } + }; + + // Support: Promises/A+ section 2.3.3.3.1 + // https://promisesaplus.com/#point-57 + // Re-resolve promises immediately to dodge false rejection from + // subsequent errors + if ( depth ) { + process(); + } else { + + // Call an optional hook to record the stack, in case of exception + // since it's otherwise lost when execution goes async + if ( jQuery.Deferred.getStackHook ) { + process.stackTrace = jQuery.Deferred.getStackHook(); + } + window.setTimeout( process ); + } + }; + } + + return jQuery.Deferred( function( newDefer ) { + + // progress_handlers.add( ... ) + tuples[ 0 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onProgress ) ? + onProgress : + Identity, + newDefer.notifyWith + ) + ); + + // fulfilled_handlers.add( ... ) + tuples[ 1 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onFulfilled ) ? + onFulfilled : + Identity + ) + ); + + // rejected_handlers.add( ... ) + tuples[ 2 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onRejected ) ? + onRejected : + Thrower + ) + ); + } ).promise(); + }, + + // Get a promise for this deferred + // If obj is provided, the promise aspect is added to the object + promise: function( obj ) { + return obj != null ? jQuery.extend( obj, promise ) : promise; + } + }, + deferred = {}; + + // Add list-specific methods + jQuery.each( tuples, function( i, tuple ) { + var list = tuple[ 2 ], + stateString = tuple[ 5 ]; + + // promise.progress = list.add + // promise.done = list.add + // promise.fail = list.add + promise[ tuple[ 1 ] ] = list.add; + + // Handle state + if ( stateString ) { + list.add( + function() { + + // state = "resolved" (i.e., fulfilled) + // state = "rejected" + state = stateString; + }, + + // rejected_callbacks.disable + // fulfilled_callbacks.disable + tuples[ 3 - i ][ 2 ].disable, + + // rejected_handlers.disable + // fulfilled_handlers.disable + tuples[ 3 - i ][ 3 ].disable, + + // progress_callbacks.lock + tuples[ 0 ][ 2 ].lock, + + // progress_handlers.lock + tuples[ 0 ][ 3 ].lock + ); + } + + // progress_handlers.fire + // fulfilled_handlers.fire + // rejected_handlers.fire + list.add( tuple[ 3 ].fire ); + + // deferred.notify = function() { deferred.notifyWith(...) } + // deferred.resolve = function() { deferred.resolveWith(...) } + // deferred.reject = function() { deferred.rejectWith(...) } + deferred[ tuple[ 0 ] ] = function() { + deferred[ tuple[ 0 ] + "With" ]( this === deferred ? undefined : this, arguments ); + return this; + }; + + // deferred.notifyWith = list.fireWith + // deferred.resolveWith = list.fireWith + // deferred.rejectWith = list.fireWith + deferred[ tuple[ 0 ] + "With" ] = list.fireWith; + } ); + + // Make the deferred a promise + promise.promise( deferred ); + + // Call given func if any + if ( func ) { + func.call( deferred, deferred ); + } + + // All done! + return deferred; + }, + + // Deferred helper + when: function( singleValue ) { + var + + // count of uncompleted subordinates + remaining = arguments.length, + + // count of unprocessed arguments + i = remaining, + + // subordinate fulfillment data + resolveContexts = Array( i ), + resolveValues = slice.call( arguments ), + + // the primary Deferred + primary = jQuery.Deferred(), + + // subordinate callback factory + updateFunc = function( i ) { + return function( value ) { + resolveContexts[ i ] = this; + resolveValues[ i ] = arguments.length > 1 ? slice.call( arguments ) : value; + if ( !( --remaining ) ) { + primary.resolveWith( resolveContexts, resolveValues ); + } + }; + }; + + // Single- and empty arguments are adopted like Promise.resolve + if ( remaining <= 1 ) { + adoptValue( singleValue, primary.done( updateFunc( i ) ).resolve, primary.reject, + !remaining ); + + // Use .then() to unwrap secondary thenables (cf. gh-3000) + if ( primary.state() === "pending" || + isFunction( resolveValues[ i ] && resolveValues[ i ].then ) ) { + + return primary.then(); + } + } + + // Multiple arguments are aggregated like Promise.all array elements + while ( i-- ) { + adoptValue( resolveValues[ i ], updateFunc( i ), primary.reject ); + } + + return primary.promise(); + } +} ); + + +// These usually indicate a programmer mistake during development, +// warn about them ASAP rather than swallowing them by default. +var rerrorNames = /^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/; + +jQuery.Deferred.exceptionHook = function( error, stack ) { + + // Support: IE 8 - 9 only + // Console exists when dev tools are open, which can happen at any time + if ( window.console && window.console.warn && error && rerrorNames.test( error.name ) ) { + window.console.warn( "jQuery.Deferred exception: " + error.message, error.stack, stack ); + } +}; + + + + +jQuery.readyException = function( error ) { + window.setTimeout( function() { + throw error; + } ); +}; + + + + +// The deferred used on DOM ready +var readyList = jQuery.Deferred(); + +jQuery.fn.ready = function( fn ) { + + readyList + .then( fn ) + + // Wrap jQuery.readyException in a function so that the lookup + // happens at the time of error handling instead of callback + // registration. + .catch( function( error ) { + jQuery.readyException( error ); + } ); + + return this; +}; + +jQuery.extend( { + + // Is the DOM ready to be used? Set to true once it occurs. + isReady: false, + + // A counter to track how many items to wait for before + // the ready event fires. See #6781 + readyWait: 1, + + // Handle when the DOM is ready + ready: function( wait ) { + + // Abort if there are pending holds or we're already ready + if ( wait === true ? --jQuery.readyWait : jQuery.isReady ) { + return; + } + + // Remember that the DOM is ready + jQuery.isReady = true; + + // If a normal DOM Ready event fired, decrement, and wait if need be + if ( wait !== true && --jQuery.readyWait > 0 ) { + return; + } + + // If there are functions bound, to execute + readyList.resolveWith( document, [ jQuery ] ); + } +} ); + +jQuery.ready.then = readyList.then; + +// The ready event handler and self cleanup method +function completed() { + document.removeEventListener( "DOMContentLoaded", completed ); + window.removeEventListener( "load", completed ); + jQuery.ready(); +} + +// Catch cases where $(document).ready() is called +// after the browser event has already occurred. +// Support: IE <=9 - 10 only +// Older IE sometimes signals "interactive" too soon +if ( document.readyState === "complete" || + ( document.readyState !== "loading" && !document.documentElement.doScroll ) ) { + + // Handle it asynchronously to allow scripts the opportunity to delay ready + window.setTimeout( jQuery.ready ); + +} else { + + // Use the handy event callback + document.addEventListener( "DOMContentLoaded", completed ); + + // A fallback to window.onload, that will always work + window.addEventListener( "load", completed ); +} + + + + +// Multifunctional method to get and set values of a collection +// The value/s can optionally be executed if it's a function +var access = function( elems, fn, key, value, chainable, emptyGet, raw ) { + var i = 0, + len = elems.length, + bulk = key == null; + + // Sets many values + if ( toType( key ) === "object" ) { + chainable = true; + for ( i in key ) { + access( elems, fn, i, key[ i ], true, emptyGet, raw ); + } + + // Sets one value + } else if ( value !== undefined ) { + chainable = true; + + if ( !isFunction( value ) ) { + raw = true; + } + + if ( bulk ) { + + // Bulk operations run against the entire set + if ( raw ) { + fn.call( elems, value ); + fn = null; + + // ...except when executing function values + } else { + bulk = fn; + fn = function( elem, _key, value ) { + return bulk.call( jQuery( elem ), value ); + }; + } + } + + if ( fn ) { + for ( ; i < len; i++ ) { + fn( + elems[ i ], key, raw ? + value : + value.call( elems[ i ], i, fn( elems[ i ], key ) ) + ); + } + } + } + + if ( chainable ) { + return elems; + } + + // Gets + if ( bulk ) { + return fn.call( elems ); + } + + return len ? fn( elems[ 0 ], key ) : emptyGet; +}; + + +// Matches dashed string for camelizing +var rmsPrefix = /^-ms-/, + rdashAlpha = /-([a-z])/g; + +// Used by camelCase as callback to replace() +function fcamelCase( _all, letter ) { + return letter.toUpperCase(); +} + +// Convert dashed to camelCase; used by the css and data modules +// Support: IE <=9 - 11, Edge 12 - 15 +// Microsoft forgot to hump their vendor prefix (#9572) +function camelCase( string ) { + return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase ); +} +var acceptData = function( owner ) { + + // Accepts only: + // - Node + // - Node.ELEMENT_NODE + // - Node.DOCUMENT_NODE + // - Object + // - Any + return owner.nodeType === 1 || owner.nodeType === 9 || !( +owner.nodeType ); +}; + + + + +function Data() { + this.expando = jQuery.expando + Data.uid++; +} + +Data.uid = 1; + +Data.prototype = { + + cache: function( owner ) { + + // Check if the owner object already has a cache + var value = owner[ this.expando ]; + + // If not, create one + if ( !value ) { + value = {}; + + // We can accept data for non-element nodes in modern browsers, + // but we should not, see #8335. + // Always return an empty object. + if ( acceptData( owner ) ) { + + // If it is a node unlikely to be stringify-ed or looped over + // use plain assignment + if ( owner.nodeType ) { + owner[ this.expando ] = value; + + // Otherwise secure it in a non-enumerable property + // configurable must be true to allow the property to be + // deleted when data is removed + } else { + Object.defineProperty( owner, this.expando, { + value: value, + configurable: true + } ); + } + } + } + + return value; + }, + set: function( owner, data, value ) { + var prop, + cache = this.cache( owner ); + + // Handle: [ owner, key, value ] args + // Always use camelCase key (gh-2257) + if ( typeof data === "string" ) { + cache[ camelCase( data ) ] = value; + + // Handle: [ owner, { properties } ] args + } else { + + // Copy the properties one-by-one to the cache object + for ( prop in data ) { + cache[ camelCase( prop ) ] = data[ prop ]; + } + } + return cache; + }, + get: function( owner, key ) { + return key === undefined ? + this.cache( owner ) : + + // Always use camelCase key (gh-2257) + owner[ this.expando ] && owner[ this.expando ][ camelCase( key ) ]; + }, + access: function( owner, key, value ) { + + // In cases where either: + // + // 1. No key was specified + // 2. A string key was specified, but no value provided + // + // Take the "read" path and allow the get method to determine + // which value to return, respectively either: + // + // 1. The entire cache object + // 2. The data stored at the key + // + if ( key === undefined || + ( ( key && typeof key === "string" ) && value === undefined ) ) { + + return this.get( owner, key ); + } + + // When the key is not a string, or both a key and value + // are specified, set or extend (existing objects) with either: + // + // 1. An object of properties + // 2. A key and value + // + this.set( owner, key, value ); + + // Since the "set" path can have two possible entry points + // return the expected data based on which path was taken[*] + return value !== undefined ? value : key; + }, + remove: function( owner, key ) { + var i, + cache = owner[ this.expando ]; + + if ( cache === undefined ) { + return; + } + + if ( key !== undefined ) { + + // Support array or space separated string of keys + if ( Array.isArray( key ) ) { + + // If key is an array of keys... + // We always set camelCase keys, so remove that. + key = key.map( camelCase ); + } else { + key = camelCase( key ); + + // If a key with the spaces exists, use it. + // Otherwise, create an array by matching non-whitespace + key = key in cache ? + [ key ] : + ( key.match( rnothtmlwhite ) || [] ); + } + + i = key.length; + + while ( i-- ) { + delete cache[ key[ i ] ]; + } + } + + // Remove the expando if there's no more data + if ( key === undefined || jQuery.isEmptyObject( cache ) ) { + + // Support: Chrome <=35 - 45 + // Webkit & Blink performance suffers when deleting properties + // from DOM nodes, so set to undefined instead + // https://bugs.chromium.org/p/chromium/issues/detail?id=378607 (bug restricted) + if ( owner.nodeType ) { + owner[ this.expando ] = undefined; + } else { + delete owner[ this.expando ]; + } + } + }, + hasData: function( owner ) { + var cache = owner[ this.expando ]; + return cache !== undefined && !jQuery.isEmptyObject( cache ); + } +}; +var dataPriv = new Data(); + +var dataUser = new Data(); + + + +// Implementation Summary +// +// 1. Enforce API surface and semantic compatibility with 1.9.x branch +// 2. Improve the module's maintainability by reducing the storage +// paths to a single mechanism. +// 3. Use the same single mechanism to support "private" and "user" data. +// 4. _Never_ expose "private" data to user code (TODO: Drop _data, _removeData) +// 5. Avoid exposing implementation details on user objects (eg. expando properties) +// 6. Provide a clear path for implementation upgrade to WeakMap in 2014 + +var rbrace = /^(?:\{[\w\W]*\}|\[[\w\W]*\])$/, + rmultiDash = /[A-Z]/g; + +function getData( data ) { + if ( data === "true" ) { + return true; + } + + if ( data === "false" ) { + return false; + } + + if ( data === "null" ) { + return null; + } + + // Only convert to a number if it doesn't change the string + if ( data === +data + "" ) { + return +data; + } + + if ( rbrace.test( data ) ) { + return JSON.parse( data ); + } + + return data; +} + +function dataAttr( elem, key, data ) { + var name; + + // If nothing was found internally, try to fetch any + // data from the HTML5 data-* attribute + if ( data === undefined && elem.nodeType === 1 ) { + name = "data-" + key.replace( rmultiDash, "-$&" ).toLowerCase(); + data = elem.getAttribute( name ); + + if ( typeof data === "string" ) { + try { + data = getData( data ); + } catch ( e ) {} + + // Make sure we set the data so it isn't changed later + dataUser.set( elem, key, data ); + } else { + data = undefined; + } + } + return data; +} + +jQuery.extend( { + hasData: function( elem ) { + return dataUser.hasData( elem ) || dataPriv.hasData( elem ); + }, + + data: function( elem, name, data ) { + return dataUser.access( elem, name, data ); + }, + + removeData: function( elem, name ) { + dataUser.remove( elem, name ); + }, + + // TODO: Now that all calls to _data and _removeData have been replaced + // with direct calls to dataPriv methods, these can be deprecated. + _data: function( elem, name, data ) { + return dataPriv.access( elem, name, data ); + }, + + _removeData: function( elem, name ) { + dataPriv.remove( elem, name ); + } +} ); + +jQuery.fn.extend( { + data: function( key, value ) { + var i, name, data, + elem = this[ 0 ], + attrs = elem && elem.attributes; + + // Gets all values + if ( key === undefined ) { + if ( this.length ) { + data = dataUser.get( elem ); + + if ( elem.nodeType === 1 && !dataPriv.get( elem, "hasDataAttrs" ) ) { + i = attrs.length; + while ( i-- ) { + + // Support: IE 11 only + // The attrs elements can be null (#14894) + if ( attrs[ i ] ) { + name = attrs[ i ].name; + if ( name.indexOf( "data-" ) === 0 ) { + name = camelCase( name.slice( 5 ) ); + dataAttr( elem, name, data[ name ] ); + } + } + } + dataPriv.set( elem, "hasDataAttrs", true ); + } + } + + return data; + } + + // Sets multiple values + if ( typeof key === "object" ) { + return this.each( function() { + dataUser.set( this, key ); + } ); + } + + return access( this, function( value ) { + var data; + + // The calling jQuery object (element matches) is not empty + // (and therefore has an element appears at this[ 0 ]) and the + // `value` parameter was not undefined. An empty jQuery object + // will result in `undefined` for elem = this[ 0 ] which will + // throw an exception if an attempt to read a data cache is made. + if ( elem && value === undefined ) { + + // Attempt to get data from the cache + // The key will always be camelCased in Data + data = dataUser.get( elem, key ); + if ( data !== undefined ) { + return data; + } + + // Attempt to "discover" the data in + // HTML5 custom data-* attrs + data = dataAttr( elem, key ); + if ( data !== undefined ) { + return data; + } + + // We tried really hard, but the data doesn't exist. + return; + } + + // Set the data... + this.each( function() { + + // We always store the camelCased key + dataUser.set( this, key, value ); + } ); + }, null, value, arguments.length > 1, null, true ); + }, + + removeData: function( key ) { + return this.each( function() { + dataUser.remove( this, key ); + } ); + } +} ); + + +jQuery.extend( { + queue: function( elem, type, data ) { + var queue; + + if ( elem ) { + type = ( type || "fx" ) + "queue"; + queue = dataPriv.get( elem, type ); + + // Speed up dequeue by getting out quickly if this is just a lookup + if ( data ) { + if ( !queue || Array.isArray( data ) ) { + queue = dataPriv.access( elem, type, jQuery.makeArray( data ) ); + } else { + queue.push( data ); + } + } + return queue || []; + } + }, + + dequeue: function( elem, type ) { + type = type || "fx"; + + var queue = jQuery.queue( elem, type ), + startLength = queue.length, + fn = queue.shift(), + hooks = jQuery._queueHooks( elem, type ), + next = function() { + jQuery.dequeue( elem, type ); + }; + + // If the fx queue is dequeued, always remove the progress sentinel + if ( fn === "inprogress" ) { + fn = queue.shift(); + startLength--; + } + + if ( fn ) { + + // Add a progress sentinel to prevent the fx queue from being + // automatically dequeued + if ( type === "fx" ) { + queue.unshift( "inprogress" ); + } + + // Clear up the last queue stop function + delete hooks.stop; + fn.call( elem, next, hooks ); + } + + if ( !startLength && hooks ) { + hooks.empty.fire(); + } + }, + + // Not public - generate a queueHooks object, or return the current one + _queueHooks: function( elem, type ) { + var key = type + "queueHooks"; + return dataPriv.get( elem, key ) || dataPriv.access( elem, key, { + empty: jQuery.Callbacks( "once memory" ).add( function() { + dataPriv.remove( elem, [ type + "queue", key ] ); + } ) + } ); + } +} ); + +jQuery.fn.extend( { + queue: function( type, data ) { + var setter = 2; + + if ( typeof type !== "string" ) { + data = type; + type = "fx"; + setter--; + } + + if ( arguments.length < setter ) { + return jQuery.queue( this[ 0 ], type ); + } + + return data === undefined ? + this : + this.each( function() { + var queue = jQuery.queue( this, type, data ); + + // Ensure a hooks for this queue + jQuery._queueHooks( this, type ); + + if ( type === "fx" && queue[ 0 ] !== "inprogress" ) { + jQuery.dequeue( this, type ); + } + } ); + }, + dequeue: function( type ) { + return this.each( function() { + jQuery.dequeue( this, type ); + } ); + }, + clearQueue: function( type ) { + return this.queue( type || "fx", [] ); + }, + + // Get a promise resolved when queues of a certain type + // are emptied (fx is the type by default) + promise: function( type, obj ) { + var tmp, + count = 1, + defer = jQuery.Deferred(), + elements = this, + i = this.length, + resolve = function() { + if ( !( --count ) ) { + defer.resolveWith( elements, [ elements ] ); + } + }; + + if ( typeof type !== "string" ) { + obj = type; + type = undefined; + } + type = type || "fx"; + + while ( i-- ) { + tmp = dataPriv.get( elements[ i ], type + "queueHooks" ); + if ( tmp && tmp.empty ) { + count++; + tmp.empty.add( resolve ); + } + } + resolve(); + return defer.promise( obj ); + } +} ); +var pnum = ( /[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/ ).source; + +var rcssNum = new RegExp( "^(?:([+-])=|)(" + pnum + ")([a-z%]*)$", "i" ); + + +var cssExpand = [ "Top", "Right", "Bottom", "Left" ]; + +var documentElement = document.documentElement; + + + + var isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ); + }, + composed = { composed: true }; + + // Support: IE 9 - 11+, Edge 12 - 18+, iOS 10.0 - 10.2 only + // Check attachment across shadow DOM boundaries when possible (gh-3504) + // Support: iOS 10.0-10.2 only + // Early iOS 10 versions support `attachShadow` but not `getRootNode`, + // leading to errors. We need to check for `getRootNode`. + if ( documentElement.getRootNode ) { + isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ) || + elem.getRootNode( composed ) === elem.ownerDocument; + }; + } +var isHiddenWithinTree = function( elem, el ) { + + // isHiddenWithinTree might be called from jQuery#filter function; + // in that case, element will be second argument + elem = el || elem; + + // Inline style trumps all + return elem.style.display === "none" || + elem.style.display === "" && + + // Otherwise, check computed style + // Support: Firefox <=43 - 45 + // Disconnected elements can have computed display: none, so first confirm that elem is + // in the document. + isAttached( elem ) && + + jQuery.css( elem, "display" ) === "none"; + }; + + + +function adjustCSS( elem, prop, valueParts, tween ) { + var adjusted, scale, + maxIterations = 20, + currentValue = tween ? + function() { + return tween.cur(); + } : + function() { + return jQuery.css( elem, prop, "" ); + }, + initial = currentValue(), + unit = valueParts && valueParts[ 3 ] || ( jQuery.cssNumber[ prop ] ? "" : "px" ), + + // Starting value computation is required for potential unit mismatches + initialInUnit = elem.nodeType && + ( jQuery.cssNumber[ prop ] || unit !== "px" && +initial ) && + rcssNum.exec( jQuery.css( elem, prop ) ); + + if ( initialInUnit && initialInUnit[ 3 ] !== unit ) { + + // Support: Firefox <=54 + // Halve the iteration target value to prevent interference from CSS upper bounds (gh-2144) + initial = initial / 2; + + // Trust units reported by jQuery.css + unit = unit || initialInUnit[ 3 ]; + + // Iteratively approximate from a nonzero starting point + initialInUnit = +initial || 1; + + while ( maxIterations-- ) { + + // Evaluate and update our best guess (doubling guesses that zero out). + // Finish if the scale equals or crosses 1 (making the old*new product non-positive). + jQuery.style( elem, prop, initialInUnit + unit ); + if ( ( 1 - scale ) * ( 1 - ( scale = currentValue() / initial || 0.5 ) ) <= 0 ) { + maxIterations = 0; + } + initialInUnit = initialInUnit / scale; + + } + + initialInUnit = initialInUnit * 2; + jQuery.style( elem, prop, initialInUnit + unit ); + + // Make sure we update the tween properties later on + valueParts = valueParts || []; + } + + if ( valueParts ) { + initialInUnit = +initialInUnit || +initial || 0; + + // Apply relative offset (+=/-=) if specified + adjusted = valueParts[ 1 ] ? + initialInUnit + ( valueParts[ 1 ] + 1 ) * valueParts[ 2 ] : + +valueParts[ 2 ]; + if ( tween ) { + tween.unit = unit; + tween.start = initialInUnit; + tween.end = adjusted; + } + } + return adjusted; +} + + +var defaultDisplayMap = {}; + +function getDefaultDisplay( elem ) { + var temp, + doc = elem.ownerDocument, + nodeName = elem.nodeName, + display = defaultDisplayMap[ nodeName ]; + + if ( display ) { + return display; + } + + temp = doc.body.appendChild( doc.createElement( nodeName ) ); + display = jQuery.css( temp, "display" ); + + temp.parentNode.removeChild( temp ); + + if ( display === "none" ) { + display = "block"; + } + defaultDisplayMap[ nodeName ] = display; + + return display; +} + +function showHide( elements, show ) { + var display, elem, + values = [], + index = 0, + length = elements.length; + + // Determine new display value for elements that need to change + for ( ; index < length; index++ ) { + elem = elements[ index ]; + if ( !elem.style ) { + continue; + } + + display = elem.style.display; + if ( show ) { + + // Since we force visibility upon cascade-hidden elements, an immediate (and slow) + // check is required in this first loop unless we have a nonempty display value (either + // inline or about-to-be-restored) + if ( display === "none" ) { + values[ index ] = dataPriv.get( elem, "display" ) || null; + if ( !values[ index ] ) { + elem.style.display = ""; + } + } + if ( elem.style.display === "" && isHiddenWithinTree( elem ) ) { + values[ index ] = getDefaultDisplay( elem ); + } + } else { + if ( display !== "none" ) { + values[ index ] = "none"; + + // Remember what we're overwriting + dataPriv.set( elem, "display", display ); + } + } + } + + // Set the display of the elements in a second loop to avoid constant reflow + for ( index = 0; index < length; index++ ) { + if ( values[ index ] != null ) { + elements[ index ].style.display = values[ index ]; + } + } + + return elements; +} + +jQuery.fn.extend( { + show: function() { + return showHide( this, true ); + }, + hide: function() { + return showHide( this ); + }, + toggle: function( state ) { + if ( typeof state === "boolean" ) { + return state ? this.show() : this.hide(); + } + + return this.each( function() { + if ( isHiddenWithinTree( this ) ) { + jQuery( this ).show(); + } else { + jQuery( this ).hide(); + } + } ); + } +} ); +var rcheckableType = ( /^(?:checkbox|radio)$/i ); + +var rtagName = ( /<([a-z][^\/\0>\x20\t\r\n\f]*)/i ); + +var rscriptType = ( /^$|^module$|\/(?:java|ecma)script/i ); + + + +( function() { + var fragment = document.createDocumentFragment(), + div = fragment.appendChild( document.createElement( "div" ) ), + input = document.createElement( "input" ); + + // Support: Android 4.0 - 4.3 only + // Check state lost if the name is set (#11217) + // Support: Windows Web Apps (WWA) + // `name` and `type` must use .setAttribute for WWA (#14901) + input.setAttribute( "type", "radio" ); + input.setAttribute( "checked", "checked" ); + input.setAttribute( "name", "t" ); + + div.appendChild( input ); + + // Support: Android <=4.1 only + // Older WebKit doesn't clone checked state correctly in fragments + support.checkClone = div.cloneNode( true ).cloneNode( true ).lastChild.checked; + + // Support: IE <=11 only + // Make sure textarea (and checkbox) defaultValue is properly cloned + div.innerHTML = ""; + support.noCloneChecked = !!div.cloneNode( true ).lastChild.defaultValue; + + // Support: IE <=9 only + // IE <=9 replaces "; + support.option = !!div.lastChild; +} )(); + + +// We have to close these tags to support XHTML (#13200) +var wrapMap = { + + // XHTML parsers do not magically insert elements in the + // same way that tag soup parsers do. So we cannot shorten + // this by omitting or other required elements. + thead: [ 1, "", "
" ], + col: [ 2, "", "
" ], + tr: [ 2, "", "
" ], + td: [ 3, "", "
" ], + + _default: [ 0, "", "" ] +}; + +wrapMap.tbody = wrapMap.tfoot = wrapMap.colgroup = wrapMap.caption = wrapMap.thead; +wrapMap.th = wrapMap.td; + +// Support: IE <=9 only +if ( !support.option ) { + wrapMap.optgroup = wrapMap.option = [ 1, "" ]; +} + + +function getAll( context, tag ) { + + // Support: IE <=9 - 11 only + // Use typeof to avoid zero-argument method invocation on host objects (#15151) + var ret; + + if ( typeof context.getElementsByTagName !== "undefined" ) { + ret = context.getElementsByTagName( tag || "*" ); + + } else if ( typeof context.querySelectorAll !== "undefined" ) { + ret = context.querySelectorAll( tag || "*" ); + + } else { + ret = []; + } + + if ( tag === undefined || tag && nodeName( context, tag ) ) { + return jQuery.merge( [ context ], ret ); + } + + return ret; +} + + +// Mark scripts as having already been evaluated +function setGlobalEval( elems, refElements ) { + var i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + dataPriv.set( + elems[ i ], + "globalEval", + !refElements || dataPriv.get( refElements[ i ], "globalEval" ) + ); + } +} + + +var rhtml = /<|&#?\w+;/; + +function buildFragment( elems, context, scripts, selection, ignored ) { + var elem, tmp, tag, wrap, attached, j, + fragment = context.createDocumentFragment(), + nodes = [], + i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + elem = elems[ i ]; + + if ( elem || elem === 0 ) { + + // Add nodes directly + if ( toType( elem ) === "object" ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, elem.nodeType ? [ elem ] : elem ); + + // Convert non-html into a text node + } else if ( !rhtml.test( elem ) ) { + nodes.push( context.createTextNode( elem ) ); + + // Convert html into DOM nodes + } else { + tmp = tmp || fragment.appendChild( context.createElement( "div" ) ); + + // Deserialize a standard representation + tag = ( rtagName.exec( elem ) || [ "", "" ] )[ 1 ].toLowerCase(); + wrap = wrapMap[ tag ] || wrapMap._default; + tmp.innerHTML = wrap[ 1 ] + jQuery.htmlPrefilter( elem ) + wrap[ 2 ]; + + // Descend through wrappers to the right content + j = wrap[ 0 ]; + while ( j-- ) { + tmp = tmp.lastChild; + } + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, tmp.childNodes ); + + // Remember the top-level container + tmp = fragment.firstChild; + + // Ensure the created nodes are orphaned (#12392) + tmp.textContent = ""; + } + } + } + + // Remove wrapper from fragment + fragment.textContent = ""; + + i = 0; + while ( ( elem = nodes[ i++ ] ) ) { + + // Skip elements already in the context collection (trac-4087) + if ( selection && jQuery.inArray( elem, selection ) > -1 ) { + if ( ignored ) { + ignored.push( elem ); + } + continue; + } + + attached = isAttached( elem ); + + // Append to fragment + tmp = getAll( fragment.appendChild( elem ), "script" ); + + // Preserve script evaluation history + if ( attached ) { + setGlobalEval( tmp ); + } + + // Capture executables + if ( scripts ) { + j = 0; + while ( ( elem = tmp[ j++ ] ) ) { + if ( rscriptType.test( elem.type || "" ) ) { + scripts.push( elem ); + } + } + } + } + + return fragment; +} + + +var rtypenamespace = /^([^.]*)(?:\.(.+)|)/; + +function returnTrue() { + return true; +} + +function returnFalse() { + return false; +} + +// Support: IE <=9 - 11+ +// focus() and blur() are asynchronous, except when they are no-op. +// So expect focus to be synchronous when the element is already active, +// and blur to be synchronous when the element is not already active. +// (focus and blur are always synchronous in other supported browsers, +// this just defines when we can count on it). +function expectSync( elem, type ) { + return ( elem === safeActiveElement() ) === ( type === "focus" ); +} + +// Support: IE <=9 only +// Accessing document.activeElement can throw unexpectedly +// https://bugs.jquery.com/ticket/13393 +function safeActiveElement() { + try { + return document.activeElement; + } catch ( err ) { } +} + +function on( elem, types, selector, data, fn, one ) { + var origFn, type; + + // Types can be a map of types/handlers + if ( typeof types === "object" ) { + + // ( types-Object, selector, data ) + if ( typeof selector !== "string" ) { + + // ( types-Object, data ) + data = data || selector; + selector = undefined; + } + for ( type in types ) { + on( elem, type, selector, data, types[ type ], one ); + } + return elem; + } + + if ( data == null && fn == null ) { + + // ( types, fn ) + fn = selector; + data = selector = undefined; + } else if ( fn == null ) { + if ( typeof selector === "string" ) { + + // ( types, selector, fn ) + fn = data; + data = undefined; + } else { + + // ( types, data, fn ) + fn = data; + data = selector; + selector = undefined; + } + } + if ( fn === false ) { + fn = returnFalse; + } else if ( !fn ) { + return elem; + } + + if ( one === 1 ) { + origFn = fn; + fn = function( event ) { + + // Can use an empty set, since event contains the info + jQuery().off( event ); + return origFn.apply( this, arguments ); + }; + + // Use same guid so caller can remove using origFn + fn.guid = origFn.guid || ( origFn.guid = jQuery.guid++ ); + } + return elem.each( function() { + jQuery.event.add( this, types, fn, data, selector ); + } ); +} + +/* + * Helper functions for managing events -- not part of the public interface. + * Props to Dean Edwards' addEvent library for many of the ideas. + */ +jQuery.event = { + + global: {}, + + add: function( elem, types, handler, data, selector ) { + + var handleObjIn, eventHandle, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.get( elem ); + + // Only attach events to objects that accept data + if ( !acceptData( elem ) ) { + return; + } + + // Caller can pass in an object of custom data in lieu of the handler + if ( handler.handler ) { + handleObjIn = handler; + handler = handleObjIn.handler; + selector = handleObjIn.selector; + } + + // Ensure that invalid selectors throw exceptions at attach time + // Evaluate against documentElement in case elem is a non-element node (e.g., document) + if ( selector ) { + jQuery.find.matchesSelector( documentElement, selector ); + } + + // Make sure that the handler has a unique ID, used to find/remove it later + if ( !handler.guid ) { + handler.guid = jQuery.guid++; + } + + // Init the element's event structure and main handler, if this is the first + if ( !( events = elemData.events ) ) { + events = elemData.events = Object.create( null ); + } + if ( !( eventHandle = elemData.handle ) ) { + eventHandle = elemData.handle = function( e ) { + + // Discard the second event of a jQuery.event.trigger() and + // when an event is called after a page has unloaded + return typeof jQuery !== "undefined" && jQuery.event.triggered !== e.type ? + jQuery.event.dispatch.apply( elem, arguments ) : undefined; + }; + } + + // Handle multiple events separated by a space + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // There *must* be a type, no attaching namespace-only handlers + if ( !type ) { + continue; + } + + // If event changes its type, use the special event handlers for the changed type + special = jQuery.event.special[ type ] || {}; + + // If selector defined, determine special event api type, otherwise given type + type = ( selector ? special.delegateType : special.bindType ) || type; + + // Update special based on newly reset type + special = jQuery.event.special[ type ] || {}; + + // handleObj is passed to all event handlers + handleObj = jQuery.extend( { + type: type, + origType: origType, + data: data, + handler: handler, + guid: handler.guid, + selector: selector, + needsContext: selector && jQuery.expr.match.needsContext.test( selector ), + namespace: namespaces.join( "." ) + }, handleObjIn ); + + // Init the event handler queue if we're the first + if ( !( handlers = events[ type ] ) ) { + handlers = events[ type ] = []; + handlers.delegateCount = 0; + + // Only use addEventListener if the special events handler returns false + if ( !special.setup || + special.setup.call( elem, data, namespaces, eventHandle ) === false ) { + + if ( elem.addEventListener ) { + elem.addEventListener( type, eventHandle ); + } + } + } + + if ( special.add ) { + special.add.call( elem, handleObj ); + + if ( !handleObj.handler.guid ) { + handleObj.handler.guid = handler.guid; + } + } + + // Add to the element's handler list, delegates in front + if ( selector ) { + handlers.splice( handlers.delegateCount++, 0, handleObj ); + } else { + handlers.push( handleObj ); + } + + // Keep track of which events have ever been used, for event optimization + jQuery.event.global[ type ] = true; + } + + }, + + // Detach an event or set of events from an element + remove: function( elem, types, handler, selector, mappedTypes ) { + + var j, origCount, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.hasData( elem ) && dataPriv.get( elem ); + + if ( !elemData || !( events = elemData.events ) ) { + return; + } + + // Once for each type.namespace in types; type may be omitted + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // Unbind all events (on this namespace, if provided) for the element + if ( !type ) { + for ( type in events ) { + jQuery.event.remove( elem, type + types[ t ], handler, selector, true ); + } + continue; + } + + special = jQuery.event.special[ type ] || {}; + type = ( selector ? special.delegateType : special.bindType ) || type; + handlers = events[ type ] || []; + tmp = tmp[ 2 ] && + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ); + + // Remove matching events + origCount = j = handlers.length; + while ( j-- ) { + handleObj = handlers[ j ]; + + if ( ( mappedTypes || origType === handleObj.origType ) && + ( !handler || handler.guid === handleObj.guid ) && + ( !tmp || tmp.test( handleObj.namespace ) ) && + ( !selector || selector === handleObj.selector || + selector === "**" && handleObj.selector ) ) { + handlers.splice( j, 1 ); + + if ( handleObj.selector ) { + handlers.delegateCount--; + } + if ( special.remove ) { + special.remove.call( elem, handleObj ); + } + } + } + + // Remove generic event handler if we removed something and no more handlers exist + // (avoids potential for endless recursion during removal of special event handlers) + if ( origCount && !handlers.length ) { + if ( !special.teardown || + special.teardown.call( elem, namespaces, elemData.handle ) === false ) { + + jQuery.removeEvent( elem, type, elemData.handle ); + } + + delete events[ type ]; + } + } + + // Remove data and the expando if it's no longer used + if ( jQuery.isEmptyObject( events ) ) { + dataPriv.remove( elem, "handle events" ); + } + }, + + dispatch: function( nativeEvent ) { + + var i, j, ret, matched, handleObj, handlerQueue, + args = new Array( arguments.length ), + + // Make a writable jQuery.Event from the native event object + event = jQuery.event.fix( nativeEvent ), + + handlers = ( + dataPriv.get( this, "events" ) || Object.create( null ) + )[ event.type ] || [], + special = jQuery.event.special[ event.type ] || {}; + + // Use the fix-ed jQuery.Event rather than the (read-only) native event + args[ 0 ] = event; + + for ( i = 1; i < arguments.length; i++ ) { + args[ i ] = arguments[ i ]; + } + + event.delegateTarget = this; + + // Call the preDispatch hook for the mapped type, and let it bail if desired + if ( special.preDispatch && special.preDispatch.call( this, event ) === false ) { + return; + } + + // Determine handlers + handlerQueue = jQuery.event.handlers.call( this, event, handlers ); + + // Run delegates first; they may want to stop propagation beneath us + i = 0; + while ( ( matched = handlerQueue[ i++ ] ) && !event.isPropagationStopped() ) { + event.currentTarget = matched.elem; + + j = 0; + while ( ( handleObj = matched.handlers[ j++ ] ) && + !event.isImmediatePropagationStopped() ) { + + // If the event is namespaced, then each handler is only invoked if it is + // specially universal or its namespaces are a superset of the event's. + if ( !event.rnamespace || handleObj.namespace === false || + event.rnamespace.test( handleObj.namespace ) ) { + + event.handleObj = handleObj; + event.data = handleObj.data; + + ret = ( ( jQuery.event.special[ handleObj.origType ] || {} ).handle || + handleObj.handler ).apply( matched.elem, args ); + + if ( ret !== undefined ) { + if ( ( event.result = ret ) === false ) { + event.preventDefault(); + event.stopPropagation(); + } + } + } + } + } + + // Call the postDispatch hook for the mapped type + if ( special.postDispatch ) { + special.postDispatch.call( this, event ); + } + + return event.result; + }, + + handlers: function( event, handlers ) { + var i, handleObj, sel, matchedHandlers, matchedSelectors, + handlerQueue = [], + delegateCount = handlers.delegateCount, + cur = event.target; + + // Find delegate handlers + if ( delegateCount && + + // Support: IE <=9 + // Black-hole SVG instance trees (trac-13180) + cur.nodeType && + + // Support: Firefox <=42 + // Suppress spec-violating clicks indicating a non-primary pointer button (trac-3861) + // https://www.w3.org/TR/DOM-Level-3-Events/#event-type-click + // Support: IE 11 only + // ...but not arrow key "clicks" of radio inputs, which can have `button` -1 (gh-2343) + !( event.type === "click" && event.button >= 1 ) ) { + + for ( ; cur !== this; cur = cur.parentNode || this ) { + + // Don't check non-elements (#13208) + // Don't process clicks on disabled elements (#6911, #8165, #11382, #11764) + if ( cur.nodeType === 1 && !( event.type === "click" && cur.disabled === true ) ) { + matchedHandlers = []; + matchedSelectors = {}; + for ( i = 0; i < delegateCount; i++ ) { + handleObj = handlers[ i ]; + + // Don't conflict with Object.prototype properties (#13203) + sel = handleObj.selector + " "; + + if ( matchedSelectors[ sel ] === undefined ) { + matchedSelectors[ sel ] = handleObj.needsContext ? + jQuery( sel, this ).index( cur ) > -1 : + jQuery.find( sel, this, null, [ cur ] ).length; + } + if ( matchedSelectors[ sel ] ) { + matchedHandlers.push( handleObj ); + } + } + if ( matchedHandlers.length ) { + handlerQueue.push( { elem: cur, handlers: matchedHandlers } ); + } + } + } + } + + // Add the remaining (directly-bound) handlers + cur = this; + if ( delegateCount < handlers.length ) { + handlerQueue.push( { elem: cur, handlers: handlers.slice( delegateCount ) } ); + } + + return handlerQueue; + }, + + addProp: function( name, hook ) { + Object.defineProperty( jQuery.Event.prototype, name, { + enumerable: true, + configurable: true, + + get: isFunction( hook ) ? + function() { + if ( this.originalEvent ) { + return hook( this.originalEvent ); + } + } : + function() { + if ( this.originalEvent ) { + return this.originalEvent[ name ]; + } + }, + + set: function( value ) { + Object.defineProperty( this, name, { + enumerable: true, + configurable: true, + writable: true, + value: value + } ); + } + } ); + }, + + fix: function( originalEvent ) { + return originalEvent[ jQuery.expando ] ? + originalEvent : + new jQuery.Event( originalEvent ); + }, + + special: { + load: { + + // Prevent triggered image.load events from bubbling to window.load + noBubble: true + }, + click: { + + // Utilize native event to ensure correct state for checkable inputs + setup: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Claim the first handler + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + // dataPriv.set( el, "click", ... ) + leverageNative( el, "click", returnTrue ); + } + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Force setup before triggering a click + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + leverageNative( el, "click" ); + } + + // Return non-false to allow normal event-path propagation + return true; + }, + + // For cross-browser consistency, suppress native .click() on links + // Also prevent it if we're currently inside a leveraged native-event stack + _default: function( event ) { + var target = event.target; + return rcheckableType.test( target.type ) && + target.click && nodeName( target, "input" ) && + dataPriv.get( target, "click" ) || + nodeName( target, "a" ); + } + }, + + beforeunload: { + postDispatch: function( event ) { + + // Support: Firefox 20+ + // Firefox doesn't alert if the returnValue field is not set. + if ( event.result !== undefined && event.originalEvent ) { + event.originalEvent.returnValue = event.result; + } + } + } + } +}; + +// Ensure the presence of an event listener that handles manually-triggered +// synthetic events by interrupting progress until reinvoked in response to +// *native* events that it fires directly, ensuring that state changes have +// already occurred before other listeners are invoked. +function leverageNative( el, type, expectSync ) { + + // Missing expectSync indicates a trigger call, which must force setup through jQuery.event.add + if ( !expectSync ) { + if ( dataPriv.get( el, type ) === undefined ) { + jQuery.event.add( el, type, returnTrue ); + } + return; + } + + // Register the controller as a special universal handler for all event namespaces + dataPriv.set( el, type, false ); + jQuery.event.add( el, type, { + namespace: false, + handler: function( event ) { + var notAsync, result, + saved = dataPriv.get( this, type ); + + if ( ( event.isTrigger & 1 ) && this[ type ] ) { + + // Interrupt processing of the outer synthetic .trigger()ed event + // Saved data should be false in such cases, but might be a leftover capture object + // from an async native handler (gh-4350) + if ( !saved.length ) { + + // Store arguments for use when handling the inner native event + // There will always be at least one argument (an event object), so this array + // will not be confused with a leftover capture object. + saved = slice.call( arguments ); + dataPriv.set( this, type, saved ); + + // Trigger the native event and capture its result + // Support: IE <=9 - 11+ + // focus() and blur() are asynchronous + notAsync = expectSync( this, type ); + this[ type ](); + result = dataPriv.get( this, type ); + if ( saved !== result || notAsync ) { + dataPriv.set( this, type, false ); + } else { + result = {}; + } + if ( saved !== result ) { + + // Cancel the outer synthetic event + event.stopImmediatePropagation(); + event.preventDefault(); + + // Support: Chrome 86+ + // In Chrome, if an element having a focusout handler is blurred by + // clicking outside of it, it invokes the handler synchronously. If + // that handler calls `.remove()` on the element, the data is cleared, + // leaving `result` undefined. We need to guard against this. + return result && result.value; + } + + // If this is an inner synthetic event for an event with a bubbling surrogate + // (focus or blur), assume that the surrogate already propagated from triggering the + // native event and prevent that from happening again here. + // This technically gets the ordering wrong w.r.t. to `.trigger()` (in which the + // bubbling surrogate propagates *after* the non-bubbling base), but that seems + // less bad than duplication. + } else if ( ( jQuery.event.special[ type ] || {} ).delegateType ) { + event.stopPropagation(); + } + + // If this is a native event triggered above, everything is now in order + // Fire an inner synthetic event with the original arguments + } else if ( saved.length ) { + + // ...and capture the result + dataPriv.set( this, type, { + value: jQuery.event.trigger( + + // Support: IE <=9 - 11+ + // Extend with the prototype to reset the above stopImmediatePropagation() + jQuery.extend( saved[ 0 ], jQuery.Event.prototype ), + saved.slice( 1 ), + this + ) + } ); + + // Abort handling of the native event + event.stopImmediatePropagation(); + } + } + } ); +} + +jQuery.removeEvent = function( elem, type, handle ) { + + // This "if" is needed for plain objects + if ( elem.removeEventListener ) { + elem.removeEventListener( type, handle ); + } +}; + +jQuery.Event = function( src, props ) { + + // Allow instantiation without the 'new' keyword + if ( !( this instanceof jQuery.Event ) ) { + return new jQuery.Event( src, props ); + } + + // Event object + if ( src && src.type ) { + this.originalEvent = src; + this.type = src.type; + + // Events bubbling up the document may have been marked as prevented + // by a handler lower down the tree; reflect the correct value. + this.isDefaultPrevented = src.defaultPrevented || + src.defaultPrevented === undefined && + + // Support: Android <=2.3 only + src.returnValue === false ? + returnTrue : + returnFalse; + + // Create target properties + // Support: Safari <=6 - 7 only + // Target should not be a text node (#504, #13143) + this.target = ( src.target && src.target.nodeType === 3 ) ? + src.target.parentNode : + src.target; + + this.currentTarget = src.currentTarget; + this.relatedTarget = src.relatedTarget; + + // Event type + } else { + this.type = src; + } + + // Put explicitly provided properties onto the event object + if ( props ) { + jQuery.extend( this, props ); + } + + // Create a timestamp if incoming event doesn't have one + this.timeStamp = src && src.timeStamp || Date.now(); + + // Mark it as fixed + this[ jQuery.expando ] = true; +}; + +// jQuery.Event is based on DOM3 Events as specified by the ECMAScript Language Binding +// https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html +jQuery.Event.prototype = { + constructor: jQuery.Event, + isDefaultPrevented: returnFalse, + isPropagationStopped: returnFalse, + isImmediatePropagationStopped: returnFalse, + isSimulated: false, + + preventDefault: function() { + var e = this.originalEvent; + + this.isDefaultPrevented = returnTrue; + + if ( e && !this.isSimulated ) { + e.preventDefault(); + } + }, + stopPropagation: function() { + var e = this.originalEvent; + + this.isPropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopPropagation(); + } + }, + stopImmediatePropagation: function() { + var e = this.originalEvent; + + this.isImmediatePropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopImmediatePropagation(); + } + + this.stopPropagation(); + } +}; + +// Includes all common event props including KeyEvent and MouseEvent specific props +jQuery.each( { + altKey: true, + bubbles: true, + cancelable: true, + changedTouches: true, + ctrlKey: true, + detail: true, + eventPhase: true, + metaKey: true, + pageX: true, + pageY: true, + shiftKey: true, + view: true, + "char": true, + code: true, + charCode: true, + key: true, + keyCode: true, + button: true, + buttons: true, + clientX: true, + clientY: true, + offsetX: true, + offsetY: true, + pointerId: true, + pointerType: true, + screenX: true, + screenY: true, + targetTouches: true, + toElement: true, + touches: true, + which: true +}, jQuery.event.addProp ); + +jQuery.each( { focus: "focusin", blur: "focusout" }, function( type, delegateType ) { + jQuery.event.special[ type ] = { + + // Utilize native event if possible so blur/focus sequence is correct + setup: function() { + + // Claim the first handler + // dataPriv.set( this, "focus", ... ) + // dataPriv.set( this, "blur", ... ) + leverageNative( this, type, expectSync ); + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function() { + + // Force setup before trigger + leverageNative( this, type ); + + // Return non-false to allow normal event-path propagation + return true; + }, + + // Suppress native focus or blur as it's already being fired + // in leverageNative. + _default: function() { + return true; + }, + + delegateType: delegateType + }; +} ); + +// Create mouseenter/leave events using mouseover/out and event-time checks +// so that event delegation works in jQuery. +// Do the same for pointerenter/pointerleave and pointerover/pointerout +// +// Support: Safari 7 only +// Safari sends mouseenter too often; see: +// https://bugs.chromium.org/p/chromium/issues/detail?id=470258 +// for the description of the bug (it existed in older Chrome versions as well). +jQuery.each( { + mouseenter: "mouseover", + mouseleave: "mouseout", + pointerenter: "pointerover", + pointerleave: "pointerout" +}, function( orig, fix ) { + jQuery.event.special[ orig ] = { + delegateType: fix, + bindType: fix, + + handle: function( event ) { + var ret, + target = this, + related = event.relatedTarget, + handleObj = event.handleObj; + + // For mouseenter/leave call the handler if related is outside the target. + // NB: No relatedTarget if the mouse left/entered the browser window + if ( !related || ( related !== target && !jQuery.contains( target, related ) ) ) { + event.type = handleObj.origType; + ret = handleObj.handler.apply( this, arguments ); + event.type = fix; + } + return ret; + } + }; +} ); + +jQuery.fn.extend( { + + on: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn ); + }, + one: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn, 1 ); + }, + off: function( types, selector, fn ) { + var handleObj, type; + if ( types && types.preventDefault && types.handleObj ) { + + // ( event ) dispatched jQuery.Event + handleObj = types.handleObj; + jQuery( types.delegateTarget ).off( + handleObj.namespace ? + handleObj.origType + "." + handleObj.namespace : + handleObj.origType, + handleObj.selector, + handleObj.handler + ); + return this; + } + if ( typeof types === "object" ) { + + // ( types-object [, selector] ) + for ( type in types ) { + this.off( type, selector, types[ type ] ); + } + return this; + } + if ( selector === false || typeof selector === "function" ) { + + // ( types [, fn] ) + fn = selector; + selector = undefined; + } + if ( fn === false ) { + fn = returnFalse; + } + return this.each( function() { + jQuery.event.remove( this, types, fn, selector ); + } ); + } +} ); + + +var + + // Support: IE <=10 - 11, Edge 12 - 13 only + // In IE/Edge using regex groups here causes severe slowdowns. + // See https://connect.microsoft.com/IE/feedback/details/1736512/ + rnoInnerhtml = /\s*$/g; + +// Prefer a tbody over its parent table for containing new rows +function manipulationTarget( elem, content ) { + if ( nodeName( elem, "table" ) && + nodeName( content.nodeType !== 11 ? content : content.firstChild, "tr" ) ) { + + return jQuery( elem ).children( "tbody" )[ 0 ] || elem; + } + + return elem; +} + +// Replace/restore the type attribute of script elements for safe DOM manipulation +function disableScript( elem ) { + elem.type = ( elem.getAttribute( "type" ) !== null ) + "/" + elem.type; + return elem; +} +function restoreScript( elem ) { + if ( ( elem.type || "" ).slice( 0, 5 ) === "true/" ) { + elem.type = elem.type.slice( 5 ); + } else { + elem.removeAttribute( "type" ); + } + + return elem; +} + +function cloneCopyEvent( src, dest ) { + var i, l, type, pdataOld, udataOld, udataCur, events; + + if ( dest.nodeType !== 1 ) { + return; + } + + // 1. Copy private data: events, handlers, etc. + if ( dataPriv.hasData( src ) ) { + pdataOld = dataPriv.get( src ); + events = pdataOld.events; + + if ( events ) { + dataPriv.remove( dest, "handle events" ); + + for ( type in events ) { + for ( i = 0, l = events[ type ].length; i < l; i++ ) { + jQuery.event.add( dest, type, events[ type ][ i ] ); + } + } + } + } + + // 2. Copy user data + if ( dataUser.hasData( src ) ) { + udataOld = dataUser.access( src ); + udataCur = jQuery.extend( {}, udataOld ); + + dataUser.set( dest, udataCur ); + } +} + +// Fix IE bugs, see support tests +function fixInput( src, dest ) { + var nodeName = dest.nodeName.toLowerCase(); + + // Fails to persist the checked state of a cloned checkbox or radio button. + if ( nodeName === "input" && rcheckableType.test( src.type ) ) { + dest.checked = src.checked; + + // Fails to return the selected option to the default selected state when cloning options + } else if ( nodeName === "input" || nodeName === "textarea" ) { + dest.defaultValue = src.defaultValue; + } +} + +function domManip( collection, args, callback, ignored ) { + + // Flatten any nested arrays + args = flat( args ); + + var fragment, first, scripts, hasScripts, node, doc, + i = 0, + l = collection.length, + iNoClone = l - 1, + value = args[ 0 ], + valueIsFunction = isFunction( value ); + + // We can't cloneNode fragments that contain checked, in WebKit + if ( valueIsFunction || + ( l > 1 && typeof value === "string" && + !support.checkClone && rchecked.test( value ) ) ) { + return collection.each( function( index ) { + var self = collection.eq( index ); + if ( valueIsFunction ) { + args[ 0 ] = value.call( this, index, self.html() ); + } + domManip( self, args, callback, ignored ); + } ); + } + + if ( l ) { + fragment = buildFragment( args, collection[ 0 ].ownerDocument, false, collection, ignored ); + first = fragment.firstChild; + + if ( fragment.childNodes.length === 1 ) { + fragment = first; + } + + // Require either new content or an interest in ignored elements to invoke the callback + if ( first || ignored ) { + scripts = jQuery.map( getAll( fragment, "script" ), disableScript ); + hasScripts = scripts.length; + + // Use the original fragment for the last item + // instead of the first because it can end up + // being emptied incorrectly in certain situations (#8070). + for ( ; i < l; i++ ) { + node = fragment; + + if ( i !== iNoClone ) { + node = jQuery.clone( node, true, true ); + + // Keep references to cloned scripts for later restoration + if ( hasScripts ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( scripts, getAll( node, "script" ) ); + } + } + + callback.call( collection[ i ], node, i ); + } + + if ( hasScripts ) { + doc = scripts[ scripts.length - 1 ].ownerDocument; + + // Reenable scripts + jQuery.map( scripts, restoreScript ); + + // Evaluate executable scripts on first document insertion + for ( i = 0; i < hasScripts; i++ ) { + node = scripts[ i ]; + if ( rscriptType.test( node.type || "" ) && + !dataPriv.access( node, "globalEval" ) && + jQuery.contains( doc, node ) ) { + + if ( node.src && ( node.type || "" ).toLowerCase() !== "module" ) { + + // Optional AJAX dependency, but won't run scripts if not present + if ( jQuery._evalUrl && !node.noModule ) { + jQuery._evalUrl( node.src, { + nonce: node.nonce || node.getAttribute( "nonce" ) + }, doc ); + } + } else { + DOMEval( node.textContent.replace( rcleanScript, "" ), node, doc ); + } + } + } + } + } + } + + return collection; +} + +function remove( elem, selector, keepData ) { + var node, + nodes = selector ? jQuery.filter( selector, elem ) : elem, + i = 0; + + for ( ; ( node = nodes[ i ] ) != null; i++ ) { + if ( !keepData && node.nodeType === 1 ) { + jQuery.cleanData( getAll( node ) ); + } + + if ( node.parentNode ) { + if ( keepData && isAttached( node ) ) { + setGlobalEval( getAll( node, "script" ) ); + } + node.parentNode.removeChild( node ); + } + } + + return elem; +} + +jQuery.extend( { + htmlPrefilter: function( html ) { + return html; + }, + + clone: function( elem, dataAndEvents, deepDataAndEvents ) { + var i, l, srcElements, destElements, + clone = elem.cloneNode( true ), + inPage = isAttached( elem ); + + // Fix IE cloning issues + if ( !support.noCloneChecked && ( elem.nodeType === 1 || elem.nodeType === 11 ) && + !jQuery.isXMLDoc( elem ) ) { + + // We eschew Sizzle here for performance reasons: https://jsperf.com/getall-vs-sizzle/2 + destElements = getAll( clone ); + srcElements = getAll( elem ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + fixInput( srcElements[ i ], destElements[ i ] ); + } + } + + // Copy the events from the original to the clone + if ( dataAndEvents ) { + if ( deepDataAndEvents ) { + srcElements = srcElements || getAll( elem ); + destElements = destElements || getAll( clone ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + cloneCopyEvent( srcElements[ i ], destElements[ i ] ); + } + } else { + cloneCopyEvent( elem, clone ); + } + } + + // Preserve script evaluation history + destElements = getAll( clone, "script" ); + if ( destElements.length > 0 ) { + setGlobalEval( destElements, !inPage && getAll( elem, "script" ) ); + } + + // Return the cloned set + return clone; + }, + + cleanData: function( elems ) { + var data, elem, type, + special = jQuery.event.special, + i = 0; + + for ( ; ( elem = elems[ i ] ) !== undefined; i++ ) { + if ( acceptData( elem ) ) { + if ( ( data = elem[ dataPriv.expando ] ) ) { + if ( data.events ) { + for ( type in data.events ) { + if ( special[ type ] ) { + jQuery.event.remove( elem, type ); + + // This is a shortcut to avoid jQuery.event.remove's overhead + } else { + jQuery.removeEvent( elem, type, data.handle ); + } + } + } + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataPriv.expando ] = undefined; + } + if ( elem[ dataUser.expando ] ) { + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataUser.expando ] = undefined; + } + } + } + } +} ); + +jQuery.fn.extend( { + detach: function( selector ) { + return remove( this, selector, true ); + }, + + remove: function( selector ) { + return remove( this, selector ); + }, + + text: function( value ) { + return access( this, function( value ) { + return value === undefined ? + jQuery.text( this ) : + this.empty().each( function() { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + this.textContent = value; + } + } ); + }, null, value, arguments.length ); + }, + + append: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.appendChild( elem ); + } + } ); + }, + + prepend: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.insertBefore( elem, target.firstChild ); + } + } ); + }, + + before: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this ); + } + } ); + }, + + after: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this.nextSibling ); + } + } ); + }, + + empty: function() { + var elem, + i = 0; + + for ( ; ( elem = this[ i ] ) != null; i++ ) { + if ( elem.nodeType === 1 ) { + + // Prevent memory leaks + jQuery.cleanData( getAll( elem, false ) ); + + // Remove any remaining nodes + elem.textContent = ""; + } + } + + return this; + }, + + clone: function( dataAndEvents, deepDataAndEvents ) { + dataAndEvents = dataAndEvents == null ? false : dataAndEvents; + deepDataAndEvents = deepDataAndEvents == null ? dataAndEvents : deepDataAndEvents; + + return this.map( function() { + return jQuery.clone( this, dataAndEvents, deepDataAndEvents ); + } ); + }, + + html: function( value ) { + return access( this, function( value ) { + var elem = this[ 0 ] || {}, + i = 0, + l = this.length; + + if ( value === undefined && elem.nodeType === 1 ) { + return elem.innerHTML; + } + + // See if we can take a shortcut and just use innerHTML + if ( typeof value === "string" && !rnoInnerhtml.test( value ) && + !wrapMap[ ( rtagName.exec( value ) || [ "", "" ] )[ 1 ].toLowerCase() ] ) { + + value = jQuery.htmlPrefilter( value ); + + try { + for ( ; i < l; i++ ) { + elem = this[ i ] || {}; + + // Remove element nodes and prevent memory leaks + if ( elem.nodeType === 1 ) { + jQuery.cleanData( getAll( elem, false ) ); + elem.innerHTML = value; + } + } + + elem = 0; + + // If using innerHTML throws an exception, use the fallback method + } catch ( e ) {} + } + + if ( elem ) { + this.empty().append( value ); + } + }, null, value, arguments.length ); + }, + + replaceWith: function() { + var ignored = []; + + // Make the changes, replacing each non-ignored context element with the new content + return domManip( this, arguments, function( elem ) { + var parent = this.parentNode; + + if ( jQuery.inArray( this, ignored ) < 0 ) { + jQuery.cleanData( getAll( this ) ); + if ( parent ) { + parent.replaceChild( elem, this ); + } + } + + // Force callback invocation + }, ignored ); + } +} ); + +jQuery.each( { + appendTo: "append", + prependTo: "prepend", + insertBefore: "before", + insertAfter: "after", + replaceAll: "replaceWith" +}, function( name, original ) { + jQuery.fn[ name ] = function( selector ) { + var elems, + ret = [], + insert = jQuery( selector ), + last = insert.length - 1, + i = 0; + + for ( ; i <= last; i++ ) { + elems = i === last ? this : this.clone( true ); + jQuery( insert[ i ] )[ original ]( elems ); + + // Support: Android <=4.0 only, PhantomJS 1 only + // .get() because push.apply(_, arraylike) throws on ancient WebKit + push.apply( ret, elems.get() ); + } + + return this.pushStack( ret ); + }; +} ); +var rnumnonpx = new RegExp( "^(" + pnum + ")(?!px)[a-z%]+$", "i" ); + +var getStyles = function( elem ) { + + // Support: IE <=11 only, Firefox <=30 (#15098, #14150) + // IE throws on elements created in popups + // FF meanwhile throws on frame elements through "defaultView.getComputedStyle" + var view = elem.ownerDocument.defaultView; + + if ( !view || !view.opener ) { + view = window; + } + + return view.getComputedStyle( elem ); + }; + +var swap = function( elem, options, callback ) { + var ret, name, + old = {}; + + // Remember the old values, and insert the new ones + for ( name in options ) { + old[ name ] = elem.style[ name ]; + elem.style[ name ] = options[ name ]; + } + + ret = callback.call( elem ); + + // Revert the old values + for ( name in options ) { + elem.style[ name ] = old[ name ]; + } + + return ret; +}; + + +var rboxStyle = new RegExp( cssExpand.join( "|" ), "i" ); + + + +( function() { + + // Executing both pixelPosition & boxSizingReliable tests require only one layout + // so they're executed at the same time to save the second computation. + function computeStyleTests() { + + // This is a singleton, we need to execute it only once + if ( !div ) { + return; + } + + container.style.cssText = "position:absolute;left:-11111px;width:60px;" + + "margin-top:1px;padding:0;border:0"; + div.style.cssText = + "position:relative;display:block;box-sizing:border-box;overflow:scroll;" + + "margin:auto;border:1px;padding:1px;" + + "width:60%;top:1%"; + documentElement.appendChild( container ).appendChild( div ); + + var divStyle = window.getComputedStyle( div ); + pixelPositionVal = divStyle.top !== "1%"; + + // Support: Android 4.0 - 4.3 only, Firefox <=3 - 44 + reliableMarginLeftVal = roundPixelMeasures( divStyle.marginLeft ) === 12; + + // Support: Android 4.0 - 4.3 only, Safari <=9.1 - 10.1, iOS <=7.0 - 9.3 + // Some styles come back with percentage values, even though they shouldn't + div.style.right = "60%"; + pixelBoxStylesVal = roundPixelMeasures( divStyle.right ) === 36; + + // Support: IE 9 - 11 only + // Detect misreporting of content dimensions for box-sizing:border-box elements + boxSizingReliableVal = roundPixelMeasures( divStyle.width ) === 36; + + // Support: IE 9 only + // Detect overflow:scroll screwiness (gh-3699) + // Support: Chrome <=64 + // Don't get tricked when zoom affects offsetWidth (gh-4029) + div.style.position = "absolute"; + scrollboxSizeVal = roundPixelMeasures( div.offsetWidth / 3 ) === 12; + + documentElement.removeChild( container ); + + // Nullify the div so it wouldn't be stored in the memory and + // it will also be a sign that checks already performed + div = null; + } + + function roundPixelMeasures( measure ) { + return Math.round( parseFloat( measure ) ); + } + + var pixelPositionVal, boxSizingReliableVal, scrollboxSizeVal, pixelBoxStylesVal, + reliableTrDimensionsVal, reliableMarginLeftVal, + container = document.createElement( "div" ), + div = document.createElement( "div" ); + + // Finish early in limited (non-browser) environments + if ( !div.style ) { + return; + } + + // Support: IE <=9 - 11 only + // Style of cloned element affects source element cloned (#8908) + div.style.backgroundClip = "content-box"; + div.cloneNode( true ).style.backgroundClip = ""; + support.clearCloneStyle = div.style.backgroundClip === "content-box"; + + jQuery.extend( support, { + boxSizingReliable: function() { + computeStyleTests(); + return boxSizingReliableVal; + }, + pixelBoxStyles: function() { + computeStyleTests(); + return pixelBoxStylesVal; + }, + pixelPosition: function() { + computeStyleTests(); + return pixelPositionVal; + }, + reliableMarginLeft: function() { + computeStyleTests(); + return reliableMarginLeftVal; + }, + scrollboxSize: function() { + computeStyleTests(); + return scrollboxSizeVal; + }, + + // Support: IE 9 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Behavior in IE 9 is more subtle than in newer versions & it passes + // some versions of this test; make sure not to make it pass there! + // + // Support: Firefox 70+ + // Only Firefox includes border widths + // in computed dimensions. (gh-4529) + reliableTrDimensions: function() { + var table, tr, trChild, trStyle; + if ( reliableTrDimensionsVal == null ) { + table = document.createElement( "table" ); + tr = document.createElement( "tr" ); + trChild = document.createElement( "div" ); + + table.style.cssText = "position:absolute;left:-11111px;border-collapse:separate"; + tr.style.cssText = "border:1px solid"; + + // Support: Chrome 86+ + // Height set through cssText does not get applied. + // Computed height then comes back as 0. + tr.style.height = "1px"; + trChild.style.height = "9px"; + + // Support: Android 8 Chrome 86+ + // In our bodyBackground.html iframe, + // display for all div elements is set to "inline", + // which causes a problem only in Android 8 Chrome 86. + // Ensuring the div is display: block + // gets around this issue. + trChild.style.display = "block"; + + documentElement + .appendChild( table ) + .appendChild( tr ) + .appendChild( trChild ); + + trStyle = window.getComputedStyle( tr ); + reliableTrDimensionsVal = ( parseInt( trStyle.height, 10 ) + + parseInt( trStyle.borderTopWidth, 10 ) + + parseInt( trStyle.borderBottomWidth, 10 ) ) === tr.offsetHeight; + + documentElement.removeChild( table ); + } + return reliableTrDimensionsVal; + } + } ); +} )(); + + +function curCSS( elem, name, computed ) { + var width, minWidth, maxWidth, ret, + + // Support: Firefox 51+ + // Retrieving style before computed somehow + // fixes an issue with getting wrong values + // on detached elements + style = elem.style; + + computed = computed || getStyles( elem ); + + // getPropertyValue is needed for: + // .css('filter') (IE 9 only, #12537) + // .css('--customProperty) (#3144) + if ( computed ) { + ret = computed.getPropertyValue( name ) || computed[ name ]; + + if ( ret === "" && !isAttached( elem ) ) { + ret = jQuery.style( elem, name ); + } + + // A tribute to the "awesome hack by Dean Edwards" + // Android Browser returns percentage for some values, + // but width seems to be reliably pixels. + // This is against the CSSOM draft spec: + // https://drafts.csswg.org/cssom/#resolved-values + if ( !support.pixelBoxStyles() && rnumnonpx.test( ret ) && rboxStyle.test( name ) ) { + + // Remember the original values + width = style.width; + minWidth = style.minWidth; + maxWidth = style.maxWidth; + + // Put in the new values to get a computed value out + style.minWidth = style.maxWidth = style.width = ret; + ret = computed.width; + + // Revert the changed values + style.width = width; + style.minWidth = minWidth; + style.maxWidth = maxWidth; + } + } + + return ret !== undefined ? + + // Support: IE <=9 - 11 only + // IE returns zIndex value as an integer. + ret + "" : + ret; +} + + +function addGetHookIf( conditionFn, hookFn ) { + + // Define the hook, we'll check on the first run if it's really needed. + return { + get: function() { + if ( conditionFn() ) { + + // Hook not needed (or it's not possible to use it due + // to missing dependency), remove it. + delete this.get; + return; + } + + // Hook needed; redefine it so that the support test is not executed again. + return ( this.get = hookFn ).apply( this, arguments ); + } + }; +} + + +var cssPrefixes = [ "Webkit", "Moz", "ms" ], + emptyStyle = document.createElement( "div" ).style, + vendorProps = {}; + +// Return a vendor-prefixed property or undefined +function vendorPropName( name ) { + + // Check for vendor prefixed names + var capName = name[ 0 ].toUpperCase() + name.slice( 1 ), + i = cssPrefixes.length; + + while ( i-- ) { + name = cssPrefixes[ i ] + capName; + if ( name in emptyStyle ) { + return name; + } + } +} + +// Return a potentially-mapped jQuery.cssProps or vendor prefixed property +function finalPropName( name ) { + var final = jQuery.cssProps[ name ] || vendorProps[ name ]; + + if ( final ) { + return final; + } + if ( name in emptyStyle ) { + return name; + } + return vendorProps[ name ] = vendorPropName( name ) || name; +} + + +var + + // Swappable if display is none or starts with table + // except "table", "table-cell", or "table-caption" + // See here for display values: https://developer.mozilla.org/en-US/docs/CSS/display + rdisplayswap = /^(none|table(?!-c[ea]).+)/, + rcustomProp = /^--/, + cssShow = { position: "absolute", visibility: "hidden", display: "block" }, + cssNormalTransform = { + letterSpacing: "0", + fontWeight: "400" + }; + +function setPositiveNumber( _elem, value, subtract ) { + + // Any relative (+/-) values have already been + // normalized at this point + var matches = rcssNum.exec( value ); + return matches ? + + // Guard against undefined "subtract", e.g., when used as in cssHooks + Math.max( 0, matches[ 2 ] - ( subtract || 0 ) ) + ( matches[ 3 ] || "px" ) : + value; +} + +function boxModelAdjustment( elem, dimension, box, isBorderBox, styles, computedVal ) { + var i = dimension === "width" ? 1 : 0, + extra = 0, + delta = 0; + + // Adjustment may not be necessary + if ( box === ( isBorderBox ? "border" : "content" ) ) { + return 0; + } + + for ( ; i < 4; i += 2 ) { + + // Both box models exclude margin + if ( box === "margin" ) { + delta += jQuery.css( elem, box + cssExpand[ i ], true, styles ); + } + + // If we get here with a content-box, we're seeking "padding" or "border" or "margin" + if ( !isBorderBox ) { + + // Add padding + delta += jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + + // For "border" or "margin", add border + if ( box !== "padding" ) { + delta += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + + // But still keep track of it otherwise + } else { + extra += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + + // If we get here with a border-box (content + padding + border), we're seeking "content" or + // "padding" or "margin" + } else { + + // For "content", subtract padding + if ( box === "content" ) { + delta -= jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + } + + // For "content" or "padding", subtract border + if ( box !== "margin" ) { + delta -= jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + } + } + + // Account for positive content-box scroll gutter when requested by providing computedVal + if ( !isBorderBox && computedVal >= 0 ) { + + // offsetWidth/offsetHeight is a rounded sum of content, padding, scroll gutter, and border + // Assuming integer scroll gutter, subtract the rest and round down + delta += Math.max( 0, Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + computedVal - + delta - + extra - + 0.5 + + // If offsetWidth/offsetHeight is unknown, then we can't determine content-box scroll gutter + // Use an explicit zero to avoid NaN (gh-3964) + ) ) || 0; + } + + return delta; +} + +function getWidthOrHeight( elem, dimension, extra ) { + + // Start with computed style + var styles = getStyles( elem ), + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-4322). + // Fake content-box until we know it's needed to know the true value. + boxSizingNeeded = !support.boxSizingReliable() || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + valueIsBorderBox = isBorderBox, + + val = curCSS( elem, dimension, styles ), + offsetProp = "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ); + + // Support: Firefox <=54 + // Return a confounding non-pixel value or feign ignorance, as appropriate. + if ( rnumnonpx.test( val ) ) { + if ( !extra ) { + return val; + } + val = "auto"; + } + + + // Support: IE 9 - 11 only + // Use offsetWidth/offsetHeight for when box sizing is unreliable. + // In those cases, the computed value can be trusted to be border-box. + if ( ( !support.boxSizingReliable() && isBorderBox || + + // Support: IE 10 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Interestingly, in some cases IE 9 doesn't suffer from this issue. + !support.reliableTrDimensions() && nodeName( elem, "tr" ) || + + // Fall back to offsetWidth/offsetHeight when value is "auto" + // This happens for inline elements with no explicit setting (gh-3571) + val === "auto" || + + // Support: Android <=4.1 - 4.3 only + // Also use offsetWidth/offsetHeight for misreported inline dimensions (gh-3602) + !parseFloat( val ) && jQuery.css( elem, "display", false, styles ) === "inline" ) && + + // Make sure the element is visible & connected + elem.getClientRects().length ) { + + isBorderBox = jQuery.css( elem, "boxSizing", false, styles ) === "border-box"; + + // Where available, offsetWidth/offsetHeight approximate border box dimensions. + // Where not available (e.g., SVG), assume unreliable box-sizing and interpret the + // retrieved value as a content box dimension. + valueIsBorderBox = offsetProp in elem; + if ( valueIsBorderBox ) { + val = elem[ offsetProp ]; + } + } + + // Normalize "" and auto + val = parseFloat( val ) || 0; + + // Adjust for the element's box model + return ( val + + boxModelAdjustment( + elem, + dimension, + extra || ( isBorderBox ? "border" : "content" ), + valueIsBorderBox, + styles, + + // Provide the current computed size to request scroll gutter calculation (gh-3589) + val + ) + ) + "px"; +} + +jQuery.extend( { + + // Add in style property hooks for overriding the default + // behavior of getting and setting a style property + cssHooks: { + opacity: { + get: function( elem, computed ) { + if ( computed ) { + + // We should always get a number back from opacity + var ret = curCSS( elem, "opacity" ); + return ret === "" ? "1" : ret; + } + } + } + }, + + // Don't automatically add "px" to these possibly-unitless properties + cssNumber: { + "animationIterationCount": true, + "columnCount": true, + "fillOpacity": true, + "flexGrow": true, + "flexShrink": true, + "fontWeight": true, + "gridArea": true, + "gridColumn": true, + "gridColumnEnd": true, + "gridColumnStart": true, + "gridRow": true, + "gridRowEnd": true, + "gridRowStart": true, + "lineHeight": true, + "opacity": true, + "order": true, + "orphans": true, + "widows": true, + "zIndex": true, + "zoom": true + }, + + // Add in properties whose names you wish to fix before + // setting or getting the value + cssProps: {}, + + // Get and set the style property on a DOM Node + style: function( elem, name, value, extra ) { + + // Don't set styles on text and comment nodes + if ( !elem || elem.nodeType === 3 || elem.nodeType === 8 || !elem.style ) { + return; + } + + // Make sure that we're working with the right name + var ret, type, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ), + style = elem.style; + + // Make sure that we're working with the right name. We don't + // want to query the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Gets hook for the prefixed version, then unprefixed version + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // Check if we're setting a value + if ( value !== undefined ) { + type = typeof value; + + // Convert "+=" or "-=" to relative numbers (#7345) + if ( type === "string" && ( ret = rcssNum.exec( value ) ) && ret[ 1 ] ) { + value = adjustCSS( elem, name, ret ); + + // Fixes bug #9237 + type = "number"; + } + + // Make sure that null and NaN values aren't set (#7116) + if ( value == null || value !== value ) { + return; + } + + // If a number was passed in, add the unit (except for certain CSS properties) + // The isCustomProp check can be removed in jQuery 4.0 when we only auto-append + // "px" to a few hardcoded values. + if ( type === "number" && !isCustomProp ) { + value += ret && ret[ 3 ] || ( jQuery.cssNumber[ origName ] ? "" : "px" ); + } + + // background-* props affect original clone's values + if ( !support.clearCloneStyle && value === "" && name.indexOf( "background" ) === 0 ) { + style[ name ] = "inherit"; + } + + // If a hook was provided, use that value, otherwise just set the specified value + if ( !hooks || !( "set" in hooks ) || + ( value = hooks.set( elem, value, extra ) ) !== undefined ) { + + if ( isCustomProp ) { + style.setProperty( name, value ); + } else { + style[ name ] = value; + } + } + + } else { + + // If a hook was provided get the non-computed value from there + if ( hooks && "get" in hooks && + ( ret = hooks.get( elem, false, extra ) ) !== undefined ) { + + return ret; + } + + // Otherwise just get the value from the style object + return style[ name ]; + } + }, + + css: function( elem, name, extra, styles ) { + var val, num, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ); + + // Make sure that we're working with the right name. We don't + // want to modify the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Try prefixed name followed by the unprefixed name + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // If a hook was provided get the computed value from there + if ( hooks && "get" in hooks ) { + val = hooks.get( elem, true, extra ); + } + + // Otherwise, if a way to get the computed value exists, use that + if ( val === undefined ) { + val = curCSS( elem, name, styles ); + } + + // Convert "normal" to computed value + if ( val === "normal" && name in cssNormalTransform ) { + val = cssNormalTransform[ name ]; + } + + // Make numeric if forced or a qualifier was provided and val looks numeric + if ( extra === "" || extra ) { + num = parseFloat( val ); + return extra === true || isFinite( num ) ? num || 0 : val; + } + + return val; + } +} ); + +jQuery.each( [ "height", "width" ], function( _i, dimension ) { + jQuery.cssHooks[ dimension ] = { + get: function( elem, computed, extra ) { + if ( computed ) { + + // Certain elements can have dimension info if we invisibly show them + // but it must have a current display style that would benefit + return rdisplayswap.test( jQuery.css( elem, "display" ) ) && + + // Support: Safari 8+ + // Table columns in Safari have non-zero offsetWidth & zero + // getBoundingClientRect().width unless display is changed. + // Support: IE <=11 only + // Running getBoundingClientRect on a disconnected node + // in IE throws an error. + ( !elem.getClientRects().length || !elem.getBoundingClientRect().width ) ? + swap( elem, cssShow, function() { + return getWidthOrHeight( elem, dimension, extra ); + } ) : + getWidthOrHeight( elem, dimension, extra ); + } + }, + + set: function( elem, value, extra ) { + var matches, + styles = getStyles( elem ), + + // Only read styles.position if the test has a chance to fail + // to avoid forcing a reflow. + scrollboxSizeBuggy = !support.scrollboxSize() && + styles.position === "absolute", + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-3991) + boxSizingNeeded = scrollboxSizeBuggy || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + subtract = extra ? + boxModelAdjustment( + elem, + dimension, + extra, + isBorderBox, + styles + ) : + 0; + + // Account for unreliable border-box dimensions by comparing offset* to computed and + // faking a content-box to get border and padding (gh-3699) + if ( isBorderBox && scrollboxSizeBuggy ) { + subtract -= Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + parseFloat( styles[ dimension ] ) - + boxModelAdjustment( elem, dimension, "border", false, styles ) - + 0.5 + ); + } + + // Convert to pixels if value adjustment is needed + if ( subtract && ( matches = rcssNum.exec( value ) ) && + ( matches[ 3 ] || "px" ) !== "px" ) { + + elem.style[ dimension ] = value; + value = jQuery.css( elem, dimension ); + } + + return setPositiveNumber( elem, value, subtract ); + } + }; +} ); + +jQuery.cssHooks.marginLeft = addGetHookIf( support.reliableMarginLeft, + function( elem, computed ) { + if ( computed ) { + return ( parseFloat( curCSS( elem, "marginLeft" ) ) || + elem.getBoundingClientRect().left - + swap( elem, { marginLeft: 0 }, function() { + return elem.getBoundingClientRect().left; + } ) + ) + "px"; + } + } +); + +// These hooks are used by animate to expand properties +jQuery.each( { + margin: "", + padding: "", + border: "Width" +}, function( prefix, suffix ) { + jQuery.cssHooks[ prefix + suffix ] = { + expand: function( value ) { + var i = 0, + expanded = {}, + + // Assumes a single number if not a string + parts = typeof value === "string" ? value.split( " " ) : [ value ]; + + for ( ; i < 4; i++ ) { + expanded[ prefix + cssExpand[ i ] + suffix ] = + parts[ i ] || parts[ i - 2 ] || parts[ 0 ]; + } + + return expanded; + } + }; + + if ( prefix !== "margin" ) { + jQuery.cssHooks[ prefix + suffix ].set = setPositiveNumber; + } +} ); + +jQuery.fn.extend( { + css: function( name, value ) { + return access( this, function( elem, name, value ) { + var styles, len, + map = {}, + i = 0; + + if ( Array.isArray( name ) ) { + styles = getStyles( elem ); + len = name.length; + + for ( ; i < len; i++ ) { + map[ name[ i ] ] = jQuery.css( elem, name[ i ], false, styles ); + } + + return map; + } + + return value !== undefined ? + jQuery.style( elem, name, value ) : + jQuery.css( elem, name ); + }, name, value, arguments.length > 1 ); + } +} ); + + +function Tween( elem, options, prop, end, easing ) { + return new Tween.prototype.init( elem, options, prop, end, easing ); +} +jQuery.Tween = Tween; + +Tween.prototype = { + constructor: Tween, + init: function( elem, options, prop, end, easing, unit ) { + this.elem = elem; + this.prop = prop; + this.easing = easing || jQuery.easing._default; + this.options = options; + this.start = this.now = this.cur(); + this.end = end; + this.unit = unit || ( jQuery.cssNumber[ prop ] ? "" : "px" ); + }, + cur: function() { + var hooks = Tween.propHooks[ this.prop ]; + + return hooks && hooks.get ? + hooks.get( this ) : + Tween.propHooks._default.get( this ); + }, + run: function( percent ) { + var eased, + hooks = Tween.propHooks[ this.prop ]; + + if ( this.options.duration ) { + this.pos = eased = jQuery.easing[ this.easing ]( + percent, this.options.duration * percent, 0, 1, this.options.duration + ); + } else { + this.pos = eased = percent; + } + this.now = ( this.end - this.start ) * eased + this.start; + + if ( this.options.step ) { + this.options.step.call( this.elem, this.now, this ); + } + + if ( hooks && hooks.set ) { + hooks.set( this ); + } else { + Tween.propHooks._default.set( this ); + } + return this; + } +}; + +Tween.prototype.init.prototype = Tween.prototype; + +Tween.propHooks = { + _default: { + get: function( tween ) { + var result; + + // Use a property on the element directly when it is not a DOM element, + // or when there is no matching style property that exists. + if ( tween.elem.nodeType !== 1 || + tween.elem[ tween.prop ] != null && tween.elem.style[ tween.prop ] == null ) { + return tween.elem[ tween.prop ]; + } + + // Passing an empty string as a 3rd parameter to .css will automatically + // attempt a parseFloat and fallback to a string if the parse fails. + // Simple values such as "10px" are parsed to Float; + // complex values such as "rotate(1rad)" are returned as-is. + result = jQuery.css( tween.elem, tween.prop, "" ); + + // Empty strings, null, undefined and "auto" are converted to 0. + return !result || result === "auto" ? 0 : result; + }, + set: function( tween ) { + + // Use step hook for back compat. + // Use cssHook if its there. + // Use .style if available and use plain properties where available. + if ( jQuery.fx.step[ tween.prop ] ) { + jQuery.fx.step[ tween.prop ]( tween ); + } else if ( tween.elem.nodeType === 1 && ( + jQuery.cssHooks[ tween.prop ] || + tween.elem.style[ finalPropName( tween.prop ) ] != null ) ) { + jQuery.style( tween.elem, tween.prop, tween.now + tween.unit ); + } else { + tween.elem[ tween.prop ] = tween.now; + } + } + } +}; + +// Support: IE <=9 only +// Panic based approach to setting things on disconnected nodes +Tween.propHooks.scrollTop = Tween.propHooks.scrollLeft = { + set: function( tween ) { + if ( tween.elem.nodeType && tween.elem.parentNode ) { + tween.elem[ tween.prop ] = tween.now; + } + } +}; + +jQuery.easing = { + linear: function( p ) { + return p; + }, + swing: function( p ) { + return 0.5 - Math.cos( p * Math.PI ) / 2; + }, + _default: "swing" +}; + +jQuery.fx = Tween.prototype.init; + +// Back compat <1.8 extension point +jQuery.fx.step = {}; + + + + +var + fxNow, inProgress, + rfxtypes = /^(?:toggle|show|hide)$/, + rrun = /queueHooks$/; + +function schedule() { + if ( inProgress ) { + if ( document.hidden === false && window.requestAnimationFrame ) { + window.requestAnimationFrame( schedule ); + } else { + window.setTimeout( schedule, jQuery.fx.interval ); + } + + jQuery.fx.tick(); + } +} + +// Animations created synchronously will run synchronously +function createFxNow() { + window.setTimeout( function() { + fxNow = undefined; + } ); + return ( fxNow = Date.now() ); +} + +// Generate parameters to create a standard animation +function genFx( type, includeWidth ) { + var which, + i = 0, + attrs = { height: type }; + + // If we include width, step value is 1 to do all cssExpand values, + // otherwise step value is 2 to skip over Left and Right + includeWidth = includeWidth ? 1 : 0; + for ( ; i < 4; i += 2 - includeWidth ) { + which = cssExpand[ i ]; + attrs[ "margin" + which ] = attrs[ "padding" + which ] = type; + } + + if ( includeWidth ) { + attrs.opacity = attrs.width = type; + } + + return attrs; +} + +function createTween( value, prop, animation ) { + var tween, + collection = ( Animation.tweeners[ prop ] || [] ).concat( Animation.tweeners[ "*" ] ), + index = 0, + length = collection.length; + for ( ; index < length; index++ ) { + if ( ( tween = collection[ index ].call( animation, prop, value ) ) ) { + + // We're done with this property + return tween; + } + } +} + +function defaultPrefilter( elem, props, opts ) { + var prop, value, toggle, hooks, oldfire, propTween, restoreDisplay, display, + isBox = "width" in props || "height" in props, + anim = this, + orig = {}, + style = elem.style, + hidden = elem.nodeType && isHiddenWithinTree( elem ), + dataShow = dataPriv.get( elem, "fxshow" ); + + // Queue-skipping animations hijack the fx hooks + if ( !opts.queue ) { + hooks = jQuery._queueHooks( elem, "fx" ); + if ( hooks.unqueued == null ) { + hooks.unqueued = 0; + oldfire = hooks.empty.fire; + hooks.empty.fire = function() { + if ( !hooks.unqueued ) { + oldfire(); + } + }; + } + hooks.unqueued++; + + anim.always( function() { + + // Ensure the complete handler is called before this completes + anim.always( function() { + hooks.unqueued--; + if ( !jQuery.queue( elem, "fx" ).length ) { + hooks.empty.fire(); + } + } ); + } ); + } + + // Detect show/hide animations + for ( prop in props ) { + value = props[ prop ]; + if ( rfxtypes.test( value ) ) { + delete props[ prop ]; + toggle = toggle || value === "toggle"; + if ( value === ( hidden ? "hide" : "show" ) ) { + + // Pretend to be hidden if this is a "show" and + // there is still data from a stopped show/hide + if ( value === "show" && dataShow && dataShow[ prop ] !== undefined ) { + hidden = true; + + // Ignore all other no-op show/hide data + } else { + continue; + } + } + orig[ prop ] = dataShow && dataShow[ prop ] || jQuery.style( elem, prop ); + } + } + + // Bail out if this is a no-op like .hide().hide() + propTween = !jQuery.isEmptyObject( props ); + if ( !propTween && jQuery.isEmptyObject( orig ) ) { + return; + } + + // Restrict "overflow" and "display" styles during box animations + if ( isBox && elem.nodeType === 1 ) { + + // Support: IE <=9 - 11, Edge 12 - 15 + // Record all 3 overflow attributes because IE does not infer the shorthand + // from identically-valued overflowX and overflowY and Edge just mirrors + // the overflowX value there. + opts.overflow = [ style.overflow, style.overflowX, style.overflowY ]; + + // Identify a display type, preferring old show/hide data over the CSS cascade + restoreDisplay = dataShow && dataShow.display; + if ( restoreDisplay == null ) { + restoreDisplay = dataPriv.get( elem, "display" ); + } + display = jQuery.css( elem, "display" ); + if ( display === "none" ) { + if ( restoreDisplay ) { + display = restoreDisplay; + } else { + + // Get nonempty value(s) by temporarily forcing visibility + showHide( [ elem ], true ); + restoreDisplay = elem.style.display || restoreDisplay; + display = jQuery.css( elem, "display" ); + showHide( [ elem ] ); + } + } + + // Animate inline elements as inline-block + if ( display === "inline" || display === "inline-block" && restoreDisplay != null ) { + if ( jQuery.css( elem, "float" ) === "none" ) { + + // Restore the original display value at the end of pure show/hide animations + if ( !propTween ) { + anim.done( function() { + style.display = restoreDisplay; + } ); + if ( restoreDisplay == null ) { + display = style.display; + restoreDisplay = display === "none" ? "" : display; + } + } + style.display = "inline-block"; + } + } + } + + if ( opts.overflow ) { + style.overflow = "hidden"; + anim.always( function() { + style.overflow = opts.overflow[ 0 ]; + style.overflowX = opts.overflow[ 1 ]; + style.overflowY = opts.overflow[ 2 ]; + } ); + } + + // Implement show/hide animations + propTween = false; + for ( prop in orig ) { + + // General show/hide setup for this element animation + if ( !propTween ) { + if ( dataShow ) { + if ( "hidden" in dataShow ) { + hidden = dataShow.hidden; + } + } else { + dataShow = dataPriv.access( elem, "fxshow", { display: restoreDisplay } ); + } + + // Store hidden/visible for toggle so `.stop().toggle()` "reverses" + if ( toggle ) { + dataShow.hidden = !hidden; + } + + // Show elements before animating them + if ( hidden ) { + showHide( [ elem ], true ); + } + + /* eslint-disable no-loop-func */ + + anim.done( function() { + + /* eslint-enable no-loop-func */ + + // The final step of a "hide" animation is actually hiding the element + if ( !hidden ) { + showHide( [ elem ] ); + } + dataPriv.remove( elem, "fxshow" ); + for ( prop in orig ) { + jQuery.style( elem, prop, orig[ prop ] ); + } + } ); + } + + // Per-property setup + propTween = createTween( hidden ? dataShow[ prop ] : 0, prop, anim ); + if ( !( prop in dataShow ) ) { + dataShow[ prop ] = propTween.start; + if ( hidden ) { + propTween.end = propTween.start; + propTween.start = 0; + } + } + } +} + +function propFilter( props, specialEasing ) { + var index, name, easing, value, hooks; + + // camelCase, specialEasing and expand cssHook pass + for ( index in props ) { + name = camelCase( index ); + easing = specialEasing[ name ]; + value = props[ index ]; + if ( Array.isArray( value ) ) { + easing = value[ 1 ]; + value = props[ index ] = value[ 0 ]; + } + + if ( index !== name ) { + props[ name ] = value; + delete props[ index ]; + } + + hooks = jQuery.cssHooks[ name ]; + if ( hooks && "expand" in hooks ) { + value = hooks.expand( value ); + delete props[ name ]; + + // Not quite $.extend, this won't overwrite existing keys. + // Reusing 'index' because we have the correct "name" + for ( index in value ) { + if ( !( index in props ) ) { + props[ index ] = value[ index ]; + specialEasing[ index ] = easing; + } + } + } else { + specialEasing[ name ] = easing; + } + } +} + +function Animation( elem, properties, options ) { + var result, + stopped, + index = 0, + length = Animation.prefilters.length, + deferred = jQuery.Deferred().always( function() { + + // Don't match elem in the :animated selector + delete tick.elem; + } ), + tick = function() { + if ( stopped ) { + return false; + } + var currentTime = fxNow || createFxNow(), + remaining = Math.max( 0, animation.startTime + animation.duration - currentTime ), + + // Support: Android 2.3 only + // Archaic crash bug won't allow us to use `1 - ( 0.5 || 0 )` (#12497) + temp = remaining / animation.duration || 0, + percent = 1 - temp, + index = 0, + length = animation.tweens.length; + + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( percent ); + } + + deferred.notifyWith( elem, [ animation, percent, remaining ] ); + + // If there's more to do, yield + if ( percent < 1 && length ) { + return remaining; + } + + // If this was an empty animation, synthesize a final progress notification + if ( !length ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + } + + // Resolve the animation and report its conclusion + deferred.resolveWith( elem, [ animation ] ); + return false; + }, + animation = deferred.promise( { + elem: elem, + props: jQuery.extend( {}, properties ), + opts: jQuery.extend( true, { + specialEasing: {}, + easing: jQuery.easing._default + }, options ), + originalProperties: properties, + originalOptions: options, + startTime: fxNow || createFxNow(), + duration: options.duration, + tweens: [], + createTween: function( prop, end ) { + var tween = jQuery.Tween( elem, animation.opts, prop, end, + animation.opts.specialEasing[ prop ] || animation.opts.easing ); + animation.tweens.push( tween ); + return tween; + }, + stop: function( gotoEnd ) { + var index = 0, + + // If we are going to the end, we want to run all the tweens + // otherwise we skip this part + length = gotoEnd ? animation.tweens.length : 0; + if ( stopped ) { + return this; + } + stopped = true; + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( 1 ); + } + + // Resolve when we played the last frame; otherwise, reject + if ( gotoEnd ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + deferred.resolveWith( elem, [ animation, gotoEnd ] ); + } else { + deferred.rejectWith( elem, [ animation, gotoEnd ] ); + } + return this; + } + } ), + props = animation.props; + + propFilter( props, animation.opts.specialEasing ); + + for ( ; index < length; index++ ) { + result = Animation.prefilters[ index ].call( animation, elem, props, animation.opts ); + if ( result ) { + if ( isFunction( result.stop ) ) { + jQuery._queueHooks( animation.elem, animation.opts.queue ).stop = + result.stop.bind( result ); + } + return result; + } + } + + jQuery.map( props, createTween, animation ); + + if ( isFunction( animation.opts.start ) ) { + animation.opts.start.call( elem, animation ); + } + + // Attach callbacks from options + animation + .progress( animation.opts.progress ) + .done( animation.opts.done, animation.opts.complete ) + .fail( animation.opts.fail ) + .always( animation.opts.always ); + + jQuery.fx.timer( + jQuery.extend( tick, { + elem: elem, + anim: animation, + queue: animation.opts.queue + } ) + ); + + return animation; +} + +jQuery.Animation = jQuery.extend( Animation, { + + tweeners: { + "*": [ function( prop, value ) { + var tween = this.createTween( prop, value ); + adjustCSS( tween.elem, prop, rcssNum.exec( value ), tween ); + return tween; + } ] + }, + + tweener: function( props, callback ) { + if ( isFunction( props ) ) { + callback = props; + props = [ "*" ]; + } else { + props = props.match( rnothtmlwhite ); + } + + var prop, + index = 0, + length = props.length; + + for ( ; index < length; index++ ) { + prop = props[ index ]; + Animation.tweeners[ prop ] = Animation.tweeners[ prop ] || []; + Animation.tweeners[ prop ].unshift( callback ); + } + }, + + prefilters: [ defaultPrefilter ], + + prefilter: function( callback, prepend ) { + if ( prepend ) { + Animation.prefilters.unshift( callback ); + } else { + Animation.prefilters.push( callback ); + } + } +} ); + +jQuery.speed = function( speed, easing, fn ) { + var opt = speed && typeof speed === "object" ? jQuery.extend( {}, speed ) : { + complete: fn || !fn && easing || + isFunction( speed ) && speed, + duration: speed, + easing: fn && easing || easing && !isFunction( easing ) && easing + }; + + // Go to the end state if fx are off + if ( jQuery.fx.off ) { + opt.duration = 0; + + } else { + if ( typeof opt.duration !== "number" ) { + if ( opt.duration in jQuery.fx.speeds ) { + opt.duration = jQuery.fx.speeds[ opt.duration ]; + + } else { + opt.duration = jQuery.fx.speeds._default; + } + } + } + + // Normalize opt.queue - true/undefined/null -> "fx" + if ( opt.queue == null || opt.queue === true ) { + opt.queue = "fx"; + } + + // Queueing + opt.old = opt.complete; + + opt.complete = function() { + if ( isFunction( opt.old ) ) { + opt.old.call( this ); + } + + if ( opt.queue ) { + jQuery.dequeue( this, opt.queue ); + } + }; + + return opt; +}; + +jQuery.fn.extend( { + fadeTo: function( speed, to, easing, callback ) { + + // Show any hidden elements after setting opacity to 0 + return this.filter( isHiddenWithinTree ).css( "opacity", 0 ).show() + + // Animate to the value specified + .end().animate( { opacity: to }, speed, easing, callback ); + }, + animate: function( prop, speed, easing, callback ) { + var empty = jQuery.isEmptyObject( prop ), + optall = jQuery.speed( speed, easing, callback ), + doAnimation = function() { + + // Operate on a copy of prop so per-property easing won't be lost + var anim = Animation( this, jQuery.extend( {}, prop ), optall ); + + // Empty animations, or finishing resolves immediately + if ( empty || dataPriv.get( this, "finish" ) ) { + anim.stop( true ); + } + }; + + doAnimation.finish = doAnimation; + + return empty || optall.queue === false ? + this.each( doAnimation ) : + this.queue( optall.queue, doAnimation ); + }, + stop: function( type, clearQueue, gotoEnd ) { + var stopQueue = function( hooks ) { + var stop = hooks.stop; + delete hooks.stop; + stop( gotoEnd ); + }; + + if ( typeof type !== "string" ) { + gotoEnd = clearQueue; + clearQueue = type; + type = undefined; + } + if ( clearQueue ) { + this.queue( type || "fx", [] ); + } + + return this.each( function() { + var dequeue = true, + index = type != null && type + "queueHooks", + timers = jQuery.timers, + data = dataPriv.get( this ); + + if ( index ) { + if ( data[ index ] && data[ index ].stop ) { + stopQueue( data[ index ] ); + } + } else { + for ( index in data ) { + if ( data[ index ] && data[ index ].stop && rrun.test( index ) ) { + stopQueue( data[ index ] ); + } + } + } + + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && + ( type == null || timers[ index ].queue === type ) ) { + + timers[ index ].anim.stop( gotoEnd ); + dequeue = false; + timers.splice( index, 1 ); + } + } + + // Start the next in the queue if the last step wasn't forced. + // Timers currently will call their complete callbacks, which + // will dequeue but only if they were gotoEnd. + if ( dequeue || !gotoEnd ) { + jQuery.dequeue( this, type ); + } + } ); + }, + finish: function( type ) { + if ( type !== false ) { + type = type || "fx"; + } + return this.each( function() { + var index, + data = dataPriv.get( this ), + queue = data[ type + "queue" ], + hooks = data[ type + "queueHooks" ], + timers = jQuery.timers, + length = queue ? queue.length : 0; + + // Enable finishing flag on private data + data.finish = true; + + // Empty the queue first + jQuery.queue( this, type, [] ); + + if ( hooks && hooks.stop ) { + hooks.stop.call( this, true ); + } + + // Look for any active animations, and finish them + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && timers[ index ].queue === type ) { + timers[ index ].anim.stop( true ); + timers.splice( index, 1 ); + } + } + + // Look for any animations in the old queue and finish them + for ( index = 0; index < length; index++ ) { + if ( queue[ index ] && queue[ index ].finish ) { + queue[ index ].finish.call( this ); + } + } + + // Turn off finishing flag + delete data.finish; + } ); + } +} ); + +jQuery.each( [ "toggle", "show", "hide" ], function( _i, name ) { + var cssFn = jQuery.fn[ name ]; + jQuery.fn[ name ] = function( speed, easing, callback ) { + return speed == null || typeof speed === "boolean" ? + cssFn.apply( this, arguments ) : + this.animate( genFx( name, true ), speed, easing, callback ); + }; +} ); + +// Generate shortcuts for custom animations +jQuery.each( { + slideDown: genFx( "show" ), + slideUp: genFx( "hide" ), + slideToggle: genFx( "toggle" ), + fadeIn: { opacity: "show" }, + fadeOut: { opacity: "hide" }, + fadeToggle: { opacity: "toggle" } +}, function( name, props ) { + jQuery.fn[ name ] = function( speed, easing, callback ) { + return this.animate( props, speed, easing, callback ); + }; +} ); + +jQuery.timers = []; +jQuery.fx.tick = function() { + var timer, + i = 0, + timers = jQuery.timers; + + fxNow = Date.now(); + + for ( ; i < timers.length; i++ ) { + timer = timers[ i ]; + + // Run the timer and safely remove it when done (allowing for external removal) + if ( !timer() && timers[ i ] === timer ) { + timers.splice( i--, 1 ); + } + } + + if ( !timers.length ) { + jQuery.fx.stop(); + } + fxNow = undefined; +}; + +jQuery.fx.timer = function( timer ) { + jQuery.timers.push( timer ); + jQuery.fx.start(); +}; + +jQuery.fx.interval = 13; +jQuery.fx.start = function() { + if ( inProgress ) { + return; + } + + inProgress = true; + schedule(); +}; + +jQuery.fx.stop = function() { + inProgress = null; +}; + +jQuery.fx.speeds = { + slow: 600, + fast: 200, + + // Default speed + _default: 400 +}; + + +// Based off of the plugin by Clint Helfers, with permission. +// https://web.archive.org/web/20100324014747/http://blindsignals.com/index.php/2009/07/jquery-delay/ +jQuery.fn.delay = function( time, type ) { + time = jQuery.fx ? jQuery.fx.speeds[ time ] || time : time; + type = type || "fx"; + + return this.queue( type, function( next, hooks ) { + var timeout = window.setTimeout( next, time ); + hooks.stop = function() { + window.clearTimeout( timeout ); + }; + } ); +}; + + +( function() { + var input = document.createElement( "input" ), + select = document.createElement( "select" ), + opt = select.appendChild( document.createElement( "option" ) ); + + input.type = "checkbox"; + + // Support: Android <=4.3 only + // Default value for a checkbox should be "on" + support.checkOn = input.value !== ""; + + // Support: IE <=11 only + // Must access selectedIndex to make default options select + support.optSelected = opt.selected; + + // Support: IE <=11 only + // An input loses its value after becoming a radio + input = document.createElement( "input" ); + input.value = "t"; + input.type = "radio"; + support.radioValue = input.value === "t"; +} )(); + + +var boolHook, + attrHandle = jQuery.expr.attrHandle; + +jQuery.fn.extend( { + attr: function( name, value ) { + return access( this, jQuery.attr, name, value, arguments.length > 1 ); + }, + + removeAttr: function( name ) { + return this.each( function() { + jQuery.removeAttr( this, name ); + } ); + } +} ); + +jQuery.extend( { + attr: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set attributes on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + // Fallback to prop when attributes are not supported + if ( typeof elem.getAttribute === "undefined" ) { + return jQuery.prop( elem, name, value ); + } + + // Attribute hooks are determined by the lowercase version + // Grab necessary hook if one is defined + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + hooks = jQuery.attrHooks[ name.toLowerCase() ] || + ( jQuery.expr.match.bool.test( name ) ? boolHook : undefined ); + } + + if ( value !== undefined ) { + if ( value === null ) { + jQuery.removeAttr( elem, name ); + return; + } + + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + elem.setAttribute( name, value + "" ); + return value; + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + ret = jQuery.find.attr( elem, name ); + + // Non-existent attributes return null, we normalize to undefined + return ret == null ? undefined : ret; + }, + + attrHooks: { + type: { + set: function( elem, value ) { + if ( !support.radioValue && value === "radio" && + nodeName( elem, "input" ) ) { + var val = elem.value; + elem.setAttribute( "type", value ); + if ( val ) { + elem.value = val; + } + return value; + } + } + } + }, + + removeAttr: function( elem, value ) { + var name, + i = 0, + + // Attribute names can contain non-HTML whitespace characters + // https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 + attrNames = value && value.match( rnothtmlwhite ); + + if ( attrNames && elem.nodeType === 1 ) { + while ( ( name = attrNames[ i++ ] ) ) { + elem.removeAttribute( name ); + } + } + } +} ); + +// Hooks for boolean attributes +boolHook = { + set: function( elem, value, name ) { + if ( value === false ) { + + // Remove boolean attributes when set to false + jQuery.removeAttr( elem, name ); + } else { + elem.setAttribute( name, name ); + } + return name; + } +}; + +jQuery.each( jQuery.expr.match.bool.source.match( /\w+/g ), function( _i, name ) { + var getter = attrHandle[ name ] || jQuery.find.attr; + + attrHandle[ name ] = function( elem, name, isXML ) { + var ret, handle, + lowercaseName = name.toLowerCase(); + + if ( !isXML ) { + + // Avoid an infinite loop by temporarily removing this function from the getter + handle = attrHandle[ lowercaseName ]; + attrHandle[ lowercaseName ] = ret; + ret = getter( elem, name, isXML ) != null ? + lowercaseName : + null; + attrHandle[ lowercaseName ] = handle; + } + return ret; + }; +} ); + + + + +var rfocusable = /^(?:input|select|textarea|button)$/i, + rclickable = /^(?:a|area)$/i; + +jQuery.fn.extend( { + prop: function( name, value ) { + return access( this, jQuery.prop, name, value, arguments.length > 1 ); + }, + + removeProp: function( name ) { + return this.each( function() { + delete this[ jQuery.propFix[ name ] || name ]; + } ); + } +} ); + +jQuery.extend( { + prop: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set properties on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + + // Fix name and attach hooks + name = jQuery.propFix[ name ] || name; + hooks = jQuery.propHooks[ name ]; + } + + if ( value !== undefined ) { + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + return ( elem[ name ] = value ); + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + return elem[ name ]; + }, + + propHooks: { + tabIndex: { + get: function( elem ) { + + // Support: IE <=9 - 11 only + // elem.tabIndex doesn't always return the + // correct value when it hasn't been explicitly set + // https://web.archive.org/web/20141116233347/http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ + // Use proper attribute retrieval(#12072) + var tabindex = jQuery.find.attr( elem, "tabindex" ); + + if ( tabindex ) { + return parseInt( tabindex, 10 ); + } + + if ( + rfocusable.test( elem.nodeName ) || + rclickable.test( elem.nodeName ) && + elem.href + ) { + return 0; + } + + return -1; + } + } + }, + + propFix: { + "for": "htmlFor", + "class": "className" + } +} ); + +// Support: IE <=11 only +// Accessing the selectedIndex property +// forces the browser to respect setting selected +// on the option +// The getter ensures a default option is selected +// when in an optgroup +// eslint rule "no-unused-expressions" is disabled for this code +// since it considers such accessions noop +if ( !support.optSelected ) { + jQuery.propHooks.selected = { + get: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent && parent.parentNode ) { + parent.parentNode.selectedIndex; + } + return null; + }, + set: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent ) { + parent.selectedIndex; + + if ( parent.parentNode ) { + parent.parentNode.selectedIndex; + } + } + } + }; +} + +jQuery.each( [ + "tabIndex", + "readOnly", + "maxLength", + "cellSpacing", + "cellPadding", + "rowSpan", + "colSpan", + "useMap", + "frameBorder", + "contentEditable" +], function() { + jQuery.propFix[ this.toLowerCase() ] = this; +} ); + + + + + // Strip and collapse whitespace according to HTML spec + // https://infra.spec.whatwg.org/#strip-and-collapse-ascii-whitespace + function stripAndCollapse( value ) { + var tokens = value.match( rnothtmlwhite ) || []; + return tokens.join( " " ); + } + + +function getClass( elem ) { + return elem.getAttribute && elem.getAttribute( "class" ) || ""; +} + +function classesToArray( value ) { + if ( Array.isArray( value ) ) { + return value; + } + if ( typeof value === "string" ) { + return value.match( rnothtmlwhite ) || []; + } + return []; +} + +jQuery.fn.extend( { + addClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).addClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + if ( cur.indexOf( " " + clazz + " " ) < 0 ) { + cur += clazz + " "; + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + removeClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).removeClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + if ( !arguments.length ) { + return this.attr( "class", "" ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + + // This expression is here for better compressibility (see addClass) + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + + // Remove *all* instances + while ( cur.indexOf( " " + clazz + " " ) > -1 ) { + cur = cur.replace( " " + clazz + " ", " " ); + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + toggleClass: function( value, stateVal ) { + var type = typeof value, + isValidValue = type === "string" || Array.isArray( value ); + + if ( typeof stateVal === "boolean" && isValidValue ) { + return stateVal ? this.addClass( value ) : this.removeClass( value ); + } + + if ( isFunction( value ) ) { + return this.each( function( i ) { + jQuery( this ).toggleClass( + value.call( this, i, getClass( this ), stateVal ), + stateVal + ); + } ); + } + + return this.each( function() { + var className, i, self, classNames; + + if ( isValidValue ) { + + // Toggle individual class names + i = 0; + self = jQuery( this ); + classNames = classesToArray( value ); + + while ( ( className = classNames[ i++ ] ) ) { + + // Check each className given, space separated list + if ( self.hasClass( className ) ) { + self.removeClass( className ); + } else { + self.addClass( className ); + } + } + + // Toggle whole class name + } else if ( value === undefined || type === "boolean" ) { + className = getClass( this ); + if ( className ) { + + // Store className if set + dataPriv.set( this, "__className__", className ); + } + + // If the element has a class name or if we're passed `false`, + // then remove the whole classname (if there was one, the above saved it). + // Otherwise bring back whatever was previously saved (if anything), + // falling back to the empty string if nothing was stored. + if ( this.setAttribute ) { + this.setAttribute( "class", + className || value === false ? + "" : + dataPriv.get( this, "__className__" ) || "" + ); + } + } + } ); + }, + + hasClass: function( selector ) { + var className, elem, + i = 0; + + className = " " + selector + " "; + while ( ( elem = this[ i++ ] ) ) { + if ( elem.nodeType === 1 && + ( " " + stripAndCollapse( getClass( elem ) ) + " " ).indexOf( className ) > -1 ) { + return true; + } + } + + return false; + } +} ); + + + + +var rreturn = /\r/g; + +jQuery.fn.extend( { + val: function( value ) { + var hooks, ret, valueIsFunction, + elem = this[ 0 ]; + + if ( !arguments.length ) { + if ( elem ) { + hooks = jQuery.valHooks[ elem.type ] || + jQuery.valHooks[ elem.nodeName.toLowerCase() ]; + + if ( hooks && + "get" in hooks && + ( ret = hooks.get( elem, "value" ) ) !== undefined + ) { + return ret; + } + + ret = elem.value; + + // Handle most common string cases + if ( typeof ret === "string" ) { + return ret.replace( rreturn, "" ); + } + + // Handle cases where value is null/undef or number + return ret == null ? "" : ret; + } + + return; + } + + valueIsFunction = isFunction( value ); + + return this.each( function( i ) { + var val; + + if ( this.nodeType !== 1 ) { + return; + } + + if ( valueIsFunction ) { + val = value.call( this, i, jQuery( this ).val() ); + } else { + val = value; + } + + // Treat null/undefined as ""; convert numbers to string + if ( val == null ) { + val = ""; + + } else if ( typeof val === "number" ) { + val += ""; + + } else if ( Array.isArray( val ) ) { + val = jQuery.map( val, function( value ) { + return value == null ? "" : value + ""; + } ); + } + + hooks = jQuery.valHooks[ this.type ] || jQuery.valHooks[ this.nodeName.toLowerCase() ]; + + // If set returns undefined, fall back to normal setting + if ( !hooks || !( "set" in hooks ) || hooks.set( this, val, "value" ) === undefined ) { + this.value = val; + } + } ); + } +} ); + +jQuery.extend( { + valHooks: { + option: { + get: function( elem ) { + + var val = jQuery.find.attr( elem, "value" ); + return val != null ? + val : + + // Support: IE <=10 - 11 only + // option.text throws exceptions (#14686, #14858) + // Strip and collapse whitespace + // https://html.spec.whatwg.org/#strip-and-collapse-whitespace + stripAndCollapse( jQuery.text( elem ) ); + } + }, + select: { + get: function( elem ) { + var value, option, i, + options = elem.options, + index = elem.selectedIndex, + one = elem.type === "select-one", + values = one ? null : [], + max = one ? index + 1 : options.length; + + if ( index < 0 ) { + i = max; + + } else { + i = one ? index : 0; + } + + // Loop through all the selected options + for ( ; i < max; i++ ) { + option = options[ i ]; + + // Support: IE <=9 only + // IE8-9 doesn't update selected after form reset (#2551) + if ( ( option.selected || i === index ) && + + // Don't return options that are disabled or in a disabled optgroup + !option.disabled && + ( !option.parentNode.disabled || + !nodeName( option.parentNode, "optgroup" ) ) ) { + + // Get the specific value for the option + value = jQuery( option ).val(); + + // We don't need an array for one selects + if ( one ) { + return value; + } + + // Multi-Selects return an array + values.push( value ); + } + } + + return values; + }, + + set: function( elem, value ) { + var optionSet, option, + options = elem.options, + values = jQuery.makeArray( value ), + i = options.length; + + while ( i-- ) { + option = options[ i ]; + + /* eslint-disable no-cond-assign */ + + if ( option.selected = + jQuery.inArray( jQuery.valHooks.option.get( option ), values ) > -1 + ) { + optionSet = true; + } + + /* eslint-enable no-cond-assign */ + } + + // Force browsers to behave consistently when non-matching value is set + if ( !optionSet ) { + elem.selectedIndex = -1; + } + return values; + } + } + } +} ); + +// Radios and checkboxes getter/setter +jQuery.each( [ "radio", "checkbox" ], function() { + jQuery.valHooks[ this ] = { + set: function( elem, value ) { + if ( Array.isArray( value ) ) { + return ( elem.checked = jQuery.inArray( jQuery( elem ).val(), value ) > -1 ); + } + } + }; + if ( !support.checkOn ) { + jQuery.valHooks[ this ].get = function( elem ) { + return elem.getAttribute( "value" ) === null ? "on" : elem.value; + }; + } +} ); + + + + +// Return jQuery for attributes-only inclusion + + +support.focusin = "onfocusin" in window; + + +var rfocusMorph = /^(?:focusinfocus|focusoutblur)$/, + stopPropagationCallback = function( e ) { + e.stopPropagation(); + }; + +jQuery.extend( jQuery.event, { + + trigger: function( event, data, elem, onlyHandlers ) { + + var i, cur, tmp, bubbleType, ontype, handle, special, lastElement, + eventPath = [ elem || document ], + type = hasOwn.call( event, "type" ) ? event.type : event, + namespaces = hasOwn.call( event, "namespace" ) ? event.namespace.split( "." ) : []; + + cur = lastElement = tmp = elem = elem || document; + + // Don't do events on text and comment nodes + if ( elem.nodeType === 3 || elem.nodeType === 8 ) { + return; + } + + // focus/blur morphs to focusin/out; ensure we're not firing them right now + if ( rfocusMorph.test( type + jQuery.event.triggered ) ) { + return; + } + + if ( type.indexOf( "." ) > -1 ) { + + // Namespaced trigger; create a regexp to match event type in handle() + namespaces = type.split( "." ); + type = namespaces.shift(); + namespaces.sort(); + } + ontype = type.indexOf( ":" ) < 0 && "on" + type; + + // Caller can pass in a jQuery.Event object, Object, or just an event type string + event = event[ jQuery.expando ] ? + event : + new jQuery.Event( type, typeof event === "object" && event ); + + // Trigger bitmask: & 1 for native handlers; & 2 for jQuery (always true) + event.isTrigger = onlyHandlers ? 2 : 3; + event.namespace = namespaces.join( "." ); + event.rnamespace = event.namespace ? + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ) : + null; + + // Clean up the event in case it is being reused + event.result = undefined; + if ( !event.target ) { + event.target = elem; + } + + // Clone any incoming data and prepend the event, creating the handler arg list + data = data == null ? + [ event ] : + jQuery.makeArray( data, [ event ] ); + + // Allow special events to draw outside the lines + special = jQuery.event.special[ type ] || {}; + if ( !onlyHandlers && special.trigger && special.trigger.apply( elem, data ) === false ) { + return; + } + + // Determine event propagation path in advance, per W3C events spec (#9951) + // Bubble up to document, then to window; watch for a global ownerDocument var (#9724) + if ( !onlyHandlers && !special.noBubble && !isWindow( elem ) ) { + + bubbleType = special.delegateType || type; + if ( !rfocusMorph.test( bubbleType + type ) ) { + cur = cur.parentNode; + } + for ( ; cur; cur = cur.parentNode ) { + eventPath.push( cur ); + tmp = cur; + } + + // Only add window if we got to document (e.g., not plain obj or detached DOM) + if ( tmp === ( elem.ownerDocument || document ) ) { + eventPath.push( tmp.defaultView || tmp.parentWindow || window ); + } + } + + // Fire handlers on the event path + i = 0; + while ( ( cur = eventPath[ i++ ] ) && !event.isPropagationStopped() ) { + lastElement = cur; + event.type = i > 1 ? + bubbleType : + special.bindType || type; + + // jQuery handler + handle = ( dataPriv.get( cur, "events" ) || Object.create( null ) )[ event.type ] && + dataPriv.get( cur, "handle" ); + if ( handle ) { + handle.apply( cur, data ); + } + + // Native handler + handle = ontype && cur[ ontype ]; + if ( handle && handle.apply && acceptData( cur ) ) { + event.result = handle.apply( cur, data ); + if ( event.result === false ) { + event.preventDefault(); + } + } + } + event.type = type; + + // If nobody prevented the default action, do it now + if ( !onlyHandlers && !event.isDefaultPrevented() ) { + + if ( ( !special._default || + special._default.apply( eventPath.pop(), data ) === false ) && + acceptData( elem ) ) { + + // Call a native DOM method on the target with the same name as the event. + // Don't do default actions on window, that's where global variables be (#6170) + if ( ontype && isFunction( elem[ type ] ) && !isWindow( elem ) ) { + + // Don't re-trigger an onFOO event when we call its FOO() method + tmp = elem[ ontype ]; + + if ( tmp ) { + elem[ ontype ] = null; + } + + // Prevent re-triggering of the same event, since we already bubbled it above + jQuery.event.triggered = type; + + if ( event.isPropagationStopped() ) { + lastElement.addEventListener( type, stopPropagationCallback ); + } + + elem[ type ](); + + if ( event.isPropagationStopped() ) { + lastElement.removeEventListener( type, stopPropagationCallback ); + } + + jQuery.event.triggered = undefined; + + if ( tmp ) { + elem[ ontype ] = tmp; + } + } + } + } + + return event.result; + }, + + // Piggyback on a donor event to simulate a different one + // Used only for `focus(in | out)` events + simulate: function( type, elem, event ) { + var e = jQuery.extend( + new jQuery.Event(), + event, + { + type: type, + isSimulated: true + } + ); + + jQuery.event.trigger( e, null, elem ); + } + +} ); + +jQuery.fn.extend( { + + trigger: function( type, data ) { + return this.each( function() { + jQuery.event.trigger( type, data, this ); + } ); + }, + triggerHandler: function( type, data ) { + var elem = this[ 0 ]; + if ( elem ) { + return jQuery.event.trigger( type, data, elem, true ); + } + } +} ); + + +// Support: Firefox <=44 +// Firefox doesn't have focus(in | out) events +// Related ticket - https://bugzilla.mozilla.org/show_bug.cgi?id=687787 +// +// Support: Chrome <=48 - 49, Safari <=9.0 - 9.1 +// focus(in | out) events fire after focus & blur events, +// which is spec violation - http://www.w3.org/TR/DOM-Level-3-Events/#events-focusevent-event-order +// Related ticket - https://bugs.chromium.org/p/chromium/issues/detail?id=449857 +if ( !support.focusin ) { + jQuery.each( { focus: "focusin", blur: "focusout" }, function( orig, fix ) { + + // Attach a single capturing handler on the document while someone wants focusin/focusout + var handler = function( event ) { + jQuery.event.simulate( fix, event.target, jQuery.event.fix( event ) ); + }; + + jQuery.event.special[ fix ] = { + setup: function() { + + // Handle: regular nodes (via `this.ownerDocument`), window + // (via `this.document`) & document (via `this`). + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ); + + if ( !attaches ) { + doc.addEventListener( orig, handler, true ); + } + dataPriv.access( doc, fix, ( attaches || 0 ) + 1 ); + }, + teardown: function() { + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ) - 1; + + if ( !attaches ) { + doc.removeEventListener( orig, handler, true ); + dataPriv.remove( doc, fix ); + + } else { + dataPriv.access( doc, fix, attaches ); + } + } + }; + } ); +} +var location = window.location; + +var nonce = { guid: Date.now() }; + +var rquery = ( /\?/ ); + + + +// Cross-browser xml parsing +jQuery.parseXML = function( data ) { + var xml, parserErrorElem; + if ( !data || typeof data !== "string" ) { + return null; + } + + // Support: IE 9 - 11 only + // IE throws on parseFromString with invalid input. + try { + xml = ( new window.DOMParser() ).parseFromString( data, "text/xml" ); + } catch ( e ) {} + + parserErrorElem = xml && xml.getElementsByTagName( "parsererror" )[ 0 ]; + if ( !xml || parserErrorElem ) { + jQuery.error( "Invalid XML: " + ( + parserErrorElem ? + jQuery.map( parserErrorElem.childNodes, function( el ) { + return el.textContent; + } ).join( "\n" ) : + data + ) ); + } + return xml; +}; + + +var + rbracket = /\[\]$/, + rCRLF = /\r?\n/g, + rsubmitterTypes = /^(?:submit|button|image|reset|file)$/i, + rsubmittable = /^(?:input|select|textarea|keygen)/i; + +function buildParams( prefix, obj, traditional, add ) { + var name; + + if ( Array.isArray( obj ) ) { + + // Serialize array item. + jQuery.each( obj, function( i, v ) { + if ( traditional || rbracket.test( prefix ) ) { + + // Treat each array item as a scalar. + add( prefix, v ); + + } else { + + // Item is non-scalar (array or object), encode its numeric index. + buildParams( + prefix + "[" + ( typeof v === "object" && v != null ? i : "" ) + "]", + v, + traditional, + add + ); + } + } ); + + } else if ( !traditional && toType( obj ) === "object" ) { + + // Serialize object item. + for ( name in obj ) { + buildParams( prefix + "[" + name + "]", obj[ name ], traditional, add ); + } + + } else { + + // Serialize scalar item. + add( prefix, obj ); + } +} + +// Serialize an array of form elements or a set of +// key/values into a query string +jQuery.param = function( a, traditional ) { + var prefix, + s = [], + add = function( key, valueOrFunction ) { + + // If value is a function, invoke it and use its return value + var value = isFunction( valueOrFunction ) ? + valueOrFunction() : + valueOrFunction; + + s[ s.length ] = encodeURIComponent( key ) + "=" + + encodeURIComponent( value == null ? "" : value ); + }; + + if ( a == null ) { + return ""; + } + + // If an array was passed in, assume that it is an array of form elements. + if ( Array.isArray( a ) || ( a.jquery && !jQuery.isPlainObject( a ) ) ) { + + // Serialize the form elements + jQuery.each( a, function() { + add( this.name, this.value ); + } ); + + } else { + + // If traditional, encode the "old" way (the way 1.3.2 or older + // did it), otherwise encode params recursively. + for ( prefix in a ) { + buildParams( prefix, a[ prefix ], traditional, add ); + } + } + + // Return the resulting serialization + return s.join( "&" ); +}; + +jQuery.fn.extend( { + serialize: function() { + return jQuery.param( this.serializeArray() ); + }, + serializeArray: function() { + return this.map( function() { + + // Can add propHook for "elements" to filter or add form elements + var elements = jQuery.prop( this, "elements" ); + return elements ? jQuery.makeArray( elements ) : this; + } ).filter( function() { + var type = this.type; + + // Use .is( ":disabled" ) so that fieldset[disabled] works + return this.name && !jQuery( this ).is( ":disabled" ) && + rsubmittable.test( this.nodeName ) && !rsubmitterTypes.test( type ) && + ( this.checked || !rcheckableType.test( type ) ); + } ).map( function( _i, elem ) { + var val = jQuery( this ).val(); + + if ( val == null ) { + return null; + } + + if ( Array.isArray( val ) ) { + return jQuery.map( val, function( val ) { + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ); + } + + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ).get(); + } +} ); + + +var + r20 = /%20/g, + rhash = /#.*$/, + rantiCache = /([?&])_=[^&]*/, + rheaders = /^(.*?):[ \t]*([^\r\n]*)$/mg, + + // #7653, #8125, #8152: local protocol detection + rlocalProtocol = /^(?:about|app|app-storage|.+-extension|file|res|widget):$/, + rnoContent = /^(?:GET|HEAD)$/, + rprotocol = /^\/\//, + + /* Prefilters + * 1) They are useful to introduce custom dataTypes (see ajax/jsonp.js for an example) + * 2) These are called: + * - BEFORE asking for a transport + * - AFTER param serialization (s.data is a string if s.processData is true) + * 3) key is the dataType + * 4) the catchall symbol "*" can be used + * 5) execution will start with transport dataType and THEN continue down to "*" if needed + */ + prefilters = {}, + + /* Transports bindings + * 1) key is the dataType + * 2) the catchall symbol "*" can be used + * 3) selection will start with transport dataType and THEN go to "*" if needed + */ + transports = {}, + + // Avoid comment-prolog char sequence (#10098); must appease lint and evade compression + allTypes = "*/".concat( "*" ), + + // Anchor tag for parsing the document origin + originAnchor = document.createElement( "a" ); + +originAnchor.href = location.href; + +// Base "constructor" for jQuery.ajaxPrefilter and jQuery.ajaxTransport +function addToPrefiltersOrTransports( structure ) { + + // dataTypeExpression is optional and defaults to "*" + return function( dataTypeExpression, func ) { + + if ( typeof dataTypeExpression !== "string" ) { + func = dataTypeExpression; + dataTypeExpression = "*"; + } + + var dataType, + i = 0, + dataTypes = dataTypeExpression.toLowerCase().match( rnothtmlwhite ) || []; + + if ( isFunction( func ) ) { + + // For each dataType in the dataTypeExpression + while ( ( dataType = dataTypes[ i++ ] ) ) { + + // Prepend if requested + if ( dataType[ 0 ] === "+" ) { + dataType = dataType.slice( 1 ) || "*"; + ( structure[ dataType ] = structure[ dataType ] || [] ).unshift( func ); + + // Otherwise append + } else { + ( structure[ dataType ] = structure[ dataType ] || [] ).push( func ); + } + } + } + }; +} + +// Base inspection function for prefilters and transports +function inspectPrefiltersOrTransports( structure, options, originalOptions, jqXHR ) { + + var inspected = {}, + seekingTransport = ( structure === transports ); + + function inspect( dataType ) { + var selected; + inspected[ dataType ] = true; + jQuery.each( structure[ dataType ] || [], function( _, prefilterOrFactory ) { + var dataTypeOrTransport = prefilterOrFactory( options, originalOptions, jqXHR ); + if ( typeof dataTypeOrTransport === "string" && + !seekingTransport && !inspected[ dataTypeOrTransport ] ) { + + options.dataTypes.unshift( dataTypeOrTransport ); + inspect( dataTypeOrTransport ); + return false; + } else if ( seekingTransport ) { + return !( selected = dataTypeOrTransport ); + } + } ); + return selected; + } + + return inspect( options.dataTypes[ 0 ] ) || !inspected[ "*" ] && inspect( "*" ); +} + +// A special extend for ajax options +// that takes "flat" options (not to be deep extended) +// Fixes #9887 +function ajaxExtend( target, src ) { + var key, deep, + flatOptions = jQuery.ajaxSettings.flatOptions || {}; + + for ( key in src ) { + if ( src[ key ] !== undefined ) { + ( flatOptions[ key ] ? target : ( deep || ( deep = {} ) ) )[ key ] = src[ key ]; + } + } + if ( deep ) { + jQuery.extend( true, target, deep ); + } + + return target; +} + +/* Handles responses to an ajax request: + * - finds the right dataType (mediates between content-type and expected dataType) + * - returns the corresponding response + */ +function ajaxHandleResponses( s, jqXHR, responses ) { + + var ct, type, finalDataType, firstDataType, + contents = s.contents, + dataTypes = s.dataTypes; + + // Remove auto dataType and get content-type in the process + while ( dataTypes[ 0 ] === "*" ) { + dataTypes.shift(); + if ( ct === undefined ) { + ct = s.mimeType || jqXHR.getResponseHeader( "Content-Type" ); + } + } + + // Check if we're dealing with a known content-type + if ( ct ) { + for ( type in contents ) { + if ( contents[ type ] && contents[ type ].test( ct ) ) { + dataTypes.unshift( type ); + break; + } + } + } + + // Check to see if we have a response for the expected dataType + if ( dataTypes[ 0 ] in responses ) { + finalDataType = dataTypes[ 0 ]; + } else { + + // Try convertible dataTypes + for ( type in responses ) { + if ( !dataTypes[ 0 ] || s.converters[ type + " " + dataTypes[ 0 ] ] ) { + finalDataType = type; + break; + } + if ( !firstDataType ) { + firstDataType = type; + } + } + + // Or just use first one + finalDataType = finalDataType || firstDataType; + } + + // If we found a dataType + // We add the dataType to the list if needed + // and return the corresponding response + if ( finalDataType ) { + if ( finalDataType !== dataTypes[ 0 ] ) { + dataTypes.unshift( finalDataType ); + } + return responses[ finalDataType ]; + } +} + +/* Chain conversions given the request and the original response + * Also sets the responseXXX fields on the jqXHR instance + */ +function ajaxConvert( s, response, jqXHR, isSuccess ) { + var conv2, current, conv, tmp, prev, + converters = {}, + + // Work with a copy of dataTypes in case we need to modify it for conversion + dataTypes = s.dataTypes.slice(); + + // Create converters map with lowercased keys + if ( dataTypes[ 1 ] ) { + for ( conv in s.converters ) { + converters[ conv.toLowerCase() ] = s.converters[ conv ]; + } + } + + current = dataTypes.shift(); + + // Convert to each sequential dataType + while ( current ) { + + if ( s.responseFields[ current ] ) { + jqXHR[ s.responseFields[ current ] ] = response; + } + + // Apply the dataFilter if provided + if ( !prev && isSuccess && s.dataFilter ) { + response = s.dataFilter( response, s.dataType ); + } + + prev = current; + current = dataTypes.shift(); + + if ( current ) { + + // There's only work to do if current dataType is non-auto + if ( current === "*" ) { + + current = prev; + + // Convert response if prev dataType is non-auto and differs from current + } else if ( prev !== "*" && prev !== current ) { + + // Seek a direct converter + conv = converters[ prev + " " + current ] || converters[ "* " + current ]; + + // If none found, seek a pair + if ( !conv ) { + for ( conv2 in converters ) { + + // If conv2 outputs current + tmp = conv2.split( " " ); + if ( tmp[ 1 ] === current ) { + + // If prev can be converted to accepted input + conv = converters[ prev + " " + tmp[ 0 ] ] || + converters[ "* " + tmp[ 0 ] ]; + if ( conv ) { + + // Condense equivalence converters + if ( conv === true ) { + conv = converters[ conv2 ]; + + // Otherwise, insert the intermediate dataType + } else if ( converters[ conv2 ] !== true ) { + current = tmp[ 0 ]; + dataTypes.unshift( tmp[ 1 ] ); + } + break; + } + } + } + } + + // Apply converter (if not an equivalence) + if ( conv !== true ) { + + // Unless errors are allowed to bubble, catch and return them + if ( conv && s.throws ) { + response = conv( response ); + } else { + try { + response = conv( response ); + } catch ( e ) { + return { + state: "parsererror", + error: conv ? e : "No conversion from " + prev + " to " + current + }; + } + } + } + } + } + } + + return { state: "success", data: response }; +} + +jQuery.extend( { + + // Counter for holding the number of active queries + active: 0, + + // Last-Modified header cache for next request + lastModified: {}, + etag: {}, + + ajaxSettings: { + url: location.href, + type: "GET", + isLocal: rlocalProtocol.test( location.protocol ), + global: true, + processData: true, + async: true, + contentType: "application/x-www-form-urlencoded; charset=UTF-8", + + /* + timeout: 0, + data: null, + dataType: null, + username: null, + password: null, + cache: null, + throws: false, + traditional: false, + headers: {}, + */ + + accepts: { + "*": allTypes, + text: "text/plain", + html: "text/html", + xml: "application/xml, text/xml", + json: "application/json, text/javascript" + }, + + contents: { + xml: /\bxml\b/, + html: /\bhtml/, + json: /\bjson\b/ + }, + + responseFields: { + xml: "responseXML", + text: "responseText", + json: "responseJSON" + }, + + // Data converters + // Keys separate source (or catchall "*") and destination types with a single space + converters: { + + // Convert anything to text + "* text": String, + + // Text to html (true = no transformation) + "text html": true, + + // Evaluate text as a json expression + "text json": JSON.parse, + + // Parse text as xml + "text xml": jQuery.parseXML + }, + + // For options that shouldn't be deep extended: + // you can add your own custom options here if + // and when you create one that shouldn't be + // deep extended (see ajaxExtend) + flatOptions: { + url: true, + context: true + } + }, + + // Creates a full fledged settings object into target + // with both ajaxSettings and settings fields. + // If target is omitted, writes into ajaxSettings. + ajaxSetup: function( target, settings ) { + return settings ? + + // Building a settings object + ajaxExtend( ajaxExtend( target, jQuery.ajaxSettings ), settings ) : + + // Extending ajaxSettings + ajaxExtend( jQuery.ajaxSettings, target ); + }, + + ajaxPrefilter: addToPrefiltersOrTransports( prefilters ), + ajaxTransport: addToPrefiltersOrTransports( transports ), + + // Main method + ajax: function( url, options ) { + + // If url is an object, simulate pre-1.5 signature + if ( typeof url === "object" ) { + options = url; + url = undefined; + } + + // Force options to be an object + options = options || {}; + + var transport, + + // URL without anti-cache param + cacheURL, + + // Response headers + responseHeadersString, + responseHeaders, + + // timeout handle + timeoutTimer, + + // Url cleanup var + urlAnchor, + + // Request state (becomes false upon send and true upon completion) + completed, + + // To know if global events are to be dispatched + fireGlobals, + + // Loop variable + i, + + // uncached part of the url + uncached, + + // Create the final options object + s = jQuery.ajaxSetup( {}, options ), + + // Callbacks context + callbackContext = s.context || s, + + // Context for global events is callbackContext if it is a DOM node or jQuery collection + globalEventContext = s.context && + ( callbackContext.nodeType || callbackContext.jquery ) ? + jQuery( callbackContext ) : + jQuery.event, + + // Deferreds + deferred = jQuery.Deferred(), + completeDeferred = jQuery.Callbacks( "once memory" ), + + // Status-dependent callbacks + statusCode = s.statusCode || {}, + + // Headers (they are sent all at once) + requestHeaders = {}, + requestHeadersNames = {}, + + // Default abort message + strAbort = "canceled", + + // Fake xhr + jqXHR = { + readyState: 0, + + // Builds headers hashtable if needed + getResponseHeader: function( key ) { + var match; + if ( completed ) { + if ( !responseHeaders ) { + responseHeaders = {}; + while ( ( match = rheaders.exec( responseHeadersString ) ) ) { + responseHeaders[ match[ 1 ].toLowerCase() + " " ] = + ( responseHeaders[ match[ 1 ].toLowerCase() + " " ] || [] ) + .concat( match[ 2 ] ); + } + } + match = responseHeaders[ key.toLowerCase() + " " ]; + } + return match == null ? null : match.join( ", " ); + }, + + // Raw string + getAllResponseHeaders: function() { + return completed ? responseHeadersString : null; + }, + + // Caches the header + setRequestHeader: function( name, value ) { + if ( completed == null ) { + name = requestHeadersNames[ name.toLowerCase() ] = + requestHeadersNames[ name.toLowerCase() ] || name; + requestHeaders[ name ] = value; + } + return this; + }, + + // Overrides response content-type header + overrideMimeType: function( type ) { + if ( completed == null ) { + s.mimeType = type; + } + return this; + }, + + // Status-dependent callbacks + statusCode: function( map ) { + var code; + if ( map ) { + if ( completed ) { + + // Execute the appropriate callbacks + jqXHR.always( map[ jqXHR.status ] ); + } else { + + // Lazy-add the new callbacks in a way that preserves old ones + for ( code in map ) { + statusCode[ code ] = [ statusCode[ code ], map[ code ] ]; + } + } + } + return this; + }, + + // Cancel the request + abort: function( statusText ) { + var finalText = statusText || strAbort; + if ( transport ) { + transport.abort( finalText ); + } + done( 0, finalText ); + return this; + } + }; + + // Attach deferreds + deferred.promise( jqXHR ); + + // Add protocol if not provided (prefilters might expect it) + // Handle falsy url in the settings object (#10093: consistency with old signature) + // We also use the url parameter if available + s.url = ( ( url || s.url || location.href ) + "" ) + .replace( rprotocol, location.protocol + "//" ); + + // Alias method option to type as per ticket #12004 + s.type = options.method || options.type || s.method || s.type; + + // Extract dataTypes list + s.dataTypes = ( s.dataType || "*" ).toLowerCase().match( rnothtmlwhite ) || [ "" ]; + + // A cross-domain request is in order when the origin doesn't match the current origin. + if ( s.crossDomain == null ) { + urlAnchor = document.createElement( "a" ); + + // Support: IE <=8 - 11, Edge 12 - 15 + // IE throws exception on accessing the href property if url is malformed, + // e.g. http://example.com:80x/ + try { + urlAnchor.href = s.url; + + // Support: IE <=8 - 11 only + // Anchor's host property isn't correctly set when s.url is relative + urlAnchor.href = urlAnchor.href; + s.crossDomain = originAnchor.protocol + "//" + originAnchor.host !== + urlAnchor.protocol + "//" + urlAnchor.host; + } catch ( e ) { + + // If there is an error parsing the URL, assume it is crossDomain, + // it can be rejected by the transport if it is invalid + s.crossDomain = true; + } + } + + // Convert data if not already a string + if ( s.data && s.processData && typeof s.data !== "string" ) { + s.data = jQuery.param( s.data, s.traditional ); + } + + // Apply prefilters + inspectPrefiltersOrTransports( prefilters, s, options, jqXHR ); + + // If request was aborted inside a prefilter, stop there + if ( completed ) { + return jqXHR; + } + + // We can fire global events as of now if asked to + // Don't fire events if jQuery.event is undefined in an AMD-usage scenario (#15118) + fireGlobals = jQuery.event && s.global; + + // Watch for a new set of requests + if ( fireGlobals && jQuery.active++ === 0 ) { + jQuery.event.trigger( "ajaxStart" ); + } + + // Uppercase the type + s.type = s.type.toUpperCase(); + + // Determine if request has content + s.hasContent = !rnoContent.test( s.type ); + + // Save the URL in case we're toying with the If-Modified-Since + // and/or If-None-Match header later on + // Remove hash to simplify url manipulation + cacheURL = s.url.replace( rhash, "" ); + + // More options handling for requests with no content + if ( !s.hasContent ) { + + // Remember the hash so we can put it back + uncached = s.url.slice( cacheURL.length ); + + // If data is available and should be processed, append data to url + if ( s.data && ( s.processData || typeof s.data === "string" ) ) { + cacheURL += ( rquery.test( cacheURL ) ? "&" : "?" ) + s.data; + + // #9682: remove data so that it's not used in an eventual retry + delete s.data; + } + + // Add or update anti-cache param if needed + if ( s.cache === false ) { + cacheURL = cacheURL.replace( rantiCache, "$1" ); + uncached = ( rquery.test( cacheURL ) ? "&" : "?" ) + "_=" + ( nonce.guid++ ) + + uncached; + } + + // Put hash and anti-cache on the URL that will be requested (gh-1732) + s.url = cacheURL + uncached; + + // Change '%20' to '+' if this is encoded form body content (gh-2658) + } else if ( s.data && s.processData && + ( s.contentType || "" ).indexOf( "application/x-www-form-urlencoded" ) === 0 ) { + s.data = s.data.replace( r20, "+" ); + } + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + if ( jQuery.lastModified[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-Modified-Since", jQuery.lastModified[ cacheURL ] ); + } + if ( jQuery.etag[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-None-Match", jQuery.etag[ cacheURL ] ); + } + } + + // Set the correct header, if data is being sent + if ( s.data && s.hasContent && s.contentType !== false || options.contentType ) { + jqXHR.setRequestHeader( "Content-Type", s.contentType ); + } + + // Set the Accepts header for the server, depending on the dataType + jqXHR.setRequestHeader( + "Accept", + s.dataTypes[ 0 ] && s.accepts[ s.dataTypes[ 0 ] ] ? + s.accepts[ s.dataTypes[ 0 ] ] + + ( s.dataTypes[ 0 ] !== "*" ? ", " + allTypes + "; q=0.01" : "" ) : + s.accepts[ "*" ] + ); + + // Check for headers option + for ( i in s.headers ) { + jqXHR.setRequestHeader( i, s.headers[ i ] ); + } + + // Allow custom headers/mimetypes and early abort + if ( s.beforeSend && + ( s.beforeSend.call( callbackContext, jqXHR, s ) === false || completed ) ) { + + // Abort if not done already and return + return jqXHR.abort(); + } + + // Aborting is no longer a cancellation + strAbort = "abort"; + + // Install callbacks on deferreds + completeDeferred.add( s.complete ); + jqXHR.done( s.success ); + jqXHR.fail( s.error ); + + // Get transport + transport = inspectPrefiltersOrTransports( transports, s, options, jqXHR ); + + // If no transport, we auto-abort + if ( !transport ) { + done( -1, "No Transport" ); + } else { + jqXHR.readyState = 1; + + // Send global event + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxSend", [ jqXHR, s ] ); + } + + // If request was aborted inside ajaxSend, stop there + if ( completed ) { + return jqXHR; + } + + // Timeout + if ( s.async && s.timeout > 0 ) { + timeoutTimer = window.setTimeout( function() { + jqXHR.abort( "timeout" ); + }, s.timeout ); + } + + try { + completed = false; + transport.send( requestHeaders, done ); + } catch ( e ) { + + // Rethrow post-completion exceptions + if ( completed ) { + throw e; + } + + // Propagate others as results + done( -1, e ); + } + } + + // Callback for when everything is done + function done( status, nativeStatusText, responses, headers ) { + var isSuccess, success, error, response, modified, + statusText = nativeStatusText; + + // Ignore repeat invocations + if ( completed ) { + return; + } + + completed = true; + + // Clear timeout if it exists + if ( timeoutTimer ) { + window.clearTimeout( timeoutTimer ); + } + + // Dereference transport for early garbage collection + // (no matter how long the jqXHR object will be used) + transport = undefined; + + // Cache response headers + responseHeadersString = headers || ""; + + // Set readyState + jqXHR.readyState = status > 0 ? 4 : 0; + + // Determine if successful + isSuccess = status >= 200 && status < 300 || status === 304; + + // Get response data + if ( responses ) { + response = ajaxHandleResponses( s, jqXHR, responses ); + } + + // Use a noop converter for missing script but not if jsonp + if ( !isSuccess && + jQuery.inArray( "script", s.dataTypes ) > -1 && + jQuery.inArray( "json", s.dataTypes ) < 0 ) { + s.converters[ "text script" ] = function() {}; + } + + // Convert no matter what (that way responseXXX fields are always set) + response = ajaxConvert( s, response, jqXHR, isSuccess ); + + // If successful, handle type chaining + if ( isSuccess ) { + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + modified = jqXHR.getResponseHeader( "Last-Modified" ); + if ( modified ) { + jQuery.lastModified[ cacheURL ] = modified; + } + modified = jqXHR.getResponseHeader( "etag" ); + if ( modified ) { + jQuery.etag[ cacheURL ] = modified; + } + } + + // if no content + if ( status === 204 || s.type === "HEAD" ) { + statusText = "nocontent"; + + // if not modified + } else if ( status === 304 ) { + statusText = "notmodified"; + + // If we have data, let's convert it + } else { + statusText = response.state; + success = response.data; + error = response.error; + isSuccess = !error; + } + } else { + + // Extract error from statusText and normalize for non-aborts + error = statusText; + if ( status || !statusText ) { + statusText = "error"; + if ( status < 0 ) { + status = 0; + } + } + } + + // Set data for the fake xhr object + jqXHR.status = status; + jqXHR.statusText = ( nativeStatusText || statusText ) + ""; + + // Success/Error + if ( isSuccess ) { + deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] ); + } else { + deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] ); + } + + // Status-dependent callbacks + jqXHR.statusCode( statusCode ); + statusCode = undefined; + + if ( fireGlobals ) { + globalEventContext.trigger( isSuccess ? "ajaxSuccess" : "ajaxError", + [ jqXHR, s, isSuccess ? success : error ] ); + } + + // Complete + completeDeferred.fireWith( callbackContext, [ jqXHR, statusText ] ); + + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxComplete", [ jqXHR, s ] ); + + // Handle the global AJAX counter + if ( !( --jQuery.active ) ) { + jQuery.event.trigger( "ajaxStop" ); + } + } + } + + return jqXHR; + }, + + getJSON: function( url, data, callback ) { + return jQuery.get( url, data, callback, "json" ); + }, + + getScript: function( url, callback ) { + return jQuery.get( url, undefined, callback, "script" ); + } +} ); + +jQuery.each( [ "get", "post" ], function( _i, method ) { + jQuery[ method ] = function( url, data, callback, type ) { + + // Shift arguments if data argument was omitted + if ( isFunction( data ) ) { + type = type || callback; + callback = data; + data = undefined; + } + + // The url can be an options object (which then must have .url) + return jQuery.ajax( jQuery.extend( { + url: url, + type: method, + dataType: type, + data: data, + success: callback + }, jQuery.isPlainObject( url ) && url ) ); + }; +} ); + +jQuery.ajaxPrefilter( function( s ) { + var i; + for ( i in s.headers ) { + if ( i.toLowerCase() === "content-type" ) { + s.contentType = s.headers[ i ] || ""; + } + } +} ); + + +jQuery._evalUrl = function( url, options, doc ) { + return jQuery.ajax( { + url: url, + + // Make this explicit, since user can override this through ajaxSetup (#11264) + type: "GET", + dataType: "script", + cache: true, + async: false, + global: false, + + // Only evaluate the response if it is successful (gh-4126) + // dataFilter is not invoked for failure responses, so using it instead + // of the default converter is kludgy but it works. + converters: { + "text script": function() {} + }, + dataFilter: function( response ) { + jQuery.globalEval( response, options, doc ); + } + } ); +}; + + +jQuery.fn.extend( { + wrapAll: function( html ) { + var wrap; + + if ( this[ 0 ] ) { + if ( isFunction( html ) ) { + html = html.call( this[ 0 ] ); + } + + // The elements to wrap the target around + wrap = jQuery( html, this[ 0 ].ownerDocument ).eq( 0 ).clone( true ); + + if ( this[ 0 ].parentNode ) { + wrap.insertBefore( this[ 0 ] ); + } + + wrap.map( function() { + var elem = this; + + while ( elem.firstElementChild ) { + elem = elem.firstElementChild; + } + + return elem; + } ).append( this ); + } + + return this; + }, + + wrapInner: function( html ) { + if ( isFunction( html ) ) { + return this.each( function( i ) { + jQuery( this ).wrapInner( html.call( this, i ) ); + } ); + } + + return this.each( function() { + var self = jQuery( this ), + contents = self.contents(); + + if ( contents.length ) { + contents.wrapAll( html ); + + } else { + self.append( html ); + } + } ); + }, + + wrap: function( html ) { + var htmlIsFunction = isFunction( html ); + + return this.each( function( i ) { + jQuery( this ).wrapAll( htmlIsFunction ? html.call( this, i ) : html ); + } ); + }, + + unwrap: function( selector ) { + this.parent( selector ).not( "body" ).each( function() { + jQuery( this ).replaceWith( this.childNodes ); + } ); + return this; + } +} ); + + +jQuery.expr.pseudos.hidden = function( elem ) { + return !jQuery.expr.pseudos.visible( elem ); +}; +jQuery.expr.pseudos.visible = function( elem ) { + return !!( elem.offsetWidth || elem.offsetHeight || elem.getClientRects().length ); +}; + + + + +jQuery.ajaxSettings.xhr = function() { + try { + return new window.XMLHttpRequest(); + } catch ( e ) {} +}; + +var xhrSuccessStatus = { + + // File protocol always yields status code 0, assume 200 + 0: 200, + + // Support: IE <=9 only + // #1450: sometimes IE returns 1223 when it should be 204 + 1223: 204 + }, + xhrSupported = jQuery.ajaxSettings.xhr(); + +support.cors = !!xhrSupported && ( "withCredentials" in xhrSupported ); +support.ajax = xhrSupported = !!xhrSupported; + +jQuery.ajaxTransport( function( options ) { + var callback, errorCallback; + + // Cross domain only allowed if supported through XMLHttpRequest + if ( support.cors || xhrSupported && !options.crossDomain ) { + return { + send: function( headers, complete ) { + var i, + xhr = options.xhr(); + + xhr.open( + options.type, + options.url, + options.async, + options.username, + options.password + ); + + // Apply custom fields if provided + if ( options.xhrFields ) { + for ( i in options.xhrFields ) { + xhr[ i ] = options.xhrFields[ i ]; + } + } + + // Override mime type if needed + if ( options.mimeType && xhr.overrideMimeType ) { + xhr.overrideMimeType( options.mimeType ); + } + + // X-Requested-With header + // For cross-domain requests, seeing as conditions for a preflight are + // akin to a jigsaw puzzle, we simply never set it to be sure. + // (it can always be set on a per-request basis or even using ajaxSetup) + // For same-domain requests, won't change header if already provided. + if ( !options.crossDomain && !headers[ "X-Requested-With" ] ) { + headers[ "X-Requested-With" ] = "XMLHttpRequest"; + } + + // Set headers + for ( i in headers ) { + xhr.setRequestHeader( i, headers[ i ] ); + } + + // Callback + callback = function( type ) { + return function() { + if ( callback ) { + callback = errorCallback = xhr.onload = + xhr.onerror = xhr.onabort = xhr.ontimeout = + xhr.onreadystatechange = null; + + if ( type === "abort" ) { + xhr.abort(); + } else if ( type === "error" ) { + + // Support: IE <=9 only + // On a manual native abort, IE9 throws + // errors on any property access that is not readyState + if ( typeof xhr.status !== "number" ) { + complete( 0, "error" ); + } else { + complete( + + // File: protocol always yields status 0; see #8605, #14207 + xhr.status, + xhr.statusText + ); + } + } else { + complete( + xhrSuccessStatus[ xhr.status ] || xhr.status, + xhr.statusText, + + // Support: IE <=9 only + // IE9 has no XHR2 but throws on binary (trac-11426) + // For XHR2 non-text, let the caller handle it (gh-2498) + ( xhr.responseType || "text" ) !== "text" || + typeof xhr.responseText !== "string" ? + { binary: xhr.response } : + { text: xhr.responseText }, + xhr.getAllResponseHeaders() + ); + } + } + }; + }; + + // Listen to events + xhr.onload = callback(); + errorCallback = xhr.onerror = xhr.ontimeout = callback( "error" ); + + // Support: IE 9 only + // Use onreadystatechange to replace onabort + // to handle uncaught aborts + if ( xhr.onabort !== undefined ) { + xhr.onabort = errorCallback; + } else { + xhr.onreadystatechange = function() { + + // Check readyState before timeout as it changes + if ( xhr.readyState === 4 ) { + + // Allow onerror to be called first, + // but that will not handle a native abort + // Also, save errorCallback to a variable + // as xhr.onerror cannot be accessed + window.setTimeout( function() { + if ( callback ) { + errorCallback(); + } + } ); + } + }; + } + + // Create the abort callback + callback = callback( "abort" ); + + try { + + // Do send the request (this may raise an exception) + xhr.send( options.hasContent && options.data || null ); + } catch ( e ) { + + // #14683: Only rethrow if this hasn't been notified as an error yet + if ( callback ) { + throw e; + } + } + }, + + abort: function() { + if ( callback ) { + callback(); + } + } + }; + } +} ); + + + + +// Prevent auto-execution of scripts when no explicit dataType was provided (See gh-2432) +jQuery.ajaxPrefilter( function( s ) { + if ( s.crossDomain ) { + s.contents.script = false; + } +} ); + +// Install script dataType +jQuery.ajaxSetup( { + accepts: { + script: "text/javascript, application/javascript, " + + "application/ecmascript, application/x-ecmascript" + }, + contents: { + script: /\b(?:java|ecma)script\b/ + }, + converters: { + "text script": function( text ) { + jQuery.globalEval( text ); + return text; + } + } +} ); + +// Handle cache's special case and crossDomain +jQuery.ajaxPrefilter( "script", function( s ) { + if ( s.cache === undefined ) { + s.cache = false; + } + if ( s.crossDomain ) { + s.type = "GET"; + } +} ); + +// Bind script tag hack transport +jQuery.ajaxTransport( "script", function( s ) { + + // This transport only deals with cross domain or forced-by-attrs requests + if ( s.crossDomain || s.scriptAttrs ) { + var script, callback; + return { + send: function( _, complete ) { + script = jQuery( " + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Accounts

+

The accounts section is for the management of user accounts, access permissions, and tenants.

+
+

Users

+

Description of User account fields:

+
    +
  • Username - Unique username.

  • +
  • Full Name - Full Name of the user.

  • +
  • E-mail - The email address of the user. Also used for system notifications and for password retrieval.

  • +
  • E-mail CC - Send copies of email notifications to this address.

  • +
  • Phone Number - User’s phone number.

  • +
  • Company - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers.

  • +
  • Position - Position within the company.

  • +
  • User Role - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate.

  • +
  • Permission Group - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used)

  • +
  • +Tenant - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used)

  • +
+

Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin.

+User Management +

Password: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side.

+

Example: Listing of user accounts.

+List User Accounts +
+
+

Tenants

+

IP addresses and Switch Ports are network resources that can be assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs) as part of DevOps/NetOps pipeline.

+

A Tenant has just two fields, the unique name and custom description.

+

Example: Adding a tenant.

+Adding Tenants +
+
+

Permission Groups

+

Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections.

+

Example: Permission Group.

+Managing Permissions +
+
+

User Roles

+

Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username.

+User Roles +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/acls.html b/en/3.0/acls.html new file mode 100644 index 0000000000..d04a30a484 --- /dev/null +++ b/en/3.0/acls.html @@ -0,0 +1,622 @@ + + + + + + + + + + + Access Control Lists (ACL) — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Access Control Lists (ACL)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Access Control Lists (ACL)

+

Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access.

+

Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches.

+

Screenshot: TCAM utilization can be seen under Net→Inventory

+_images/TCAM.png +

Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements.

+
+

ACL Default Policy

+

The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules.

+

Example: Changing “ACL Default Policy” for the site “siteDefault”.

+_images/siteDefault.png +
+
+

ACL Rules

+

ACL rules can be created, listed, edited, approved under Services→ACL.

+

Description of ACL fields. +General

+
    +
  • Name - Unique name for the ACL entry.

  • +
  • Protocol - IP protocol to match.

    +
      +
    • All - Any IP protocols.

    • +
    • IP - Specific IP protocol number.

    • +
    • TCP - TCP.

    • +
    • UDP - UDP.

    • +
    • ICMP ALL - Any IPv4 ICMP protocol.

    • +
    • ICMP Custom - Custom IPv4 ICMP code.

    • +
    • ICMPv6 ALL - Any IPv6 ICMP protocol.

    • +
    • ICMPv6 Custom - Custom IPv6 ICMP code.

    • +
    +
  • +
  • Active Until - Disable this rule at the defined date/time.

  • +
  • Action - Permit or Deny forwarding of matched packets.

  • +
  • Established/Reverse - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination.

  • +
+

Source/Destination - Source and destination addresses and ports to match.

+
    +
  • Source IPv4/IPv6 - IPv4/IPv6 address.

  • +
  • Ports Type

    +
      +
    • Port Range - Match on the port or a port range defined in this window.

    • +
    • Port Group - Match on a group of ports defined under Services→ ACL Port Group.

    • +
    +
  • +
  • From Port - Port range starting from.

  • +
  • To Port - Port range ending with.

  • +
  • Comment - Descriptive comment, commonly used for approval workflows.

  • +
  • Check button - Check if Another ACL on the system already permits the described network access.

  • +
+

Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established).

+_images/action_permit.png +

Example: “Check” shows that requested access is already provided by a broader ACL rule.

+_images/ACL_rule.png +
+
+

ACL Approval Workflow

+

When one tenant (one team) needs to get network access to resources under the responsibility of another tenant (another team), an ACL can be created but will activate only after approval of the tenant responsible for the destination address resources. See the below example.

+

Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant.

+_images/ACL_approval.png +

Screenshot: ACL stays in “waiting for approval” state until approved.

+_images/waiting_approval.png +

Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it.

+_images/approve_reject.png +

Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric.

+_images/ACL_active.png +
+
+

ACL Processing Order

+
    +
  1. User-defined Deny Rules

  2. +
  3. User-defined Permit Rules

  4. +
  5. Deny the rest

  6. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/controller-k3s-installation.html b/en/3.0/controller-k3s-installation.html new file mode 100644 index 0000000000..4db111a2bc --- /dev/null +++ b/en/3.0/controller-k3s-installation.html @@ -0,0 +1,679 @@ + + + + + + + + + + + Netris Controller installation on a generic Linux host — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Controller installation on a generic Linux host

+
+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+
+

Note

+

K3s is expected to work on most modern Linux systems.

+

Some OSs have specific requirements:

+
    +
  • If you are using Raspbian Buster, follow these steps to switch to legacy iptables.

  • +
  • If you are using Alpine Linux, follow these steps for additional setup.

  • +
  • If you are using (Red Hat/CentOS) Enterprise Linux, follow these steps for additional setup.

  • +
+
+
+
+

Installation

+

The following command will install the Netris Controller on your Linux server:

+
curl -sfL https://get.netris.ai | sh -
+
+
+

Once installed, you will be able to log in to Netris Controller using your host’s IP address.

+
+

Note

+

The installation script does the following:

+ +
+
+

Installation with the specific host name

+

In order to set the specific ingress host name to the Netris Controller, use the --ctl-hostname installation argument:

+
curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com
+
+
+

A self-signed SSL certificate will be generated from that host name.

+
+
+

Installation with the Let’s Encrypt SSL

+

The installation script supports Let’s Encrypt SSL generation out-of-box. To instruct the installation script to do that use --ctl-ssl-issuer argument.

+
+

Note

+
+
The argument --ctl-ssl-issuer is passing cert-manager.io/cluster-issuer value to the ingress resource of the Netris Controller. The installation script can create two types of ClusterIssuer resource: selfsigned or letsencrypt, where selfsigned is just Cert-Manager self-signed SSL and the letsencrypt is the ACME issuer with HTTP01 challenge validation.
+
If the --ctl-ssl-issuer argument is not set, the installation script will proceed with selfsigned ClusterIssuer type.
+
+
+

Run the following command to install Netris Controller and use letsencrypt ClusterIssuer for SSL generation:

+
curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

To successfully validate and complete Let’s Encrypt SSL generation, a valid A/CNAME record for the domain/subdomain name should exist prior, and that name must be accessible from the Internet.

+
+
+
+

Installation with the Custom SSL Issuer

+

The HTTP01 challenge validation is the simplest way of issuing the Let’s Encrypt SSL, but it does not work when the host behind the FQDN is not accessible from the public internet. +The common approach of validating and completing Let’s Encrypt SSL generation for private deployments is DNS01 challenge validation. +If the DNS01 does not work for you either, Cert-Manager supports a number of certificate issuers, get familiar with all types of issuers here.

+

In order to install Netris Controller with the custom SSL issuer, you need to run installation script with the specified host name:

+
curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com
+
+
+

Once the installation is complete, create a yaml file with the ClusterIssuer resource, suitable for your requirements, and apply it:

+
kubectl apply -f my-cluster-issuer.yaml
+
+
+

Then rerun the installation script with the --ctl-ssl-issuer argument:

+
curl -sfL https://get.netris.ai | sh -s -- --ctl-ssl-issuer <Your ClusterIssuer resource name>
+
+
+
+
+
+

Upgrading

+

To upgrade the Netris Controller to the latest version simply run the script:

+
curl -sfL https://get.netris.ai | sh -
+
+
+

If a newer version of Netris Controller is available, it will be updated in a few minutes.

+
+
+

Uninstalling

+

To uninstall Netris Controller and K3s from a server node, run:

+
/usr/local/bin/k3s-uninstall.sh
+
+
+
+
+

Backup and Restore

+

Netris Controller stores all critical data in MariaDB. It’s highly recommended to create a cronjob with mysqldump.

+
+

Backup

+

To take database snapshot run the following command:

+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot-$(date +%Y-%m-%d-%H-%M-%S).sql
+
+
+

After command execution, you can find db-snapshot-YYYY-MM-DD-HH-MM-SS.sql file in the current working directory.

+
+
+

Restore

+

In order to restore DB from a database snapshot, follow these steps:

+
    +
  1. Copy snapshot file to the MariaDB container:

  2. +
+
kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql
+
+
+
    +
  1. Run the restore command:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql'
+
+
+
+

Note

+

In this example the snapshot file name is db-snapshot.sql and it’s located in the current working directory

+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/controller-k8s-installation.html b/en/3.0/controller-k8s-installation.html new file mode 100644 index 0000000000..9ccd3a96d4 --- /dev/null +++ b/en/3.0/controller-k8s-installation.html @@ -0,0 +1,596 @@ + + + + + + + + + + + Helm Chart Installation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Helm Chart Installation

+
+

Requirements

+
    +
  • Kubernetes 1.12+

  • +
  • Helm 3.1+

  • +
  • PV provisioner support in the underlying infrastructure

  • +
+
+
+

Get Repo Info

+

Add the Netris Helm repository:

+
helm repo add netrisai https://netrisai.github.io/charts
+helm repo update
+
+
+
+
+

Installing the Chart

+

In order to install the Helm chart, you must follow these steps:

+
    +
  1. Create the namespace for netris-controller:

  2. +
+
kubectl create namespace netris-controller
+
+
+
    +
  1. Install helm chart with netris-controller:

  2. +
+
helm install netris-controller netrisai/netris-controller \
+  --namespace netris-controller \
+  --set app.ingress.hosts={my.domain.com}
+
+
+
+
+

Uninstalling the Chart

+

To uninstall/delete the netris-controller helm release:

+
helm uninstall netris-controller
+
+
+
+
+

Chart Configuration

+

See the netris-controller README for details about configurable parameters and their default values.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/controller-k8s-quickstart.html b/en/3.0/controller-k8s-quickstart.html new file mode 100644 index 0000000000..b56317fa95 --- /dev/null +++ b/en/3.0/controller-k8s-quickstart.html @@ -0,0 +1,564 @@ + + + + + + + + + + + Quickstart Installation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Quickstart Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Quickstart Installation

+

Netris offers a simplified deployment model for users who want to quickly install the Netris Controller in the shortest amount of time.

+

This installation process is streamlined for Linux servers that do not already have Kubernetes running. The install does the following:

+ +

If you wish to install the controller on an existing Kubernetes cluster, follow these instructions instead of this Quickstart.

+
+

Quickstart Process

+
    +
  1. Install the Netris Controller w/ k3s by running the following command on your Linux server:

  2. +
+
curl -sfL https://get.netris.ai | sh -
+
+
+
    +
  1. When the installation completes, you will be provided with the login for the web UI. Login with the provided credentials.

  2. +
  3. Navigate in the UI to Net→IPAM and add a new subnet that contains the desired management IP addresses you wish to use for your SoftGates and switches.

    +

    For example, if you are planning on using 192.168.1.100 as the IP address of your Ubuntu server, then create a subnet in Netris UI for 192.168.1.0/24.

    +

    Detailed configuration documentation is available here: Netris IPAM.

    +
  4. +
  5. Navigate in the UI to Topology

  6. +
  7. Click the Add in the upper right

  8. +
  9. Fill out the fields for the SoftGate you wish to add

  10. +
  11. Select the proper Management IP address from the subnet selector

  12. +
  13. Once the SoftGate is created in the Topology, right-click on the SoftGate and select the Install Agent option

  14. +
  15. Copy the agent install command to your clipboard and run the command on the Ubuntu server you are using as your SoftGate

  16. +
  17. Congratulations. The SoftGate should now be connected to your controller.

  18. +
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/controller-vm-installation.html b/en/3.0/controller-vm-installation.html new file mode 100644 index 0000000000..c12feb9456 --- /dev/null +++ b/en/3.0/controller-vm-installation.html @@ -0,0 +1,699 @@ + + + + + + + + + + + Virtual Machine Installation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Virtual Machine Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Virtual Machine Installation

+
+

Requirements

+

Minimal system requirements for the VM:

+
    +
  • CPU - 4 Core

  • +
  • RAM - 4 Gb

  • +
  • Disk - 100Gb

  • +
  • Network - 1 virtual NIC

  • +
+

Recommended system requirements for the VM:

+
    +
  • CPU - 8 Core

  • +
  • RAM - 16 Gb

  • +
  • Disk - 100Gb

  • +
  • Network - 1 virtual NIC

  • +
+
+
+

KVM Hypervisor Installation

+

If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04)

+
sudo apt-get install virt-manager
+
+
+
+
+

VM Controller Installation

+
    +
  1. Download the Netris Controller image. (contact Netris support for repository access permissions).

  2. +
+
cd /var/lib/libvirt/images
+
+sudo wget http://img.netris.ai/netris-controller3.qcow2
+
+
+
    +
  1. Download VM definition file.

  2. +
+
cd /etc/libvirt/qemu
+
+sudo wget http://img.netris.ai/netris-controller3.xml
+
+
+
    +
  1. Define the KVM virtual machine

  2. +
+
sudo virsh define netris-controller3.xml
+
+
+
+

Note

+

Netris Controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below for the network interface configuration example.

+
+

Example: Network configuration on host (hypervisor) machine.

+
+

Note

+

replace <Physical NIC>, <host server management IP/prefix length> and <host server default gateway> +with the correct NIC and IP for your host machine.

+
+
sudo vim /etc/network/interfaces
+
+
+
#Physical NIC connected to the management network
+auto <Physical NIC>
+iface <Physical NIC> inet static
+                        address 0.0.0.0/0
+
+#bridge interface
+auto br-mgmt
+iface br-mgmt inet static
+                        address <host server management IP/prefix length>
+                        gateway <host server default gateway>
+                        bridge-ports <Physical NIC>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Set the virtual machine to autostart and start it.

  2. +
+
sudo virsh autostart netris-controller
+
+
+
sudo virsh start netris-controller
+
+
+
+
+

Accessing the Netris Controller

+

By default, Netris Controller will obtain an IP address from a DHCP server.

+

Below steps describe how to configure a Static IP address for the Netris Controller.

+
    +
  1. Connecting to the VM console.

  2. +
+

default credentials. login: netris password: newNet0ps

+
sudo virsh console netris-controller
+
+
+
+

Note

+

Do not forget to change the default password (using passwd command).

+
+
    +
  1. Setting a static IP address.

  2. +
+

Edit network configuration file.

+
sudo vim /etc/network/interfaces
+
+
+

Example: IP configuration file.

+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <Netris Controller IP/prefix length>
+        gateway <Netris Controller default gateway>
+        dns-nameserver <a DNS server address>
+
+source /etc/network/interfaces.d/*
+
+
+

Reload the network config.

+
sudo ifreload -a
+
+
+
+

Note

+

Make sure Netris Controller has Internet access.

+
+
    +
  1. Reboot the controller

  2. +
+
sudo reboot
+
+
+

After reboot, the Netris Controller GUI should be accessible using a browser. Use netris/newNet0ps credentials.

+Netris Credentials +
+
+

Replacing the SSL certificate

+
    +
  1. Replace the below file with your SSL certificate file.

  2. +
+
/etc/nginx/ssl/controller.cert.pem;
+
+
+
    +
  1. Replace the below file with your SSL private key.

  2. +
+
/etc/nginx/ssl/controller.key.pem;
+
+
+
    +
  1. Restart Nginx service.

  2. +
+
systemctl restart nginx.service
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/definitions.html b/en/3.0/definitions.html new file mode 100644 index 0000000000..ceaf10ab1c --- /dev/null +++ b/en/3.0/definitions.html @@ -0,0 +1,561 @@ + + + + + + + + + + + Definitions — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Definitions

+

When configuring and operating a Netris system, the following nomenclature is important to understand:

+
    +
  • User - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is netris, with password newNet0ps.

  • +
  • Tenant - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request and manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs) via a DevOps/NetOps pipeline.

  • +
  • Permission Group - List of permissions on a per section basis can be attached individually to a User or a User Role

  • +
  • User Role - Group of user permissions and tenants for role-based access control (RBAC)

  • +
  • Site - Each separate deployment (each data center) should be defined as a Site. All network units and resources are attached to a site. Netris Controller comes with a “default” site preconfigured. Site entry defines global attributes such as; AS numbers, default ACL policy, and Site Mesh (site to site VPN) type.

  • +
  • Subnet - IPv4/IPv6 address resources linked to Sites and Tenants

  • +
  • Switch Port - Physical ports of all switches attached to the system

  • +
  • Inventory - Inventory of all network units that are operated using Netris Agent

  • +
  • E-BGP - Defines all External BGP peers (iBGP and eBGP)

  • +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/genindex.html b/en/3.0/genindex.html new file mode 100644 index 0000000000..fe859d34e6 --- /dev/null +++ b/en/3.0/genindex.html @@ -0,0 +1,536 @@ + + + + + + + + + + Index — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ + +

Index

+ +
+ +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/index.html b/en/3.0/index.html new file mode 100644 index 0000000000..90b50e670c --- /dev/null +++ b/en/3.0/index.html @@ -0,0 +1,753 @@ + + + + + + + + + + Welcome to Netris Documentation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Documentation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Welcome to Netris Documentation

+

Netris is the Automatic NetOps platform that runs the physical network and provides cloud-like user experience for NetOps and DevOps engineers.

+ + + + + + + + + +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/installation.html b/en/3.0/installation.html new file mode 100644 index 0000000000..7e76155675 --- /dev/null +++ b/en/3.0/installation.html @@ -0,0 +1,575 @@ + + + + + + + + + + + Controller Installation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Controller Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Controller Installation

+

Netris Controller can be installed locally as a VM, deployed as a Kubernetes application, or hosted in the Netris Cloud. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime.

+
+

Note

+

Select ONE installation option below.

+
+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/installing-netris-controller.html b/en/3.0/installing-netris-controller.html new file mode 100644 index 0000000000..2a75ae9958 --- /dev/null +++ b/en/3.0/installing-netris-controller.html @@ -0,0 +1,554 @@ + + + + + + + + + + + Installing a Netris Controller — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Installing a Netris Controller
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Installing a Netris Controller

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. So you can access the console, and nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface.

+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

In this example I am running my Netris controller on an AWS hosted virtual machine (EC2) which has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

I’m using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my EC2 : 54.219.211.71.

+_images/cloudflare-dns-record.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+

To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-hostname ” will instruct the installer to generate a Let’s Encrypt SSL certificate for the provided domain name. That’s why it is important to create the DNS record before this step.

+
curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com
+
+
+

Once installation process is finished you will be able to access your newly installed Netris Controller web consonle using netris/newNet0ps credentials.

+

Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller.

+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/introduction.html b/en/3.0/introduction.html new file mode 100644 index 0000000000..f0c66c239d --- /dev/null +++ b/en/3.0/introduction.html @@ -0,0 +1,551 @@ + + + + + + + + + + + Introduction to Netris — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Introduction to Netris
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Introduction to Netris

+

Netris is an automatic netops software for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network’s health and either applies software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation.

+_images/netris-architecture.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/inventory-profiles.html b/en/3.0/inventory-profiles.html new file mode 100644 index 0000000000..8fc692072b --- /dev/null +++ b/en/3.0/inventory-profiles.html @@ -0,0 +1,574 @@ + + + + + + + + + + + Inventory Profiles — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +
+

Inventory Profiles

+

Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/softgate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except netris-defined and user-defined custom flows. Generated rules include:

+
    +
  • SSH from user defined subnets

  • +
  • NTP from user defined ntp services

  • +
  • DNS from user defined DNS servers

  • +
  • Custom user defined rules

  • +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + +
Inventory Profile Fields

Name

Profile name

Description

Free text description

Allow SSH from IPv4

List of IPv4 subnets allowed to ssh (one address per line)

Allow SSH from IPv6

List of IPv6 subnets allowed to ssh (one address per line)

Timezone

Devices using this inventory profile will adjust their system time to the selected timezone.

NTP servers

List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate.

DNS servers

List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate.

+

Example: In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup).

+Adding an inventory profile +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/ipam.html b/en/3.0/ipam.html new file mode 100644 index 0000000000..9df54b94fd --- /dev/null +++ b/en/3.0/ipam.html @@ -0,0 +1,628 @@ + + + + + + + + + + + IP Address Management — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • IP Address Management
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

IP Address Management

+

Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting.

+

Purpose: +Users define specific roles(purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-net, NAT, etc…

+
+

Allocations and Subnets

+

There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets.

+
+
+
+

Add an Allocation

+
    +
  1. Navigate to Net→IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Allocation from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + +
Allocation Fields

Name

Unique name for current allocation.

Prefix

Unique prefix for allocation, must not overlap with other allocations.

Tenant

Owner of the allocation.

+
+
+
+

Add a Subnet

+
    +
  1. Navigate to Net→IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Subnet from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + + + + +
Subnet fields

Name

Unique name for current subnet.

Prefix

Unique prefix for subnet, ust be included in one of allocations.

Tenant

Owner of the subnet.

Purpose

This field describes for what kind of services the current subnet can be used. It can have the following values:

+
+
    +
  • common - ordinary subnet, can be used in v-nets and ROH.

  • +
  • loopback - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates).

  • +
  • management - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates).

  • +
  • load-balancer - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience.

  • +
  • nat - hosts of this subnet or subnet itself can be used to define NAT services.

  • +
  • inactive - can’t be used in any services, useful for reserving/documenting prefixes for future use.

  • +
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/kubernetes-integration.html b/en/3.0/kubernetes-integration.html new file mode 100644 index 0000000000..6056763ae4 --- /dev/null +++ b/en/3.0/kubernetes-integration.html @@ -0,0 +1,915 @@ + + + + + + + + + + + Kubernetes Integration — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Kubernetes Integration
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Kubernetes Integration

+

Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris-Kubernetes integration is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters.

+
+

Install Netris Operator

+

Integration between the Netris Controller and the Kubernetes API is completed by installing the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart:

+ +
+

Regular Manifest Method

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='http://**your-netris-controller-ip-or-host**' \
+--from-literal=login='**your-netris-admin-username**' --from-literal=password='**your-netris-admin-password**'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+
+

Using Type ‘LoadBalancer’

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox pod provisioning +

After provisioning has finished, inspect the service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.202   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9898   active   50.117.59.202   9898/TCP   US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9898   active   50.117.59.202   9898/TCP   US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.203   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.203
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+View L4 LB instances +
+
+

V-Net Custom Resource

+

You can also create Netris V-Nets (L2 segments) via Kubernetes with a simple manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+
+
+

BGP Custom Resource

+

You can create BGP peers via Kubernetes manifests:

+
    +
  1. Create a yaml file:

  2. +
+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1092
+  localIP: 50.117.59.118/30
+  remoteIP: 50.117.59.117/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.192/28 le 32
+EOF
+
+
+
    +
  1. Apply the manifest file:

  2. +
+
kubectl apply -f isp2-customer.yaml
+
+
+
    +
  1. Check created BGP:

  2. +
+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.118/30   50.117.59.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.118/30   50.117.59.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Calico CNI Integration

+

Netris Operator can integrate with Calico CNI. This annotation will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.118/30   50.117.59.117/30    7m59s
+sandbox9-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox9-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox9-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.118/30   50.117.59.117/30    8m41s
+sandbox9-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox9-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox9-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.202
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/l3-load-balancer.html b/en/3.0/l3-load-balancer.html new file mode 100644 index 0000000000..9836b4212d --- /dev/null +++ b/en/3.0/l3-load-balancer.html @@ -0,0 +1,562 @@ + + + + + + + + + + + L3 Load Balancer (Anycast LB) — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • L3 Load Balancer (Anycast LB)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

L3 Load Balancer (Anycast LB)

+

L3 (Anycast) load balancer leverages ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks.

+

ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server.

+

End-user traffic should be destined to the anycast IP address. The switch fabric uses ECMP to load balance the traffic towards every server, and will hash sessions based on IP/Protocol/Port such that TCP sessions will exist between the given end-user and server pair for the lifetime of the session. Optional health checks are used to identify application failures and reroute traffic in the case of a server outage.

+
+

Creating an L3 Load Balancer

+

To configure L3 (Anycast) load balancing:

+
    +
  1. Navigate to Services→Instances (ROH) and locate an existing ROH instance

  2. +
  3. Click the ellipses and then the Edit button

  4. +
  5. Select an extra IPv4 address from the select box at the bottom, and check the Anycast option.

  6. +
  7. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances.

  8. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/l4-load-balancer.html b/en/3.0/l4-load-balancer.html new file mode 100644 index 0000000000..d24c031024 --- /dev/null +++ b/en/3.0/l4-load-balancer.html @@ -0,0 +1,597 @@ + + + + + + + + + + + L4 Load Balancer (L4LB) — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • L4 Load Balancer (L4LB)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

L4 Load Balancer (L4LB)

+

Netris L4 Load Balancer (L4LB) leverages SoftGate(Linux router) nodes to provide Layer-4 load balancing services, including on-demand cloud load balancing with native integration with Kubernetes.

+
+

Enabling L4LB service

+

L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer).

+

The IP address pool for L4LB can be defined in the Net→IPAM section by adding an Allocation and setting the purpose field to ‘load-balancer’. You can define multiple IP pools for L4LB at any given site. See the below example.

+

Example: Adding a load-balancer IP pool assignment.

+Add an IP Allocation +

Screenshot: Listing of Net→IPAM after adding a load-balancer assignment

+List IP Subnets +
+
+

Consuming L4LB service

+

This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section.

+

Click +add under Services→L4 Load Balancer to request an L4LB service.

+

Add new L4 Load Balancer fields are described below:

+

General fields

+
    +
  • Name - Unique name.

  • +
  • Protocol - TCP or UDP.

  • +
  • Tenant - Requestor Tenant should have access to the backend IP space.

  • +
  • Site - Site where L4LB service is being requested for. Backends should belong on this site.

  • +
  • State - Administrative state.

  • +
+

Frontend

+
    +
  • Address - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses.

  • +
  • Port - TCP or UDP port to be exposed.

  • +
+

Health-check

+
    +
  • Type - Probe backends on service availability.

    +
      +
    • None - load balance unconditionally.

    • +
    • TCP - probe backend service availability through TCP connect checks.

    • +
    • HTTP - probe backend service availability through HTTP GET checks.

    • +
    +
  • +
  • Timeout(ms) - Probe timeout in milliseconds.

  • +
  • Request path - HTTP request path.

  • +
+

Backend

+
    +
  • +Add - add a backend host.

  • +
  • Address - IP address of the backend host.

  • +
  • Port - Service port on the backend host.

  • +
  • Enabled - Administrative state of particular backend.

  • +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/netris-architecture.html b/en/3.0/netris-architecture.html new file mode 100644 index 0000000000..80ac205dc9 --- /dev/null +++ b/en/3.0/netris-architecture.html @@ -0,0 +1,580 @@ + + + + + + + + + + + Netris Architecture — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Architecture

+

A Netris system is composed of 4 elements:

+
    +
  • Netris Controller

  • +
  • Netris Switch Agent

  • +
  • Netris SoftGate

  • +
  • Customer Application Servers

  • +
+
+

Netris Controller

+

Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. The Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud.

+

Diagram: High level Netris architecture

+_images/netris_controller_diagram.png +
    +
  • Controller HA We highly recommend running more than one copy of the controller for database replication.

  • +
  • Multiple sites Netris is designed to operate multiple sites with just a single controller with HA.

  • +
  • What if the controller is unreachable. Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers core operations will not be affected.

  • +
+
+
+

Netris Switch Agent

+

Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet.

+
+
+

Netris SoftGate

+

Netris SoftGate is automatic configuration software and reference architecture for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), and site-to-site VPN function on a regular x86 server with a SmartNIC card.

+

Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2.

+

The server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities.

+

Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels.

+

Diagram: Netris SoftGate high level architecture

+_images/softgate_diagram.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/network-policies.html b/en/3.0/network-policies.html new file mode 100644 index 0000000000..5bd13430f9 --- /dev/null +++ b/en/3.0/network-policies.html @@ -0,0 +1,876 @@ + + + + + + + + + + + Basic BGP — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Basic BGP

+

BGP neighbors can be declared in the Net→E-BGP section. Netris will automatically generate and program the network configuration to meet the requirements.

+
+

Adding BGP Peers

+
    +
  1. Navigate to Net→E-BGP in the web UI

  2. +
  3. Click the Add button

  4. +
  5. Fill in the fields as described in the table below

  6. +
  7. Click the Add button

  8. +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BGP Peer Fields

Name

User assigned name of BGP session

Description

Free description

Site

Selects the site (data center) where this BGP session should be terminated on.

Softgate

Only if SoftGate nodes are in use. Define on which node BGP session should be terminated on.

Neighbor AS

Autonomous System number of the remote side. (Local AS is defined at Net→Sites section)

Terminate on switch

Typically used for setups without SoftGate. For connecting with upstream routers. Instructs the system to terminate the BGP session directly on the switch.

Switch port

Switch Port for the physical cable to the BGP neighbor (any port on the fabric). Optionally can bind to a V-NET service. Typically used for peering with IXPs or systems like GGC (Google Global Cache).

VLAN ID

Optionally tag with a VLAN ID (usually untagged)

IP Version

IPv4 or IPv6

Local IP

BGP peering IP address on Netris controlled side

Remote IP

BGP peering IP address on the remote end

State

Administrative state (Enabled/Disabled)

Advanced

Advanced policy settings are described in the next section

+

Example: Declare a basic BGP neighbor

+_images/add-bgp-basic.png +

Example2: Declare BGP neighbor terminated on V-Net. Netris will automatically configure BGP session on the switch closest to the remote IP. +.. image:: images/add-bgp-basic-2.png

+
+
+
align
+

center

+
+
class
+

with-shadow

+
+
+
+
+
+
+

Advanced BGP

+

BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies.

+

Click Advanced to expand the BGP neighbor add/edit window.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BGP Peer Fields - Advanced

Neighbor address

IP address of the neighbor when peering with the loopback IP address instead of the interface IP address (aka Multihop).

Update source

When Multihop BGP peering is used it allows the operator to choose one of the loopback IP addresses of the SoftGate node as a BGP speaker source IP address.

BGP password

Password for the BGP session

Allowas-in

Define the number of allowed occurrences of the self AS number in the received BGP NLRI to consider it valid (normally 0).

Default Originate

Originate default route to the current neighbor

Prefix Inbound Max

Drop the BGP session if the number of received prefixes exceeds this max limit. For switch termination maximum allowed is 1000 prefixes and SoftGate termination can handle up to one million prefixes.

Inbound Route-Map

Apply BGP policies described in a route-map for inbound BGP updates

Outbound Route-Map

Apply BGP policies described in a route-map for outbound BGP updates

Local Preference

Set local preference for all inbound routes for the current neighbor

Weight

Set weight for all inbound routes for the current neighbor

Prepend Inbound (times)

How many times to prepend self AS number for inbound routes

Prepend Outbound (times)

How many times to prepend self AS number for outbound routes

Prefix List Inbound

List of IP addresses prefixes to permit or deny inbound

Prefix List Outbound

List of IP addresses prefixes to permit or deny outbound

Send BGP Community

List of BGP communities to send to the current neighbor

+
+
+

BGP Objects

+
+
Under Net→E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy.
+
Supported objects include:
+
+
    +
  • IPv4 Prefix

  • +
  • IPv6 Prefix

  • +
  • AS-PATH

  • +
  • Community

  • +
  • Extended Community

  • +
  • Large Community

  • +
+
+

IPv4 Prefix

+
+
Rules defined one per line.
+
Each line in IPv4 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv4 prefix (mandatory).

  • +
  • Length - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv4 Prefix list.

+_images/IPv4-Prefix.png +
+
+

IPv6 Prefix

+
+
Rules defined one per line.
+
Each line in IPv6 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv6 prefix (mandatory).

  • +
  • Keyword - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv6 Prefix list.

+_images/IPv6-Prefix.png +
+
+

Community

+
+
Community field has two parts:
+
+
    +
  • Action - Possible values: permit or deny (mandatory).

  • +
  • Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive).

  • +
+

Example: Creating community.

+_images/community.png +
+
+
+
+

BGP route-maps

+
+
Under the Net→E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound.
+
Description of route-map fields:
+
+
    +
  • Sequence Number - Automatically assigned a sequence number. Drag and move sequences to organize the order.

  • +
  • Description - Free description.

  • +
  • Policy - Permit or deny the routes which match below all match clauses within the current sequence.

  • +
  • Match - Rules for route matching.

    +
      +
    • Type - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag.

    • +
    • Object - Select an object from the list.

    • +
    +
  • +
  • Action - Action when all match clauses are met.

    +
      +
    • Action type - Define whether to manipulate a particular BGP attribute or go to another sequence.

    • +
    • Attribute - The attribute to be manipulated.

    • +
    • Value - New attribute value.

    • +
    +
  • +
+

Example: route-map

+_images/route-map.png +
+
+
+
+

Static Routing

+

Located under Net→Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. +We recommend using the Routes only if BGP is not supported by the remote end.

+
+
Typical use cases for static routing:
+
+
    +
  • To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported.

  • +
  • Temporary interconnection with the old network for a migration.

  • +
  • Routing a subnet behind a VM hypervisor machine for an internal VM network.

  • +
  • Specifically routing traffic destined to a particular prefix through an out-of-band management network.

  • +
+
+
Add new static route fields description:
+
+
    +
  • Prefix - Route destination to match.

  • +
  • Next-Hop - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network.

  • +
  • Description - Free description.

  • +
  • Site - Site where Route belongs.

  • +
  • State - Administrative (enable/disable) state of the Route.

  • +
  • Apply to - Limit the scope to particular units. It’s typically used for Null routes.

  • +
+

Example: Default route pointing to a Next-Hop that belongs to one of V-NETs.

+_images/defaultroute.png +

Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network.

+_images/static_route.png +

Screenshot: This Shows that my back route is actually applied on leaf1 and spine1.

+_images/leaf1_spine1.png +
+
+
+

NAT

+

Netris SoftGate nodes are required to support NAT (Network Address Translation).

+
+

Enabling NAT

+

To enable NAT for a given site, you first need to create a subnet with NAT purpose in the IPAM section. NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need.

+
    +
  1. Allocate a public IP subnet for NAT under Net→IPAM.

  2. +
+

Example: Adding an IP allocation under Net→Subnets.

+_images/IP-allocation.png +
    +
  1. Attach NAT IP addresses and/or NAT IP Pools to just one SoftGate node. Other SoftGate Nodes on the same site will automatically add the same NAT IP/Pool resources for proper consistency and high availability.

  2. +
+

Example: Adding NAT IP addresses and NAT IP Address Pools to a SoftGate node.

+_images/NATIP-address.png +
+
+

Defining NAT rules

+

NAT rules are defined under Net→NAT.

+

Example: SNAT all hosts on 10.0.0.0/8 to the Internet using 198.51.100.65 as a global IP.

+_images/globalIP.png +

Example: Port forwarding. DNAT the traffic destined to 198.51.100.66:80 to be forwarded to the host 10.0.4.10 on port tcp/1080.

+_images/Port-Forwarding.png +
+
+
+
+

SiteMesh

+

SiteMesh is a Netris service for site-to-site interconnects over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site.

+

Edit Net->Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below.

+
    +
  • Disabled - Do not participate in SiteMesh.

  • +
  • Hub - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites)

  • +
  • Spoke - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites)

  • +
  • Dynamic Spoke - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites)

  • +
+

Screenshot: Site Mesh parameter editing a Site under Net→Sites.

+_images/Site_Mesh.png +

You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge.

+_images/SiteMesh_modes.png +

Check the Net→Site Mesh section for the listing of tunnel statuses.

+

Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh)

+_images/SiteMesh_listing.png +
+
+
+

Looking Glass

+

The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Net→Looking Glass then selecting the device from the top-left dropdown menu.

+

Looking Glass controls described for IPv4/IPv6 protocol families.

+
    +
  • BGP Summary - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes.

  • +
  • BGP Route - Lookup the BGP table (RIB) for the given address.

  • +
  • Route - Lookup switch routing table for the given address.

  • +
  • Traceroute - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address.

  • +
  • Ping - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address.

  • +
+

Example: Spine1: listing BGP neighbors and number of received prefixes.

+_images/Spine1.png +

Example: BGP Route - looking up my leaf1 switch’s loopback address from spine1’s perspective. Spine1 is load balancing between two available paths.

+_images/BGP_route.png +

Example: Ping.

+_images/ping.png +
+
Looking Glass controls described for the EVPN family.
+
+
    +
  • BGP Summary - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received.

  • +
  • VNI - List VNIs learned.

  • +
  • BGP EVPN - List detailed EVPN routing information optionally for the given route distinguisher.

  • +
  • MAC table - List MAC address table for the given VNI.

  • +
+

Example: Listing of adjacent BGP neighbors and number of EVPN prefixes received.

+_images/BGP_neighbors_listing.png +

Example: Listing MAC addresses on VNI 2.

+_images/MAC_listing.png +

Example: EVPN routing information listing for a specified route distinguisher.

+_images/EVPN_routing.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/objects.inv b/en/3.0/objects.inv new file mode 100644 index 0000000000..67b000c7f5 Binary files /dev/null and b/en/3.0/objects.inv differ diff --git a/en/3.0/reference-designs.html b/en/3.0/reference-designs.html new file mode 100644 index 0000000000..58b05adb38 --- /dev/null +++ b/en/3.0/reference-designs.html @@ -0,0 +1,543 @@ + + + + + + + + + + + Network Reference Designs — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Network Reference Designs
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Network Reference Designs

+

Netris can support any type of standard network design.

+

The majority of designs fall into one of four patterns:

+
    +
  1. Unmanaged Switch

  2. +
  3. Single L2 Domain

  4. +
  5. Collapsed Core

  6. +
  7. Data Center Leaf/Spine Designs (Modern High Throughput Redundant Design)

  8. +
  9. Data Center Core/Distribution/Access (Legacy High Throughput Redundant Design)

  10. +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/release-notes.html b/en/3.0/release-notes.html new file mode 100644 index 0000000000..6306cb7dff --- /dev/null +++ b/en/3.0/release-notes.html @@ -0,0 +1,561 @@ + + + + + + + + + + + Release notes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Release notes

+
    +
  • DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGate performance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on.

  • +
  • L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, we now support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates with Kubernetes providing cloud-like load balancer service (type: load-balancer).

  • +
  • Kubenet​ - a network service purpose-built for Kubernetes cluster nodes. Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking with modern physical networking.

  • +
  • API logs​ - Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type.

  • +
  • Site Mesh​ - a Netris service for automatically configuring site-to-site interconnect over the public Internet. Site Mesh supports configuration for WireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a few clicks, services in one site get connectivity to services in other sites over a mesh of WireGuard tunnels.

  • +
  • Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loop cable. Removed the need for IP address reservation for V-NET, switching entirely to the anycast default gateway.

  • +
  • Controller distributions​ - Netris controller, is now available in three deployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application. 3) Managed/Hosted in the cloud.

  • +
  • Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes.

  • +
  • Switch/SoftGate agents​ - New installer with easy initial config tool. Support for IP and FQDN as a controller address. Authentication key.

  • +
  • GUI​ - Improved Net→Topology section, becoming the main and required place for defining the network topology. All sections got a column organizer, so every user can order and hide/show columns to their comfort.

  • +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/roh.html b/en/3.0/roh.html new file mode 100644 index 0000000000..acc3f81405 --- /dev/null +++ b/en/3.0/roh.html @@ -0,0 +1,585 @@ + + + + + + + + + + + ROH (Routing on the Host) — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • ROH (Routing on the Host)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

ROH (Routing on the Host)

+

To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly on their servers. This is commonly known as ROH (Routing on the Host).

+

In ROH architectures, servers use a routing daemon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon/suite is FRR.

+

Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead an IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. This is a modern and optimal design, leveraging Layer-3 networking from the fabric to the servers.

+

By using only Layer-3 interfaces, Layer-2 protocols such as Spanning Tree (STP) can be minized and the reliability of the network increases.

+

The ROH architecture that is configured by Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). For each instance of ROH, you’ll need to create a ROH entry in Netris Controller.

+
+

Adding ROH Hosts

+
    +
  1. Navigate in the Netris UI to Services→Instances (ROH)

  2. +
  3. Click the Add button

  4. +
  5. Fill out the form based on the fields in the table below.

  6. +
  7. Click the Add button

  8. +
+

Description of ROH instance fields:

+
    +
  • Name - Unique name for the ROH instance

  • +
  • Site - Site where the current ROH instance belongs

  • +
  • Type - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor.

  • +
  • ROH Routing Profile - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances

    +
      +
    • Default route only (most common design) - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch

    • +
    • Default + Aggregate - Will add prefixes of defined assignments + “Default” profile

    • +
    • Full table - Will advertise all prefixes available in the routing table of the connected switch

    • +
    • Inherit - Will inherit policy from site objects defined under Net→Sites

    • +
    +
  • +
  • Legacy Mode - Switch from default zero-config mode to using /30 IP addresses. Used for MSFT Windows Servers or other OS that doesn’t support FRR.

  • +
  • +Port - Physical Switch Ports anywhere on the network.

  • +
  • +IPv4 - IPv4 addresses for the loopback interface.

  • +
  • +Inbound Prefix List - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks.

  • +
+
+

Tip

+

Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox1/configurations.html b/en/3.0/sandbox/Sandbox1/configurations.html new file mode 100644 index 0000000000..002e2563b6 --- /dev/null +++ b/en/3.0/sandbox/Sandbox1/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox1.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc1::1 (from the “2607:f358:11:ffc1::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.24 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox1/creating-services.html b/en/3.0/sandbox/Sandbox1/creating-services.html new file mode 100644 index 0000000000..610bad9dc8 --- /dev/null +++ b/en/3.0/sandbox/Sandbox1/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1012.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.0/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.1)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.4/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.4/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.4/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.8/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox1/index.html b/en/3.0/sandbox/Sandbox1/index.html new file mode 100644 index 0000000000..169c5e6049 --- /dev/null +++ b/en/3.0/sandbox/Sandbox1/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox1 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox1/onprem-k8s.html b/en/3.0/sandbox/Sandbox1/onprem-k8s.html new file mode 100644 index 0000000000..51de60fd81 --- /dev/null +++ b/en/3.0/sandbox/Sandbox1/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+images/sandbox1-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox1.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox1.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+images/sandbox1-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.13   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.13:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+images/sandbox1-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.13
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.13   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.13   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.13   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.13   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.14   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.14
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+images/sandbox1-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.14
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.14
+
+SRV05-NYC
+curl 45.38.161.14
+
+SRV05-NYC
+curl 45.38.161.14
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1012
+  localIP: 45.38.161.22/30
+  remoteIP: 45.38.161.21/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.0/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.22/30   45.38.161.21/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.22/30   45.38.161.21/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.22/30   45.38.161.21/30    7m59s
+sandbox1-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox1-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox1-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.22/30   45.38.161.21/30    8m41s
+sandbox1-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox1-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox1-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.13
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox1/sandbox-info.html b/en/3.0/sandbox/Sandbox1/sandbox-info.html new file mode 100644 index 0000000000..7b670089dd --- /dev/null +++ b/en/3.0/sandbox/Sandbox1/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox1.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.24 -p 30061
+srv02-nyc: ssh demo@166.88.17.24 -p 30062
+srv03-nyc: ssh demo@166.88.17.24 -p 30063
+srv04-nyc: ssh demo@166.88.17.24 -p 30064
+srv05-nyc: ssh demo@166.88.17.24 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1011
+Local Address:                  45.38.161.18/30
+Remote Address:                 45.38.161.17/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.0/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1011
+Local Address:                  2607:f358:11:ffc0::3/127
+Remote Address:                 2607:f358:11:ffc0::2/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc1::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1012
+Local Address:                  45.38.161.22/30
+Remote Address:                 45.38.161.21/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.0/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.0/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.0/30
+|___ NAT Subnet:               45.38.161.4/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.8/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.12/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc1::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc1::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox10/configurations.html b/en/3.0/sandbox/Sandbox10/configurations.html new file mode 100644 index 0000000000..053a8d36af --- /dev/null +++ b/en/3.0/sandbox/Sandbox10/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox10.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffca::1 (from the “2607:f358:11:ffca::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.19 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox10/creating-services.html b/en/3.0/sandbox/Sandbox10/creating-services.html new file mode 100644 index 0000000000..b85b3c7398 --- /dev/null +++ b/en/3.0/sandbox/Sandbox10/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1102.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.208/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.209)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.212/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.212/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.212/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.216/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox10/index.html b/en/3.0/sandbox/Sandbox10/index.html new file mode 100644 index 0000000000..67fbc5b4aa --- /dev/null +++ b/en/3.0/sandbox/Sandbox10/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox10 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox10/onprem-k8s.html b/en/3.0/sandbox/Sandbox10/onprem-k8s.html new file mode 100644 index 0000000000..92fa94426e --- /dev/null +++ b/en/3.0/sandbox/Sandbox10/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox10.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox10.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.221   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.221:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.221
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.221   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.221   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.221   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.221   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.222   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.222
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.222
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.222
+
+SRV05-NYC
+curl 50.117.59.222
+
+SRV05-NYC
+curl 50.117.59.222
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1102
+  localIP: 50.117.59.126/30
+  remoteIP: 50.117.59.125/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.208/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.126/30   50.117.59.125/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.126/30   50.117.59.125/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.126/30   50.117.59.125/30    7m59s
+sandbox10-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox10-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox10-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.126/30   50.117.59.125/30    8m41s
+sandbox10-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox10-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox10-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.221
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox10/sandbox-info.html b/en/3.0/sandbox/Sandbox10/sandbox-info.html new file mode 100644 index 0000000000..d34ba41c67 --- /dev/null +++ b/en/3.0/sandbox/Sandbox10/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox10.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.19 -p 30061
+srv02-nyc: ssh demo@166.88.17.19 -p 30062
+srv03-nyc: ssh demo@166.88.17.19 -p 30063
+srv04-nyc: ssh demo@166.88.17.19 -p 30064
+srv05-nyc: ssh demo@166.88.17.19 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1101
+Local Address:                  50.117.59.122/30
+Remote Address:                 50.117.59.121/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.208/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1101
+Local Address:                  2607:f358:11:ffc0::15/127
+Remote Address:                 2607:f358:11:ffc0::14/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffca::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1102
+Local Address:                  50.117.59.126/30
+Remote Address:                 50.117.59.125/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.208/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.208/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.208/30
+|___ NAT Subnet:               50.117.59.212/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.216/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.220/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffca::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffca::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox11/configurations.html b/en/3.0/sandbox/Sandbox11/configurations.html new file mode 100644 index 0000000000..2ee1629807 --- /dev/null +++ b/en/3.0/sandbox/Sandbox11/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox11.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcb::1 (from the “2607:f358:11:ffcb::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.82 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox11/creating-services.html b/en/3.0/sandbox/Sandbox11/creating-services.html new file mode 100644 index 0000000000..054e0e0d5a --- /dev/null +++ b/en/3.0/sandbox/Sandbox11/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.82 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1112.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.118.

    6. +
    7. In the Remote IP field, type in 45.38.161.117.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.96/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.97)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.82 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.100/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.100/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.100/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.82 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.104/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.104”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.104” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.104) into the browser’s address bar or simply visit http://45.38.161.104/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.104 (name_45.38.161.104)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.104 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox11/index.html b/en/3.0/sandbox/Sandbox11/index.html new file mode 100644 index 0000000000..b6461d7855 --- /dev/null +++ b/en/3.0/sandbox/Sandbox11/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox11 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox11/onprem-k8s.html b/en/3.0/sandbox/Sandbox11/onprem-k8s.html new file mode 100644 index 0000000000..7659327e07 --- /dev/null +++ b/en/3.0/sandbox/Sandbox11/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox11.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox11.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.109   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.109:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.109
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.109   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.109   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.109   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.109   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.110   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.110
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.110
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.110
+
+SRV05-NYC
+curl 45.38.161.110
+
+SRV05-NYC
+curl 45.38.161.110
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1112
+  localIP: 45.38.161.118/30
+  remoteIP: 45.38.161.117/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.96/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.118/30   45.38.161.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.118/30   45.38.161.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.118/30   45.38.161.117/30    7m59s
+sandbox11-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox11-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox11-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.118/30   45.38.161.117/30    8m41s
+sandbox11-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox11-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox11-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.109
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox11/sandbox-info.html b/en/3.0/sandbox/Sandbox11/sandbox-info.html new file mode 100644 index 0000000000..db942bf662 --- /dev/null +++ b/en/3.0/sandbox/Sandbox11/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox11.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@50.117.27.82 -p 30061
+srv02-nyc: ssh demo@50.117.27.82 -p 30062
+srv03-nyc: ssh demo@50.117.27.82 -p 30063
+srv04-nyc: ssh demo@50.117.27.82 -p 30064
+srv05-nyc: ssh demo@50.117.27.82 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1111
+Local Address:                  45.38.161.114/30
+Remote Address:                 45.38.161.113/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.96/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1111
+Local Address:                  2607:f358:11:ffc0::17/127
+Remote Address:                 2607:f358:11:ffc0::16/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcb::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1112
+Local Address:                  45.38.161.118/30
+Remote Address:                 45.38.161.117/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.96/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.96/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.96/30
+|___ NAT Subnet:               45.38.161.100/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.104/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.108/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcb::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcb::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox12/configurations.html b/en/3.0/sandbox/Sandbox12/configurations.html new file mode 100644 index 0000000000..86c935e2ea --- /dev/null +++ b/en/3.0/sandbox/Sandbox12/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox12.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcc::1 (from the “2607:f358:11:ffcc::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.83 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox12/creating-services.html b/en/3.0/sandbox/Sandbox12/creating-services.html new file mode 100644 index 0000000000..d8de791e21 --- /dev/null +++ b/en/3.0/sandbox/Sandbox12/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.83 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1122.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.126.

    6. +
    7. In the Remote IP field, type in 45.38.161.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.128/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.129)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.83 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.132/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.132/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.132/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.83 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.136/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.136”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.136” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.136) into the browser’s address bar or simply visit http://45.38.161.136/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.136 (name_45.38.161.136)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.136 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox12/index.html b/en/3.0/sandbox/Sandbox12/index.html new file mode 100644 index 0000000000..1c5516e94e --- /dev/null +++ b/en/3.0/sandbox/Sandbox12/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox12 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox12/onprem-k8s.html b/en/3.0/sandbox/Sandbox12/onprem-k8s.html new file mode 100644 index 0000000000..05a5c9043d --- /dev/null +++ b/en/3.0/sandbox/Sandbox12/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox12.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox12.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.141   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.141:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.141
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.141   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.141   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.141   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.141   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.142   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.142
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.142
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.142
+
+SRV05-NYC
+curl 45.38.161.142
+
+SRV05-NYC
+curl 45.38.161.142
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1122
+  localIP: 45.38.161.126/30
+  remoteIP: 45.38.161.125/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.128/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.126/30   45.38.161.125/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.126/30   45.38.161.125/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.126/30   45.38.161.125/30    7m59s
+sandbox12-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox12-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox12-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.126/30   45.38.161.125/30    8m41s
+sandbox12-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox12-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox12-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.141
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox12/sandbox-info.html b/en/3.0/sandbox/Sandbox12/sandbox-info.html new file mode 100644 index 0000000000..37e4ca53e6 --- /dev/null +++ b/en/3.0/sandbox/Sandbox12/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox12.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@50.117.27.83 -p 30061
+srv02-nyc: ssh demo@50.117.27.83 -p 30062
+srv03-nyc: ssh demo@50.117.27.83 -p 30063
+srv04-nyc: ssh demo@50.117.27.83 -p 30064
+srv05-nyc: ssh demo@50.117.27.83 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1121
+Local Address:                  45.38.161.122/30
+Remote Address:                 45.38.161.121/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.128/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1121
+Local Address:                  2607:f358:11:ffc0::19/127
+Remote Address:                 2607:f358:11:ffc0::18/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcc::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1122
+Local Address:                  45.38.161.126/30
+Remote Address:                 45.38.161.125/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.128/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.128/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.128/30
+|___ NAT Subnet:               45.38.161.132/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.136/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.140/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcc::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcc::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox13/configurations.html b/en/3.0/sandbox/Sandbox13/configurations.html new file mode 100644 index 0000000000..2b382431e1 --- /dev/null +++ b/en/3.0/sandbox/Sandbox13/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox13.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcd::1 (from the “2607:f358:11:ffcd::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.84 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox13/creating-services.html b/en/3.0/sandbox/Sandbox13/creating-services.html new file mode 100644 index 0000000000..bc58d34a2c --- /dev/null +++ b/en/3.0/sandbox/Sandbox13/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.84 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1132.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.166.

    6. +
    7. In the Remote IP field, type in 45.38.161.165.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.144/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.145)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.84 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.148/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.148/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.148/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.84 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.152/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.152”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.152” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.152) into the browser’s address bar or simply visit http://45.38.161.152/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.152 (name_45.38.161.152)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.152 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox13/index.html b/en/3.0/sandbox/Sandbox13/index.html new file mode 100644 index 0000000000..abf2c6a562 --- /dev/null +++ b/en/3.0/sandbox/Sandbox13/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox13 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox13/onprem-k8s.html b/en/3.0/sandbox/Sandbox13/onprem-k8s.html new file mode 100644 index 0000000000..31be5714f0 --- /dev/null +++ b/en/3.0/sandbox/Sandbox13/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox13.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox13.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.157   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.157:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.157
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.157   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.157   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.157   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.157   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.158   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.158
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.158
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.158
+
+SRV05-NYC
+curl 45.38.161.158
+
+SRV05-NYC
+curl 45.38.161.158
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1132
+  localIP: 45.38.161.166/30
+  remoteIP: 45.38.161.165/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.144/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.166/30   45.38.161.165/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.166/30   45.38.161.165/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.166/30   45.38.161.165/30    7m59s
+sandbox13-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox13-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox13-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.166/30   45.38.161.165/30    8m41s
+sandbox13-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox13-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox13-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.157
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox13/sandbox-info.html b/en/3.0/sandbox/Sandbox13/sandbox-info.html new file mode 100644 index 0000000000..77c5febe34 --- /dev/null +++ b/en/3.0/sandbox/Sandbox13/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox13.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@50.117.27.84 -p 30061
+srv02-nyc: ssh demo@50.117.27.84 -p 30062
+srv03-nyc: ssh demo@50.117.27.84 -p 30063
+srv04-nyc: ssh demo@50.117.27.84 -p 30064
+srv05-nyc: ssh demo@50.117.27.84 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1131
+Local Address:                  45.38.161.162/30
+Remote Address:                 45.38.161.161/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.144/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1131
+Local Address:                  2607:f358:11:ffc0::1b/127
+Remote Address:                 2607:f358:11:ffc0::1a/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcd::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1132
+Local Address:                  45.38.161.166/30
+Remote Address:                 45.38.161.165/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.144/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.144/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.144/30
+|___ NAT Subnet:               45.38.161.148/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.152/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.156/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcd::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcd::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox14/configurations.html b/en/3.0/sandbox/Sandbox14/configurations.html new file mode 100644 index 0000000000..4e09a07846 --- /dev/null +++ b/en/3.0/sandbox/Sandbox14/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox14.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffce::1 (from the “2607:f358:11:ffce::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.85 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox14/creating-services.html b/en/3.0/sandbox/Sandbox14/creating-services.html new file mode 100644 index 0000000000..1a1ad2189d --- /dev/null +++ b/en/3.0/sandbox/Sandbox14/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.85 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1142.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.174.

    6. +
    7. In the Remote IP field, type in 45.38.161.173.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.176/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.177)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.85 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.180/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.180/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.180/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.85 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.184/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.184”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.184” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.184) into the browser’s address bar or simply visit http://45.38.161.184/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.184 (name_45.38.161.184)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.184 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox14/index.html b/en/3.0/sandbox/Sandbox14/index.html new file mode 100644 index 0000000000..d045bd6410 --- /dev/null +++ b/en/3.0/sandbox/Sandbox14/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox14 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox14/onprem-k8s.html b/en/3.0/sandbox/Sandbox14/onprem-k8s.html new file mode 100644 index 0000000000..32392bff4a --- /dev/null +++ b/en/3.0/sandbox/Sandbox14/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox15.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox15.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.189   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.189:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.189
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.189   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.189   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.189   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.189   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.190   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.190
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.190
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.190
+
+SRV05-NYC
+curl 45.38.161.190
+
+SRV05-NYC
+curl 45.38.161.190
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1142
+  localIP: 45.38.161.174/30
+  remoteIP: 45.38.161.173/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.176/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.174/30   45.38.161.173/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.174/30   45.38.161.173/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.174/30   45.38.161.173/30    7m59s
+sandbox15-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox15-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox15-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.174/30   45.38.161.173/30    8m41s
+sandbox15-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox15-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox15-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.189
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox14/sandbox-info.html b/en/3.0/sandbox/Sandbox14/sandbox-info.html new file mode 100644 index 0000000000..70b902fe9a --- /dev/null +++ b/en/3.0/sandbox/Sandbox14/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox14.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@50.117.27.85 -p 30061
+srv02-nyc: ssh demo@50.117.27.85 -p 30062
+srv03-nyc: ssh demo@50.117.27.85 -p 30063
+srv04-nyc: ssh demo@50.117.27.85 -p 30064
+srv05-nyc: ssh demo@50.117.27.85 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1141
+Local Address:                  45.38.161.170/30
+Remote Address:                 45.38.161.169/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.176/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1141
+Local Address:                  2607:f358:11:ffc0::1d/127
+Remote Address:                 2607:f358:11:ffc0::1c/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffce::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1142
+Local Address:                  45.38.161.174/30
+Remote Address:                 45.38.161.173/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.176/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.176/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.176/30
+|___ NAT Subnet:               45.38.161.180/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.184/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.188/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffce::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffce::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox15/configurations.html b/en/3.0/sandbox/Sandbox15/configurations.html new file mode 100644 index 0000000000..25de3cb85c --- /dev/null +++ b/en/3.0/sandbox/Sandbox15/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox15.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcf::1 (from the “2607:f358:11:ffcf::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.86 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox15/creating-services.html b/en/3.0/sandbox/Sandbox15/creating-services.html new file mode 100644 index 0000000000..a757326af4 --- /dev/null +++ b/en/3.0/sandbox/Sandbox15/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.86 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1152.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.214.

    6. +
    7. In the Remote IP field, type in 45.38.161.213.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.192/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.193)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.86 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.196/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.196/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.196/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.86 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.200/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.200”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.200” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.200) into the browser’s address bar or simply visit http://45.38.161.200/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.200 (name_45.38.161.200)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.200 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox15/index.html b/en/3.0/sandbox/Sandbox15/index.html new file mode 100644 index 0000000000..489365ec83 --- /dev/null +++ b/en/3.0/sandbox/Sandbox15/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox15 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox15/onprem-k8s.html b/en/3.0/sandbox/Sandbox15/onprem-k8s.html new file mode 100644 index 0000000000..e2e9aaec25 --- /dev/null +++ b/en/3.0/sandbox/Sandbox15/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox15.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox15.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.205   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.205:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.205
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.205   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.205   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.205   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.205   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.206   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.206
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.206
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.206
+
+SRV05-NYC
+curl 45.38.161.206
+
+SRV05-NYC
+curl 45.38.161.206
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1152
+  localIP: 45.38.161.214/30
+  remoteIP: 45.38.161.213/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.192/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.214/30   45.38.161.213/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.214/30   45.38.161.213/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.214/30   45.38.161.213/30    7m59s
+sandbox15-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox15-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox15-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.214/30   45.38.161.213/30    8m41s
+sandbox15-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox15-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox15-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.205
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox15/sandbox-info.html b/en/3.0/sandbox/Sandbox15/sandbox-info.html new file mode 100644 index 0000000000..36c1c012cd --- /dev/null +++ b/en/3.0/sandbox/Sandbox15/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox15.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@50.117.27.86 -p 30061
+srv02-nyc: ssh demo@50.117.27.86 -p 30062
+srv03-nyc: ssh demo@50.117.27.86 -p 30063
+srv04-nyc: ssh demo@50.117.27.86 -p 30064
+srv05-nyc: ssh demo@50.117.27.86 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1151
+Local Address:                  45.38.161.210/30
+Remote Address:                 45.38.161.209/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.192/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1151
+Local Address:                  2607:f358:11:ffc0::1f/127
+Remote Address:                 2607:f358:11:ffc0::1e/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcf::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1152
+Local Address:                  45.38.161.214/30
+Remote Address:                 45.38.161.213/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.192/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.192/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.192/30
+|___ NAT Subnet:               45.38.161.196/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.200/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.204/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcf::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcf::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox2/configurations.html b/en/3.0/sandbox/Sandbox2/configurations.html new file mode 100644 index 0000000000..4c025783ab --- /dev/null +++ b/en/3.0/sandbox/Sandbox2/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox2.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc2::1 (from the “2607:f358:11:ffc2::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.190 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox2/creating-services.html b/en/3.0/sandbox/Sandbox2/creating-services.html new file mode 100644 index 0000000000..b644fb7135 --- /dev/null +++ b/en/3.0/sandbox/Sandbox2/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1022.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.32/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.33)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.36/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.36/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.36/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.40/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox2/index.html b/en/3.0/sandbox/Sandbox2/index.html new file mode 100644 index 0000000000..7fff599cbd --- /dev/null +++ b/en/3.0/sandbox/Sandbox2/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox2 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox2/onprem-k8s.html b/en/3.0/sandbox/Sandbox2/onprem-k8s.html new file mode 100644 index 0000000000..6817cdc002 --- /dev/null +++ b/en/3.0/sandbox/Sandbox2/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+images/sandbox2-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox2.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox2.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+images/sandbox2-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.45   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.45:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+images/sandbox2-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.45
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.45   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.45   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.45   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.45   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.46   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.46
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+images/sandbox2-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.46
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.46
+
+SRV05-NYC
+curl 45.38.161.46
+
+SRV05-NYC
+curl 45.38.161.46
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1022
+  localIP: 45.38.161.30/30
+  remoteIP: 45.38.161.29/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.32/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.30/30   45.38.161.29/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.30/30   45.38.161.29/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.30/30   45.38.161.29/30    7m59s
+sandbox2-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox2-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox2-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.30/30   45.38.161.29/30    8m41s
+sandbox2-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox2-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox2-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.45
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox2/sandbox-info.html b/en/3.0/sandbox/Sandbox2/sandbox-info.html new file mode 100644 index 0000000000..12c4a9f455 --- /dev/null +++ b/en/3.0/sandbox/Sandbox2/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox2.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.190 -p 30061
+srv02-nyc: ssh demo@166.88.17.190 -p 30062
+srv03-nyc: ssh demo@166.88.17.190 -p 30063
+srv04-nyc: ssh demo@166.88.17.190 -p 30064
+srv05-nyc: ssh demo@166.88.17.190 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1021
+Local Address:                  45.38.161.26/30
+Remote Address:                 45.38.161.25/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.32/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1021
+Local Address:                  2607:f358:11:ffc0::5/127
+Remote Address:                 2607:f358:11:ffc0::4/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc2::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1022
+Local Address:                  45.38.161.30/30
+Remote Address:                 45.38.161.29/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.32/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.32/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.32/30
+|___ NAT Subnet:               45.38.161.36/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.40/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.44/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc2::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc2::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox3/configurations.html b/en/3.0/sandbox/Sandbox3/configurations.html new file mode 100644 index 0000000000..900f16b89c --- /dev/null +++ b/en/3.0/sandbox/Sandbox3/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox3.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc3::1 (from the “2607:f358:11:ffc3::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.189 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox3/creating-services.html b/en/3.0/sandbox/Sandbox3/creating-services.html new file mode 100644 index 0000000000..eeb991c62a --- /dev/null +++ b/en/3.0/sandbox/Sandbox3/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1032.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.48/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.49)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.52/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.52/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.52/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.56/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox3/index.html b/en/3.0/sandbox/Sandbox3/index.html new file mode 100644 index 0000000000..64d71a34b1 --- /dev/null +++ b/en/3.0/sandbox/Sandbox3/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox3 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox3/onprem-k8s.html b/en/3.0/sandbox/Sandbox3/onprem-k8s.html new file mode 100644 index 0000000000..d3059774c7 --- /dev/null +++ b/en/3.0/sandbox/Sandbox3/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox3.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox3.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.61   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.61:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.61
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.61   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.61   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.61   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.61   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.62   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.62
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.62
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.62
+
+SRV05-NYC
+curl 45.38.161.62
+
+SRV05-NYC
+curl 45.38.161.62
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1032
+  localIP: 45.38.161.70/30
+  remoteIP: 45.38.161.69/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.48/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.70/30   45.38.161.69/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.70/30   45.38.161.69/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.70/30   45.38.161.69/30    7m59s
+sandbox3-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox3-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox3-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.70/30   45.38.161.69/30    8m41s
+sandbox3-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox3-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox3-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.61
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox3/sandbox-info.html b/en/3.0/sandbox/Sandbox3/sandbox-info.html new file mode 100644 index 0000000000..f674ac919a --- /dev/null +++ b/en/3.0/sandbox/Sandbox3/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox3.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.189 -p 30061
+srv02-nyc: ssh demo@166.88.17.189 -p 30062
+srv03-nyc: ssh demo@166.88.17.189 -p 30063
+srv04-nyc: ssh demo@166.88.17.189 -p 30064
+srv05-nyc: ssh demo@166.88.17.189 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1031
+Local Address:                  45.38.161.66/30
+Remote Address:                 45.38.161.65/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.48/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1031
+Local Address:                  2607:f358:11:ffc0::7/127
+Remote Address:                 2607:f358:11:ffc0::6/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc3::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1032
+Local Address:                  45.38.161.70/30
+Remote Address:                 45.38.161.69/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.48/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.48/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.48/30
+|___ NAT Subnet:               45.38.161.52/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.56/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.60/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc3::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc3::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox4/configurations.html b/en/3.0/sandbox/Sandbox4/configurations.html new file mode 100644 index 0000000000..4ea5f0f7a3 --- /dev/null +++ b/en/3.0/sandbox/Sandbox4/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox4.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc4::1 (from the “2607:f358:11:ffc4::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.188 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox4/creating-services.html b/en/3.0/sandbox/Sandbox4/creating-services.html new file mode 100644 index 0000000000..dbe2abbb16 --- /dev/null +++ b/en/3.0/sandbox/Sandbox4/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1042.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.80/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.81)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.84/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.84/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.84/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.88/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox4/index.html b/en/3.0/sandbox/Sandbox4/index.html new file mode 100644 index 0000000000..32221a9ce6 --- /dev/null +++ b/en/3.0/sandbox/Sandbox4/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox4 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox4/onprem-k8s.html b/en/3.0/sandbox/Sandbox4/onprem-k8s.html new file mode 100644 index 0000000000..f82ca079db --- /dev/null +++ b/en/3.0/sandbox/Sandbox4/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox4.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox4.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.93   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.93:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.93
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.93   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.93   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.93   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.93   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.94   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.94
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.94
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.94
+
+SRV05-NYC
+curl 45.38.161.94
+
+SRV05-NYC
+curl 45.38.161.94
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1042
+  localIP: 45.38.161.78/30
+  remoteIP: 45.38.161.77/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.80/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.78/30   45.38.161.77/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.78/30   45.38.161.77/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.78/30   45.38.161.77/30    7m59s
+sandbox4-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox4-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox4-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.78/30   45.38.161.77/30    8m41s
+sandbox4-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox4-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox4-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.93
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox4/sandbox-info.html b/en/3.0/sandbox/Sandbox4/sandbox-info.html new file mode 100644 index 0000000000..b6c71ddd73 --- /dev/null +++ b/en/3.0/sandbox/Sandbox4/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox4.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.188 -p 30061
+srv02-nyc: ssh demo@166.88.17.188 -p 30062
+srv03-nyc: ssh demo@166.88.17.188 -p 30063
+srv04-nyc: ssh demo@166.88.17.188 -p 30064
+srv05-nyc: ssh demo@166.88.17.188 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1041
+Local Address:                  45.38.161.74/30
+Remote Address:                 45.38.161.73/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.80/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1041
+Local Address:                  2607:f358:11:ffc0::9/127
+Remote Address:                 2607:f358:11:ffc0::8/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc4::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1042
+Local Address:                  45.38.161.78/30
+Remote Address:                 45.38.161.77/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.80/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.80/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.80/30
+|___ NAT Subnet:               45.38.161.84/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.88/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.92/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc4::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc4::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox5/configurations.html b/en/3.0/sandbox/Sandbox5/configurations.html new file mode 100644 index 0000000000..8455a30c7a --- /dev/null +++ b/en/3.0/sandbox/Sandbox5/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox5.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc5::1 (from the “2607:f358:11:ffc5::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.187 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox5/creating-services.html b/en/3.0/sandbox/Sandbox5/creating-services.html new file mode 100644 index 0000000000..3067f2eed2 --- /dev/null +++ b/en/3.0/sandbox/Sandbox5/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1052.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.128/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.129)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.132/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.132/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.132/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.136/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox5/index.html b/en/3.0/sandbox/Sandbox5/index.html new file mode 100644 index 0000000000..3a95731849 --- /dev/null +++ b/en/3.0/sandbox/Sandbox5/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox5 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox5/onprem-k8s.html b/en/3.0/sandbox/Sandbox5/onprem-k8s.html new file mode 100644 index 0000000000..8c2139fa3c --- /dev/null +++ b/en/3.0/sandbox/Sandbox5/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox5.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox5.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.141   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.141:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.141
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.141   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.141   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.141   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.141   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.142   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.142
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.142
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.142
+
+SRV05-NYC
+curl 50.117.59.142
+
+SRV05-NYC
+curl 50.117.59.142
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1052
+  localIP: 50.117.59.86/30
+  remoteIP: 50.117.59.85/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.128/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.86/30   50.117.59.85/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.86/30   50.117.59.85/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.86/30   50.117.59.85/30    7m59s
+sandbox5-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox5-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox5-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.86/30   50.117.59.85/30    8m41s
+sandbox5-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox5-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox5-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.141
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox5/sandbox-info.html b/en/3.0/sandbox/Sandbox5/sandbox-info.html new file mode 100644 index 0000000000..0b8b291185 --- /dev/null +++ b/en/3.0/sandbox/Sandbox5/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox5.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.187 -p 30061
+srv02-nyc: ssh demo@166.88.17.187 -p 30062
+srv03-nyc: ssh demo@166.88.17.187 -p 30063
+srv04-nyc: ssh demo@166.88.17.187 -p 30064
+srv05-nyc: ssh demo@166.88.17.187 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1051
+Local Address:                  50.117.59.82/30
+Remote Address:                 50.117.59.81/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.128/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1051
+Local Address:                  2607:f358:11:ffc0::b/127
+Remote Address:                 2607:f358:11:ffc0::a/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc5::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1052
+Local Address:                  50.117.59.86/30
+Remote Address:                 50.117.59.85/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.128/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.128/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.128/30
+|___ NAT Subnet:               50.117.59.132/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.136/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.140/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc5::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc5::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox6/configurations.html b/en/3.0/sandbox/Sandbox6/configurations.html new file mode 100644 index 0000000000..91c2284756 --- /dev/null +++ b/en/3.0/sandbox/Sandbox6/configurations.html @@ -0,0 +1,590 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox6.netris.ai and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.186 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox6/creating-services.html b/en/3.0/sandbox/Sandbox6/creating-services.html new file mode 100644 index 0000000000..3a47d8aa29 --- /dev/null +++ b/en/3.0/sandbox/Sandbox6/creating-services.html @@ -0,0 +1,673 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1062.

    4. +
    5. In the Local IP field, type in 50.117.59.94

    6. +
    7. In the Remote IP field, type in 50.117.59.93.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.144/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.150/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.ai and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox6/index.html b/en/3.0/sandbox/Sandbox6/index.html new file mode 100644 index 0000000000..f25850b50d --- /dev/null +++ b/en/3.0/sandbox/Sandbox6/index.html @@ -0,0 +1,569 @@ + + + + + + + + + + Sandbox6 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox6/onprem-k8s.html b/en/3.0/sandbox/Sandbox6/onprem-k8s.html new file mode 100644 index 0000000000..36d70a0c17 --- /dev/null +++ b/en/3.0/sandbox/Sandbox6/onprem-k8s.html @@ -0,0 +1,1020 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox6.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox6.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.154   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.154:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.154
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.155   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.155
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.155
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1062
+  localIP: 50.117.59.94/30
+  remoteIP: 50.117.59.93/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.144/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.94/30   50.117.59.93/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.94/30   50.117.59.93/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.94/30   50.117.59.93/30    7m59s
+sandbox6-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox6-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox6-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.94/30   50.117.59.93/30    8m41s
+sandbox6-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox6-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox6-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.154
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox6/sandbox-info.html b/en/3.0/sandbox/Sandbox6/sandbox-info.html new file mode 100644 index 0000000000..38a10fe3ea --- /dev/null +++ b/en/3.0/sandbox/Sandbox6/sandbox-info.html @@ -0,0 +1,606 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology_n.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing Linux servers:

+
srv01: ssh demo@166.88.17.186 -p 30061
+srv02: ssh demo@166.88.17.186 -p 30062
+srv03: ssh demo@166.88.17.186 -p 30063
+srv04: ssh demo@166.88.17.186 -p 30064
+srv05: ssh demo@166.88.17.186 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1061
+IP customer: 50.117.59.90/30
+IP Iris: 50.117.59.89/30
+
+Neighbor AS: 65007
+Vlan: 1062
+IP customer:  50.117.59.94/30
+IP Iris: 50.117.59.93/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     50.117.59.144/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox7/configurations.html b/en/3.0/sandbox/Sandbox7/configurations.html new file mode 100644 index 0000000000..02d1a4d37c --- /dev/null +++ b/en/3.0/sandbox/Sandbox7/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox7.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc7::1 (from the “2607:f358:11:ffc7::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.185 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox7/creating-services.html b/en/3.0/sandbox/Sandbox7/creating-services.html new file mode 100644 index 0000000000..ea1d724351 --- /dev/null +++ b/en/3.0/sandbox/Sandbox7/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1072.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.160/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.161)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.164/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.164/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.164/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.168/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox7/index.html b/en/3.0/sandbox/Sandbox7/index.html new file mode 100644 index 0000000000..deef30a4d9 --- /dev/null +++ b/en/3.0/sandbox/Sandbox7/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox7 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox7/onprem-k8s.html b/en/3.0/sandbox/Sandbox7/onprem-k8s.html new file mode 100644 index 0000000000..f617d71926 --- /dev/null +++ b/en/3.0/sandbox/Sandbox7/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox7.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox7.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.173   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.173:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.173
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.173   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.173   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.173   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.173   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.174   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.174
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.174
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.174
+
+SRV05-NYC
+curl 50.117.59.174
+
+SRV05-NYC
+curl 50.117.59.174
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1072
+  localIP: 50.117.59.102/30
+  remoteIP: 50.117.59.101/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.160/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.102/30   50.117.59.101/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.102/30   50.117.59.101/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.102/30   50.117.59.101/30    7m59s
+sandbox7-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox7-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox7-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.102/30   50.117.59.101/30    8m41s
+sandbox7-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox7-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox7-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.173
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox7/sandbox-info.html b/en/3.0/sandbox/Sandbox7/sandbox-info.html new file mode 100644 index 0000000000..6ace435e9e --- /dev/null +++ b/en/3.0/sandbox/Sandbox7/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox7.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.185 -p 30061
+srv02-nyc: ssh demo@166.88.17.185 -p 30062
+srv03-nyc: ssh demo@166.88.17.185 -p 30063
+srv04-nyc: ssh demo@166.88.17.185 -p 30064
+srv05-nyc: ssh demo@166.88.17.185 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1071
+Local Address:                  50.117.59.98/30
+Remote Address:                 50.117.59.97/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.160/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1071
+Local Address:                  2607:f358:11:ffc0::f/127
+Remote Address:                 2607:f358:11:ffc0::e/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc7::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1072
+Local Address:                  50.117.59.102/30
+Remote Address:                 50.117.59.101/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.160/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.160/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.160/30
+|___ NAT Subnet:               50.117.59.164/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.168/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.172/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc7::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc7::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox8/configurations.html b/en/3.0/sandbox/Sandbox8/configurations.html new file mode 100644 index 0000000000..c4f5a46688 --- /dev/null +++ b/en/3.0/sandbox/Sandbox8/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox8.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc8::1 (from the “2607:f358:11:ffc8::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.29 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox8/creating-services.html b/en/3.0/sandbox/Sandbox8/creating-services.html new file mode 100644 index 0000000000..fdf925726d --- /dev/null +++ b/en/3.0/sandbox/Sandbox8/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.29 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1082.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.176/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.177)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.29 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.180/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.180/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.180/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.29 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.184/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox8/index.html b/en/3.0/sandbox/Sandbox8/index.html new file mode 100644 index 0000000000..01ab2e91a5 --- /dev/null +++ b/en/3.0/sandbox/Sandbox8/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox8 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox8/onprem-k8s.html b/en/3.0/sandbox/Sandbox8/onprem-k8s.html new file mode 100644 index 0000000000..c36e189df8 --- /dev/null +++ b/en/3.0/sandbox/Sandbox8/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox8.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox8.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.189   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.189:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.189
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.189   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.189   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.189   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.189   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.190   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.190
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.190
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.190
+
+SRV05-NYC
+curl 50.117.59.190
+
+SRV05-NYC
+curl 50.117.59.190
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1082
+  localIP: 50.117.59.110/30
+  remoteIP: 50.117.59.109/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.176/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.110/30   50.117.59.109/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.110/30   50.117.59.109/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.110/30   50.117.59.109/30    7m59s
+sandbox8-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox8-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox8-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.110/30   50.117.59.109/30    8m41s
+sandbox8-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox8-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox8-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.189
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox8/sandbox-info.html b/en/3.0/sandbox/Sandbox8/sandbox-info.html new file mode 100644 index 0000000000..88321034c3 --- /dev/null +++ b/en/3.0/sandbox/Sandbox8/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox8.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.29 -p 30061
+srv02-nyc: ssh demo@166.88.17.29 -p 30062
+srv03-nyc: ssh demo@166.88.17.29 -p 30063
+srv04-nyc: ssh demo@166.88.17.29 -p 30064
+srv05-nyc: ssh demo@166.88.17.29 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1081
+Local Address:                  50.117.59.106/30
+Remote Address:                 50.117.59.105/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.176/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1081
+Local Address:                  2607:f358:11:ffc0::11/127
+Remote Address:                 2607:f358:11:ffc0::10/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc8::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1082
+Local Address:                  50.117.59.110/30
+Remote Address:                 50.117.59.109/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.176/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.176/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.176/30
+|___ NAT Subnet:               50.117.59.180/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.184/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.188/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc8::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc8::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox9/configurations.html b/en/3.0/sandbox/Sandbox9/configurations.html new file mode 100644 index 0000000000..bc8d3ba621 --- /dev/null +++ b/en/3.0/sandbox/Sandbox9/configurations.html @@ -0,0 +1,596 @@ + + + + + + + + + + Provided Example Configurations — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox9.netris.ai and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc9::1 (from the “2607:f358:11:ffc9::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric, you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.22 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox9/creating-services.html b/en/3.0/sandbox/Sandbox9/creating-services.html new file mode 100644 index 0000000000..68d42bc61a --- /dev/null +++ b/en/3.0/sandbox/Sandbox9/creating-services.html @@ -0,0 +1,748 @@ + + + + + + + + + + Learn by Creating Services — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision desired services within a matter of minutes.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give the srv05-nyc server the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.22 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.ai and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Owner drop-down menu, select “Demo”.

    8. +
    9. From the Sites drop-down menu, select “US/NYC”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Port drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.ai and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1092.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.192/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.193)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.22 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.ai and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.196/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.196/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.196/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.22 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.ai and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris software has finished syncing the new ACL policy with all the member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.ai and navigate to Services → Instances(ROH).

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.200/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox9/index.html b/en/3.0/sandbox/Sandbox9/index.html new file mode 100644 index 0000000000..bcf27c0aaf --- /dev/null +++ b/en/3.0/sandbox/Sandbox9/index.html @@ -0,0 +1,570 @@ + + + + + + + + + + Sandbox9 — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox9/onprem-k8s.html b/en/3.0/sandbox/Sandbox9/onprem-k8s.html new file mode 100644 index 0000000000..3f16376bbc --- /dev/null +++ b/en/3.0/sandbox/Sandbox9/onprem-k8s.html @@ -0,0 +1,1022 @@ + + + + + + + + + + Learn Netris operations with Kubernetes — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in each sandbox to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox3-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox9.netris.ai:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v1.4.4/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox9.netris.ai' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox3-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.205   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.205:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox3-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.205
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.205   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.205   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.205   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.205   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.206   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.206
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox3-l4lbs.png +
+
+

VNet Custom Resource

+

If you see the same as shown in the previous screenshot, it means you didn’t create “vnet-customer” VNet as stated in the “Learn by Creating Services” manual. If so, let’s create it from Kubernetes using the VNet custom resource.

+

Let’s create our VNet manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.206
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.206
+
+SRV05-NYC
+curl 50.117.59.206
+
+SRV05-NYC
+curl 50.117.59.206
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” VNet as stated in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in VNet manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

After applying the manifest containing “import” annotation, the VNet, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1092
+  localIP: 50.117.59.118/30
+  remoteIP: 50.117.59.117/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.192/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.118/30   50.117.59.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.118/30   50.117.59.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

And one more time about importing resources. You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.118/30   50.117.59.117/30    7m59s
+sandbox9-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox9-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox9-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.118/30   50.117.59.117/30    8m41s
+sandbox9-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox9-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox9-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.205
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/sandbox/Sandbox9/sandbox-info.html b/en/3.0/sandbox/Sandbox9/sandbox-info.html new file mode 100644 index 0000000000..9bce01efc4 --- /dev/null +++ b/en/3.0/sandbox/Sandbox9/sandbox-info.html @@ -0,0 +1,646 @@ + + + + + + + + + + Welcome to Netris Sandbox — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. Reach out to us if you have any questions at https://netris.ai/slack

+

The credentials for the sandbox have been provided to you by email in response to your sandbox request.

+

This environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all Netris-operated.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both Netris-operated.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox9.netris.ai

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing Linux servers:

+
srv01-nyc: ssh demo@166.88.17.22 -p 30061
+srv02-nyc: ssh demo@166.88.17.22 -p 30062
+srv03-nyc: ssh demo@166.88.17.22 -p 30063
+srv04-nyc: ssh demo@166.88.17.22 -p 30064
+srv05-nyc: ssh demo@166.88.17.22 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris controller by installing netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This sandbox provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1091
+Local Address:                  50.117.59.114/30
+Remote Address:                 50.117.59.113/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.192/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1091
+Local Address:                  2607:f358:11:ffc0::13/127
+Remote Address:                 2607:f358:11:ffc0::12/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc9::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1092
+Local Address:                  50.117.59.118/30
+Remote Address:                 50.117.59.117/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.192/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.192/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.192/30
+|___ NAT Subnet:               50.117.59.196/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.200/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.204/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc9::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc9::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/search.html b/en/3.0/search.html new file mode 100644 index 0000000000..a573173a33 --- /dev/null +++ b/en/3.0/search.html @@ -0,0 +1,545 @@ + + + + + + + + + + Search — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Search
  • + + +
  • + +
  • + +
+ + +
+
+
+
+ + + + +
+ +
+ +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/searchindex.js b/en/3.0/searchindex.js new file mode 100644 index 0000000000..1a9cfc518a --- /dev/null +++ b/en/3.0/searchindex.js @@ -0,0 +1 @@ +Search.setIndex({"docnames": ["SoftGate-PRO-installation", "SoftGate-installation", "accounts", "acls", "controller-k3s-installation", "controller-k8s-installation", "controller-k8s-quickstart", "controller-vm-installation", "definitions", "index", "installation", "installing-netris-controller", "introduction", "inventory-profiles", "ipam", "kubernetes-integration", "l3-load-balancer", "l4-load-balancer", "netris-architecture", "network-policies", "reference-designs", "release-notes", "roh", "sandbox/Sandbox1/configurations", "sandbox/Sandbox1/creating-services", "sandbox/Sandbox1/index", "sandbox/Sandbox1/onprem-k8s", "sandbox/Sandbox1/sandbox-info", "sandbox/Sandbox10/configurations", "sandbox/Sandbox10/creating-services", "sandbox/Sandbox10/index", "sandbox/Sandbox10/onprem-k8s", "sandbox/Sandbox10/sandbox-info", "sandbox/Sandbox11/configurations", "sandbox/Sandbox11/creating-services", "sandbox/Sandbox11/index", "sandbox/Sandbox11/onprem-k8s", "sandbox/Sandbox11/sandbox-info", "sandbox/Sandbox12/configurations", "sandbox/Sandbox12/creating-services", "sandbox/Sandbox12/index", "sandbox/Sandbox12/onprem-k8s", "sandbox/Sandbox12/sandbox-info", "sandbox/Sandbox13/configurations", "sandbox/Sandbox13/creating-services", "sandbox/Sandbox13/index", "sandbox/Sandbox13/onprem-k8s", "sandbox/Sandbox13/sandbox-info", "sandbox/Sandbox14/configurations", "sandbox/Sandbox14/creating-services", "sandbox/Sandbox14/index", "sandbox/Sandbox14/onprem-k8s", "sandbox/Sandbox14/sandbox-info", "sandbox/Sandbox15/configurations", "sandbox/Sandbox15/creating-services", "sandbox/Sandbox15/index", "sandbox/Sandbox15/onprem-k8s", "sandbox/Sandbox15/sandbox-info", "sandbox/Sandbox2/configurations", "sandbox/Sandbox2/creating-services", "sandbox/Sandbox2/index", "sandbox/Sandbox2/onprem-k8s", "sandbox/Sandbox2/sandbox-info", "sandbox/Sandbox3/configurations", "sandbox/Sandbox3/creating-services", "sandbox/Sandbox3/index", "sandbox/Sandbox3/onprem-k8s", "sandbox/Sandbox3/sandbox-info", "sandbox/Sandbox4/configurations", "sandbox/Sandbox4/creating-services", "sandbox/Sandbox4/index", "sandbox/Sandbox4/onprem-k8s", "sandbox/Sandbox4/sandbox-info", "sandbox/Sandbox5/configurations", "sandbox/Sandbox5/creating-services", "sandbox/Sandbox5/index", "sandbox/Sandbox5/onprem-k8s", "sandbox/Sandbox5/sandbox-info", "sandbox/Sandbox6/configurations", "sandbox/Sandbox6/creating-services", "sandbox/Sandbox6/index", "sandbox/Sandbox6/onprem-k8s", "sandbox/Sandbox6/sandbox-info", "sandbox/Sandbox7/configurations", "sandbox/Sandbox7/creating-services", "sandbox/Sandbox7/index", "sandbox/Sandbox7/onprem-k8s", "sandbox/Sandbox7/sandbox-info", "sandbox/Sandbox8/configurations", "sandbox/Sandbox8/creating-services", "sandbox/Sandbox8/index", "sandbox/Sandbox8/onprem-k8s", "sandbox/Sandbox8/sandbox-info", "sandbox/Sandbox9/configurations", "sandbox/Sandbox9/creating-services", "sandbox/Sandbox9/index", "sandbox/Sandbox9/onprem-k8s", "sandbox/Sandbox9/sandbox-info", "softgate-performance", "supported-networks", "switch-agent-installation", "switch-ports", "terraform-integration", "topology-management", "tutorials/activating-bgp-on-equinix-metal-project", "tutorials/adding-netris-softgate-nodes-in-equinix-metal", "tutorials/enable-services-on-equinix-metal-project", "tutorials/equinix-metal-api-integration-enablement", "tutorials/getting-started-for-equinix-metal", "tutorials/index", "tutorials/installing-netris-controller", "tutorials/upgrading-netris", "tutorials/using-l4-load-balancer", "tutorials/using-vnet-in-equinix-metal-project", "visibility", "vnet"], "filenames": ["SoftGate-PRO-installation.rst", "SoftGate-installation.rst", "accounts.rst", "acls.rst", "controller-k3s-installation.rst", "controller-k8s-installation.rst", "controller-k8s-quickstart.rst", "controller-vm-installation.rst", "definitions.rst", "index.rst", "installation.rst", "installing-netris-controller.rst", "introduction.rst", "inventory-profiles.rst", "ipam.rst", "kubernetes-integration.rst", "l3-load-balancer.rst", "l4-load-balancer.rst", "netris-architecture.rst", "network-policies.rst", "reference-designs.rst", "release-notes.rst", "roh.rst", "sandbox/Sandbox1/configurations.rst", "sandbox/Sandbox1/creating-services.rst", "sandbox/Sandbox1/index.rst", "sandbox/Sandbox1/onprem-k8s.rst", "sandbox/Sandbox1/sandbox-info.rst", "sandbox/Sandbox10/configurations.rst", "sandbox/Sandbox10/creating-services.rst", "sandbox/Sandbox10/index.rst", "sandbox/Sandbox10/onprem-k8s.rst", "sandbox/Sandbox10/sandbox-info.rst", "sandbox/Sandbox11/configurations.rst", "sandbox/Sandbox11/creating-services.rst", "sandbox/Sandbox11/index.rst", "sandbox/Sandbox11/onprem-k8s.rst", "sandbox/Sandbox11/sandbox-info.rst", "sandbox/Sandbox12/configurations.rst", "sandbox/Sandbox12/creating-services.rst", "sandbox/Sandbox12/index.rst", "sandbox/Sandbox12/onprem-k8s.rst", "sandbox/Sandbox12/sandbox-info.rst", "sandbox/Sandbox13/configurations.rst", "sandbox/Sandbox13/creating-services.rst", "sandbox/Sandbox13/index.rst", "sandbox/Sandbox13/onprem-k8s.rst", "sandbox/Sandbox13/sandbox-info.rst", "sandbox/Sandbox14/configurations.rst", "sandbox/Sandbox14/creating-services.rst", "sandbox/Sandbox14/index.rst", "sandbox/Sandbox14/onprem-k8s.rst", "sandbox/Sandbox14/sandbox-info.rst", "sandbox/Sandbox15/configurations.rst", "sandbox/Sandbox15/creating-services.rst", "sandbox/Sandbox15/index.rst", "sandbox/Sandbox15/onprem-k8s.rst", "sandbox/Sandbox15/sandbox-info.rst", "sandbox/Sandbox2/configurations.rst", "sandbox/Sandbox2/creating-services.rst", "sandbox/Sandbox2/index.rst", "sandbox/Sandbox2/onprem-k8s.rst", "sandbox/Sandbox2/sandbox-info.rst", "sandbox/Sandbox3/configurations.rst", "sandbox/Sandbox3/creating-services.rst", "sandbox/Sandbox3/index.rst", "sandbox/Sandbox3/onprem-k8s.rst", "sandbox/Sandbox3/sandbox-info.rst", "sandbox/Sandbox4/configurations.rst", "sandbox/Sandbox4/creating-services.rst", "sandbox/Sandbox4/index.rst", "sandbox/Sandbox4/onprem-k8s.rst", "sandbox/Sandbox4/sandbox-info.rst", "sandbox/Sandbox5/configurations.rst", "sandbox/Sandbox5/creating-services.rst", "sandbox/Sandbox5/index.rst", "sandbox/Sandbox5/onprem-k8s.rst", "sandbox/Sandbox5/sandbox-info.rst", "sandbox/Sandbox6/configurations.rst", "sandbox/Sandbox6/creating-services.rst", "sandbox/Sandbox6/index.rst", "sandbox/Sandbox6/onprem-k8s.rst", "sandbox/Sandbox6/sandbox-info.rst", "sandbox/Sandbox7/configurations.rst", "sandbox/Sandbox7/creating-services.rst", "sandbox/Sandbox7/index.rst", "sandbox/Sandbox7/onprem-k8s.rst", "sandbox/Sandbox7/sandbox-info.rst", "sandbox/Sandbox8/configurations.rst", "sandbox/Sandbox8/creating-services.rst", "sandbox/Sandbox8/index.rst", "sandbox/Sandbox8/onprem-k8s.rst", "sandbox/Sandbox8/sandbox-info.rst", "sandbox/Sandbox9/configurations.rst", "sandbox/Sandbox9/creating-services.rst", "sandbox/Sandbox9/index.rst", "sandbox/Sandbox9/onprem-k8s.rst", "sandbox/Sandbox9/sandbox-info.rst", "softgate-performance.rst", "supported-networks.rst", "switch-agent-installation.rst", "switch-ports.rst", "terraform-integration.rst", "topology-management.rst", "tutorials/activating-bgp-on-equinix-metal-project.rst", "tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst", "tutorials/enable-services-on-equinix-metal-project.rst", "tutorials/equinix-metal-api-integration-enablement.rst", "tutorials/getting-started-for-equinix-metal.rst", "tutorials/index.rst", "tutorials/installing-netris-controller.rst", "tutorials/upgrading-netris.rst", "tutorials/using-l4-load-balancer.rst", "tutorials/using-vnet-in-equinix-metal-project.rst", "visibility.rst", "vnet.rst"], "titles": ["SoftGate PRO Installation", "SoftGate Installation", "Accounts", "Access Control Lists (ACL)", "Netris Controller installation on a generic Linux host", "Helm Chart Installation", "Quickstart Installation", "Virtual Machine Installation", "Definitions", "Welcome to Netris Documentation", "Controller Installation", "Installing a Netris Controller", "Introduction to Netris", "Inventory Profiles", "IP Address Management", "Kubernetes Integration", "L3 Load Balancer (Anycast LB)", "L4 Load Balancer (L4LB)", "Netris Architecture", "Basic BGP", "Network Reference Designs", "Release notes", "ROH (Routing on the Host)", "Provided Example Configurations", "Learn by Creating Services", "Sandbox1", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox10", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox11", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox12", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox13", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox14", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox15", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox2", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox3", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox4", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox5", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox6", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox7", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox8", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox9", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "SoftGate Performance", "Reference Network Architectures", "Switch Agent Installation", "Switch Ports", "Terraform: Netris provider", "Inventory", "Activating BGP on Equinix Metal Project", "Provisioning Netris SoftGate nodes in Equinix Metal Project", "Enabling services (NAT, V-Net, Load Balancer, IP pools)", "Equinix Metal API integration enablement", "Equinix Metal API integration enablement", "Generic Netris Tutorials", "Installing a Netris Controller", "Netris Upgrade and Rollback Procedures", "Using on-demand (elastic) L4 Load Balancer service", "Using V-Net (isolated virtual network) services in Equinix Metal Project", "Visibility (Telescope)", "V-Net"], "terms": {"A": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115], "variabl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115], "2": [0, 14, 18, 19, 21, 22, 23, 27, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 102, 104, 105, 108, 109, 110, 113, 115], "x": [0, 18, 100], "intel": [0, 18], "xeon": [0, 98], "silver": 0, "processor": 0, "10": [0, 3, 19, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 103, 106, 111, 113], "physic": [0, 1, 7, 8, 9, 12, 15, 19, 21, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 103, 106, 115], "core": [0, 1, 4, 7, 11, 18, 20, 98, 110], "per": [0, 2, 3, 8, 13, 19, 103, 105, 108, 113, 114, 115], "socket": 0, "20": 0, "total": [0, 104], "128": [0, 39, 41, 42, 74, 76, 77], "gb": [0, 1, 4, 7, 11, 110], "64": [0, 4, 11, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 110], "ram": [0, 1, 4, 7, 11, 110, 114], "multichannel": 0, "300": [0, 1], "hdd": [0, 1], "nvidia": [0, 9], "mellanox": [0, 100], "connect": [0, 1, 6, 7, 15, 16, 17, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 102, 103, 111, 115], "5": [0, 3, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 62, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 107, 108, 111], "6": [0, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 67, 71, 76, 81, 86, 91, 96, 98, 102], "smartnic": [0, 18, 103], "card": [0, 18], "The": [0, 1, 2, 3, 4, 6, 7, 8, 11, 15, 16, 17, 18, 19, 20, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 98, 100, 101, 102, 103, 110, 111], "follow": [0, 4, 5, 6, 8, 14, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94, 98, 101, 102, 103, 111], "ar": [0, 2, 4, 6, 8, 11, 14, 15, 16, 17, 19, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 98, 101, 102, 103, 104, 105, 106, 108, 110, 111, 113, 115], "some": [0, 4, 15, 22, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 100, 101], "recommend": [0, 4, 7, 18, 19, 103, 111], "set": [0, 2, 4, 5, 7, 11, 15, 17, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 101, 102, 103, 106, 107, 108, 110, 111, 114, 115], "differ": [0, 2, 8, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 110, 114, 115], "vendor": 0, "have": [0, 1, 2, 4, 6, 8, 11, 14, 17, 18, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 100, 101, 102, 103, 106, 110, 111, 113], "name": [0, 1, 2, 3, 11, 13, 14, 15, 17, 19, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 101, 102, 103, 105, 107, 108, 110, 111, 114, 115], "so": [0, 11, 15, 18, 19, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 104, 106, 110, 113, 114], "i": [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 101, 102, 103, 105, 106, 108, 110, 111, 112, 113, 114, 115], "mainli": 0, "refer": [0, 9, 11, 18, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94, 110], "befor": [0, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 110, 111], "start": [0, 3, 7, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 103, 105, 108, 111, 114], "consid": [0, 19], "reset": 0, "all": [0, 2, 3, 4, 8, 10, 13, 15, 18, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 101, 102, 103, 111, 114], "default": [0, 1, 2, 5, 7, 8, 9, 11, 13, 15, 19, 21, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 101, 102, 103, 107, 108, 110, 113, 115], "disabl": [0, 3, 9, 19, 100, 101, 115], "power": [0, 114], "save": [0, 1, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 102, 103, 114], "option": [0, 1, 3, 6, 10, 15, 16, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 100, 101, 102, 111, 114], "perform": [0, 1, 3, 9, 12, 14, 18, 21, 22, 105, 108, 111], "tune": [0, 19], "cpu": [0, 1, 4, 7, 11, 18, 98, 110, 114], "p": [0, 4, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 111], "state": [0, 3, 15, 17, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102, 114, 115], "c3": [0, 105], "report": [0, 18], "c6": 0, "select": [0, 1, 6, 10, 13, 14, 16, 17, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 101, 103, 106, 107, 108, 114], "polici": [0, 2, 8, 12, 18, 19, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 103], "enabl": [0, 9, 15, 18, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103, 104, 113, 115], "turbo": 0, "boost": 0, "memori": 0, "frequenc": 0, "highest": 0, "avail": [0, 1, 4, 6, 15, 17, 19, 21, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94, 100, 101, 102, 114, 115], "number": [0, 2, 3, 4, 8, 15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 98, 103, 111, 114, 115], "NOT": 0, "auto": [0, 1, 7, 22, 100, 102, 115], "virtual": [0, 9, 11, 21, 101, 106, 109, 115], "when": [0, 1, 2, 3, 4, 6, 8, 15, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 100, 101, 102, 103, 105, 106, 108, 113], "you": [0, 1, 3, 4, 5, 6, 8, 11, 12, 13, 15, 17, 19, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 102, 103, 104, 105, 106, 107, 108, 110, 111, 112, 113, 114, 115], "test": [0, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 98, 102], "function": [0, 1, 10, 18, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 103, 114], "nic": [0, 7, 18], "turn": [0, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "off": [0, 18], "vt": 0, "d": [0, 1, 4, 7, 100], "hyper": 0, "thread": 0, "freshli": [0, 1, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 110], "ubuntu": [0, 1, 6, 7, 21, 105, 108], "linux": [0, 1, 6, 7, 9, 10, 11, 15, 17, 18, 22, 25, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 103, 110], "18": [0, 7, 15, 26, 27, 31, 36, 41, 42, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100], "04": [0, 1, 7, 100, 105, 108], "lt": [0, 1], "internet": [0, 1, 4, 7, 11, 18, 19, 21, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 102, 104, 106, 110, 113], "from": [0, 1, 3, 4, 6, 7, 9, 13, 14, 16, 17, 18, 19, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 101, 103, 105, 106, 107, 108, 111, 113, 114, 115], "netplan": [0, 1], "via": [0, 1, 8, 14, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 100, 101, 102, 111], "manag": [0, 1, 2, 4, 6, 7, 8, 9, 11, 15, 18, 19, 21, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 101, 102, 110, 111, 114, 115], "port": [0, 1, 2, 3, 7, 8, 9, 15, 16, 17, 19, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 102, 103, 114, 115], "add": [0, 1, 5, 6, 9, 15, 17, 19, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 100, 101, 105, 107, 108, 114, 115], "control": [0, 1, 2, 5, 6, 8, 9, 13, 19, 21, 22, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 82, 85, 90, 95, 100, 101, 102, 103, 105, 106, 107, 108, 109, 111, 113, 114, 115], "inventori": [0, 1, 3, 8, 9, 21, 100, 101, 102, 105, 108, 111], "topologi": [0, 1, 6, 9, 15, 19, 21, 24, 25, 26, 29, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 80, 81, 84, 85, 86, 89, 90, 91, 94, 95, 96, 100, 102], "section": [0, 1, 2, 8, 15, 17, 19, 21, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 101, 103, 106, 111, 112], "detail": [0, 1, 5, 6, 12, 19, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 100, 102, 110, 114], "document": [0, 1, 6, 14, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 100, 102, 112], "here": [0, 1, 4, 6, 15, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 100, 102, 107, 108], "ad": [0, 1, 2, 9, 15, 17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 101, 102, 115], "onc": [0, 1, 3, 4, 6, 11, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 101, 103, 106, 110, 111], "creat": [0, 1, 2, 3, 4, 5, 6, 9, 11, 15, 19, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 100, 101, 103, 105, 106, 107, 108, 110, 111, 113, 114, 115], "navig": [0, 1, 6, 14, 15, 16, 19, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 103, 111, 114], "click": [0, 1, 2, 6, 14, 16, 17, 19, 21, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 103, 104, 105, 106, 107, 108, 114], "three": [0, 1, 10, 19, 21, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 103, 113], "vertic": [0, 1, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100], "dot": [0, 1, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 105, 106, 108, 111], "right": [0, 1, 2, 6, 11, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 103, 110, 114], "side": [0, 1, 2, 15, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 104, 114], "newli": [0, 1, 11, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 105, 106, 108, 110, 113], "copi": [0, 1, 2, 4, 6, 18, 100, 105, 107, 108, 111], "line": [0, 1, 3, 13, 16, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 103], "your": [0, 1, 4, 6, 7, 11, 12, 13, 15, 19, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 100, 103, 104, 106, 107, 108, 110, 113], "clipboard": [0, 1, 6, 100], "run": [0, 1, 4, 6, 9, 11, 15, 18, 19, 21, 22, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 100, 102, 110, 111], "an": [0, 1, 2, 3, 6, 7, 9, 11, 12, 13, 17, 18, 19, 21, 22, 23, 24, 25, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 62, 63, 64, 65, 67, 68, 69, 70, 72, 73, 74, 75, 77, 78, 79, 80, 82, 83, 84, 85, 87, 88, 89, 90, 92, 93, 94, 95, 97, 103, 107, 108, 110, 112, 113, 115], "ordinari": [0, 1, 14], "user": [0, 1, 3, 6, 8, 9, 11, 12, 13, 14, 15, 16, 17, 18, 19, 21, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 101, 103, 105, 106, 108, 110, 114], "complet": [0, 1, 4, 6, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 111], "review": [0, 1, 3, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "ifupdown": [0, 1], "file": [0, 1, 4, 7, 9, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "verifi": [0, 1, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 111], "present": [0, 1, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 111], "correspond": [0, 1, 105, 108, 111, 113], "what": [0, 1, 11, 14, 15, 18, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 103, 107, 108, 110], "dure": [0, 1, 18, 111], "o": [0, 1, 4, 11, 15, 18, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 98, 100, 105, 108, 110], "gener": [0, 1, 3, 9, 10, 11, 13, 15, 17, 18, 19, 21, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 103, 107, 108, 110, 111], "base": [0, 1, 2, 3, 8, 12, 14, 15, 16, 19, 21, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 106], "initi": [0, 1, 21, 78, 102, 103], "If": [0, 1, 4, 6, 7, 15, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 111, 115], "same": [0, 1, 10, 11, 16, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 110], "oob": [0, 1], "network": [0, 1, 2, 3, 7, 8, 11, 12, 14, 15, 18, 19, 21, 22, 25, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 102, 103, 104, 106, 109, 110, 114, 115], "rout": [0, 1, 9, 12, 18, 21, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 82, 84, 87, 89, 92, 94, 97, 98, 100, 104, 115], "No": [0, 1, 102], "other": [0, 1, 2, 3, 8, 14, 15, 18, 19, 21, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 84, 87, 89, 92, 94, 97, 101, 103, 104, 106, 111, 113, 115], "ip": [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 13, 15, 16, 17, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 103, 104, 105, 108, 109, 110, 113], "address": [0, 1, 2, 3, 4, 6, 7, 8, 9, 11, 13, 15, 16, 17, 18, 19, 21, 22, 25, 26, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 102, 103, 104, 105, 108, 109, 110, 113, 114, 115], "should": [0, 1, 4, 6, 7, 8, 15, 16, 17, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 102, 104, 105, 106, 108, 113, 115], "host": [0, 1, 3, 5, 7, 9, 10, 11, 14, 15, 17, 19, 21, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 104, 110, 111, 113, 115], "sudo": [0, 1, 7, 100, 111], "vim": [0, 1, 7, 100], "etc": [0, 1, 7, 14, 100, 101, 102, 114], "interfac": [0, 1, 7, 11, 15, 18, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 103, 110, 111], "loopback": [0, 1, 7, 14, 16, 19, 22, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 100, 102, 103], "lo": [0, 1, 7, 100], "ifac": [0, 1, 7, 100], "inet": [0, 1, 7, 100], "ensz": [0, 1], "static": [0, 1, 7, 9, 100], "prefix": [0, 1, 7, 9, 14, 15, 22, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 84, 86, 87, 89, 91, 92, 94, 96, 97, 100, 102], "length": [0, 1, 7, 19, 100], "up": [0, 1, 11, 15, 19, 21, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 101, 102, 104, 110, 111, 114], "gatewai": [0, 1, 7, 15, 21, 25, 26, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 102, 106, 113, 115], "pleas": [0, 1, 11, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 100, 106, 110, 111, 112], "delet": [0, 5, 9, 15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 103], "thi": [0, 1, 2, 3, 4, 6, 11, 13, 14, 15, 16, 17, 19, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 101, 102, 103, 104, 105, 106, 108, 110, 111, 113, 115], "locat": [0, 1, 4, 16, 19, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 105, 107, 108], "node": [0, 1, 4, 9, 11, 15, 17, 19, 21, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 79, 81, 82, 86, 87, 91, 92, 96, 97, 103, 104, 109, 110, 113], "sourc": [0, 1, 3, 7, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 100, 102, 114], "everyth": [0, 1, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "seem": [0, 1], "ok": [0, 1, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 105, 108, 110], "remov": [0, 1, 15, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 115], "comment": [0, 1, 3], "do": [0, 1, 4, 6, 7, 14, 15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103, 107, 108], "ani": [0, 1, 2, 3, 11, 14, 15, 17, 19, 20, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102, 106, 110, 111, 112, 113, 115], "addit": [0, 1, 4, 15, 16, 19, 21, 22, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 81, 83, 86, 88, 91, 93, 96], "than": [0, 1, 18, 111], "those": [0, 1, 14], "describ": [0, 1, 3, 7, 14, 17, 19, 22, 103, 106], "exampl": [0, 1, 2, 3, 4, 6, 7, 11, 13, 15, 17, 19, 24, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59, 60, 61, 62, 64, 65, 66, 67, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 101, 102, 103, 104, 106, 110, 111, 113, 114, 115], "abov": [0, 1, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "further": [0, 1, 18, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 100, 104], "reboot": [0, 1, 7, 100, 105, 108], "server": [0, 1, 4, 6, 7, 13, 15, 16, 18, 22, 23, 24, 25, 26, 28, 29, 30, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96, 100, 102, 103, 105, 106, 108, 112, 113], "boot": [0, 1, 100], "see": [0, 1, 2, 3, 5, 7, 15, 17, 19, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 102, 103, 104, 105, 106, 108, 111, 114], "its": [0, 1, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 100, 102], "heartbeat": [0, 1, 100, 103, 105, 108], "go": [0, 1, 2, 11, 14, 15, 18, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 102, 103, 104, 105, 106, 107, 108, 110], "critic": [0, 1, 4, 100, 114], "net": [0, 1, 3, 6, 9, 14, 17, 19, 21, 22, 25, 26, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 103, 104, 105, 107, 108, 109, 111, 114], "telescop": [0, 1, 9, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 100, 111], "dashboard": [0, 1, 9, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 100], "color": [0, 1, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100], "reflect": [0, 1, 100, 111], "health": [0, 1, 12, 16, 17, 18, 100, 103, 114], "8": [1, 4, 7, 11, 15, 18, 19, 24, 26, 27, 31, 36, 41, 46, 51, 56, 61, 66, 71, 72, 76, 81, 86, 91, 96, 98, 102, 106, 110], "16": [1, 7, 15, 19, 26, 31, 36, 37, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 106], "22": [1, 26, 27, 93, 94, 97, 105, 108], "configur": [1, 6, 7, 8, 10, 12, 15, 16, 18, 19, 21, 22, 24, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59, 60, 61, 62, 64, 65, 66, 67, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 101, 103, 104, 106, 114, 115], "For": [1, 3, 6, 17, 19, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 101, 102, 103, 105, 106, 107, 108, 110, 111], "proper": [1, 6, 19, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 102], "oper": [1, 8, 12, 18, 19, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 103, 106], "bond": 1, "look": [1, 9, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "below": [1, 3, 7, 10, 11, 14, 17, 19, 22, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 103, 104, 105, 106, 107, 108, 110, 112], "pleasedelet": 1, "first": [1, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 103, 111, 114], "ensx": 1, "replac": [1, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "actual": [1, 19, 114], "0": [1, 3, 4, 6, 7, 15, 19, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 98, 102, 106, 111, 113], "second": [1, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 105, 108, 114], "ensi": 1, "bond0": 1, "slave": 1, "y": [1, 4, 100], "mode": [1, 19, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 101, 115], "activ": [1, 3, 9, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 109, 115], "backup": [1, 10, 111], "adjust": [1, 13, 103], "accord": [1, 18, 103], "desir": [1, 6, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 103], "access": [2, 4, 8, 9, 11, 15, 17, 18, 19, 20, 21, 25, 26, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 103, 106, 110, 113, 114], "descript": [2, 3, 13, 15, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 102, 103, 105, 107, 108, 114], "field": [2, 3, 6, 9, 17, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 102, 106, 107, 108, 114], "usernam": [2, 8, 15, 100], "uniqu": [2, 3, 14, 16, 17, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 113, 115], "full": [2, 18, 19, 22], "e": [2, 8, 19, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 104, 114], "mail": [2, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "email": [2, 3, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97], "also": [2, 3, 11, 15, 18, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 101, 102, 103, 105, 106, 108, 110, 114], "us": [2, 3, 4, 6, 7, 8, 9, 11, 13, 14, 16, 17, 18, 19, 22, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 100, 101, 102, 103, 104, 105, 106, 108, 109, 110, 111, 115], "system": [2, 3, 4, 7, 8, 13, 18, 19, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 103, 111, 114], "notif": [2, 3], "password": [2, 7, 8, 11, 15, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 102, 110], "retriev": 2, "cc": 2, "send": [2, 18, 19], "phone": 2, "": [2, 3, 11, 12, 15, 18, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 101, 102, 103, 106, 110, 111, 115], "compani": [2, 22], "work": [2, 4, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102], "usual": [2, 19, 103], "multi": [2, 115], "where": [2, 3, 4, 11, 17, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 103, 110, 115], "provid": [2, 3, 6, 7, 9, 10, 11, 13, 14, 15, 17, 21, 24, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59, 60, 61, 62, 64, 65, 66, 67, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 101, 103, 105, 106, 108, 114, 115], "netri": [2, 3, 5, 6, 8, 10, 13, 14, 17, 19, 20, 21, 22, 23, 24, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58, 59, 60, 63, 64, 65, 68, 69, 70, 73, 74, 75, 78, 79, 80, 83, 84, 85, 88, 89, 90, 93, 94, 95, 101, 103, 104, 106, 107, 112, 113, 114, 115], "custom": [2, 3, 9, 13, 18, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 60, 62, 64, 65, 67, 69, 70, 72, 74, 75, 77, 79, 80, 82, 84, 85, 87, 89, 90, 92, 94, 95, 97, 100, 103, 114], "posit": [2, 103], "within": [2, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94, 104], "object": [2, 3, 9, 15, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "defin": [2, 3, 7, 8, 9, 12, 13, 14, 17, 18, 21, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 84, 87, 89, 92, 94, 97, 102, 103, 107, 108], "rbac": [2, 3, 8], "deactiv": 2, "view": [2, 9, 14, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 78, 82, 87, 92, 97, 114, 115], "edit": [2, 3, 7, 15, 16, 19, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 101, 103, 106, 115], "part": [2, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 102, 103, 114], "servic": [2, 3, 7, 8, 12, 13, 14, 15, 16, 18, 19, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 103, 104, 114, 115], "switch": [2, 3, 4, 6, 8, 12, 13, 14, 15, 16, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102, 107, 108, 111, 114, 115], "resourc": [2, 3, 4, 8, 9, 19, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 101, 106], "assign": [2, 8, 17, 19, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 101, 103, 114, 115], "variou": [2, 19, 21, 98, 114], "read": [2, 106, 107, 108], "onli": [2, 3, 9, 11, 14, 18, 19, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 100, 101, 102, 104, 106, 107, 108, 110, 114, 115], "admin": [2, 3, 8, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 101, 102, 103, 106], "To": [2, 3, 4, 5, 11, 15, 16, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103, 106, 110, 111, 114], "form": [2, 18, 19, 21, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "list": [2, 8, 9, 13, 14, 15, 17, 19, 22, 25, 26, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 85, 86, 87, 90, 91, 92, 95, 96, 97, 101, 102, 103, 105, 108, 113, 114, 115], "menu": [2, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 101, 105, 106, 107, 108, 111, 114], "can": [2, 3, 4, 8, 10, 11, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 101, 102, 103, 104, 105, 106, 108, 110, 111, 112, 113, 114, 115], "under": [2, 3, 8, 16, 17, 19, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 82, 83, 87, 88, 92, 93, 97, 104, 105, 106, 107, 108, 111, 113, 114, 115], "own": [2, 8], "concept": 2, "share": [2, 8, 19, 22], "deleg": [2, 8, 106], "over": [2, 8, 11, 18, 19, 21, 22, 106, 110, 113, 115], "typic": [2, 8, 19, 103], "team": [2, 3, 8, 115], "grant": [2, 8], "request": [2, 3, 8, 15, 17, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 104, 105, 108, 109, 112], "self": [2, 4, 8, 19, 100], "portal": [2, 8, 107, 108, 112], "programmat": [2, 8], "kubernet": [2, 5, 6, 8, 9, 10, 14, 17, 18, 21, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 106, 112], "crd": [2, 8, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 112], "devop": [2, 8, 9], "netop": [2, 8, 9, 12, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "pipelin": [2, 8], "ha": [2, 7, 9, 11, 15, 18, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102, 103, 104, 110, 111, 113], "just": [2, 4, 15, 18, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 106], "two": [2, 4, 15, 19, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102, 103, 105, 106, 108, 114], "basi": [2, 3, 8, 115], "attach": [2, 8, 13, 19, 22, 103, 115], "individu": [2, 8, 11, 16, 19, 22, 107, 108, 110, 115], "everi": [2, 16, 19, 21, 22, 103, 113], "attribut": [2, 8, 19, 21, 114], "particular": [2, 15, 17, 18, 19, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 105, 108, 114, 115], "specif": [2, 3, 14, 15, 19, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "either": [2, 3, 4, 12, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 83, 84, 88, 89, 93, 94, 106, 111], "link": [2, 8, 9, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 86, 91, 96, 102, 114], "directli": [2, 15, 18, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "which": [2, 11, 14, 15, 18, 19, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 102, 103, 114], "support": [3, 4, 5, 7, 18, 19, 20, 21, 22, 100, 101, 102, 106, 114, 115], "acl2": 3, "destin": [3, 13, 16, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 103], "format": [3, 19, 115], "orient": 3, "wai": [3, 4, 11, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 110, 113], "both": [3, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 101, 104, 106], "tenant": [3, 8, 9, 14, 15, 17, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 101, 102, 103, 106, 114, 115], "execut": [3, 4, 19, 102, 111], "hardwar": [3, 9, 14, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 86, 91, 96, 98, 102, 103, 114], "rate": [3, 16], "secur": [3, 12, 13, 18, 21, 103], "enforc": 3, "It": [3, 4, 11, 14, 15, 18, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 102, 106, 110], "import": [3, 8, 9, 11, 106, 110], "keep": [3, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "mind": 3, "limit": [3, 19], "size": [3, 98], "tcam": 3, "screenshot": [3, 17, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 103, 104, 105, 106, 108, 112, 114], "util": [3, 18, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 114], "seen": [3, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "appli": [3, 4, 12, 15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "sever": [3, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "optim": [3, 22], "algorithm": 3, "minim": [3, 7, 106], "usag": [3, 14, 18, 102], "while": [3, 11, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 110, 111], "achiev": 3, "requir": [3, 9, 10, 11, 12, 14, 15, 17, 18, 19, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 100, 102, 103, 110, 115], "permit": [3, 15, 16, 19, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102], "commun": [3, 9, 18, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 103], "each": [3, 8, 11, 14, 15, 16, 18, 19, 21, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 101, 102, 103, 105, 107, 108, 110, 111, 113, 114, 115], "chang": [3, 7, 11, 15, 18, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 105, 106, 108, 110, 111, 115], "site": [3, 8, 15, 17, 18, 19, 21, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102, 103, 105, 107, 108, 113, 115], "featur": [3, 15, 19, 21, 103], "deni": [3, 13, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 103], "given": [3, 16, 17, 19, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 115], "drop": [3, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "traffic": [3, 13, 16, 18, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 103, 104, 111, 113, 114, 115], "unless": [3, 115], "through": [3, 8, 11, 17, 18, 19, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 83, 84, 87, 88, 89, 92, 93, 94, 97, 106, 110, 111, 112], "sitedefault": 3, "entri": [3, 8, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "protocol": [3, 15, 16, 17, 18, 19, 22, 25, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96], "match": [3, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "tcp": [3, 15, 16, 17, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 112], "udp": [3, 17, 112], "icmp": 3, "ipv4": [3, 8, 9, 13, 16, 22, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 103, 115], "code": 3, "icmpv6": 3, "ipv6": [3, 8, 9, 13, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 83, 87, 88, 92, 93, 97, 103, 115], "until": [3, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 105, 108], "date": [3, 4, 111], "time": [3, 6, 13, 15, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 103, 111, 114], "action": [3, 12, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 101], "forward": [3, 19, 98, 106], "packet": [3, 18, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 106, 114], "establish": [3, 15, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102, 104], "revers": 3, "except": [3, 13, 103], "syn": 3, "flag": 3, "non": [3, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103], "swap": 3, "type": [3, 4, 8, 9, 14, 17, 19, 20, 21, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 98, 102, 103, 106, 107, 108, 114], "rang": [3, 14, 19, 101, 104], "window": [3, 18, 19, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "group": [3, 8, 9], "end": [3, 16, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 115], "commonli": [3, 22], "check": [3, 15, 16, 17, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102, 103, 112, 114], "button": [3, 14, 16, 19, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 78, 84, 89, 94, 103], "anoth": [3, 18, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103], "alreadi": [3, 6, 7, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102], "3": [3, 5, 21, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 98, 100, 102, 103, 105, 108, 109, 111, 115], "24": [3, 6, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102, 113], "ssh": [3, 13, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 103, 105, 108, 111], "return": [3, 15, 18, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "show": [3, 19, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103, 114], "broader": 3, "one": [3, 11, 13, 14, 15, 17, 18, 19, 20, 21, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 103, 105, 106, 108, 110, 111, 114], "need": [3, 4, 11, 15, 16, 19, 21, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 98, 101, 103, 104, 105, 106, 107, 108, 110, 111, 113, 114, 115], "get": [3, 4, 6, 7, 10, 11, 15, 17, 19, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 102, 106, 110, 111, 113], "respons": [3, 18, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 79, 82, 87, 92, 97], "after": [3, 4, 7, 14, 15, 17, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 101, 102, 104, 111], "repres": [3, 18], "qa_ten": 3, "belong": [3, 15, 17, 19, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 86, 87, 91, 92, 96, 97, 101, 103], "stai": 3, "wait": [3, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 105, 108], "receiv": [3, 18, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "gui": [3, 7, 8, 17, 18, 19, 21, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 79, 80, 83, 88, 93], "Then": [3, 4, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 105, 106, 107, 108, 111, 113], "reject": 3, "soon": [3, 13, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 103], "agent": [3, 6, 8, 9, 21, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 103, 105, 108, 111], "push": 3, "appropri": [3, 19, 106, 111], "config": [3, 7, 21, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "throughout": 3, "fabric": [3, 16, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 102, 107, 108, 115], "rest": [3, 14, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93, 104, 107, 108], "4": [4, 7, 11, 15, 17, 18, 19, 23, 24, 26, 27, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 62, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 104, 109, 110, 111], "disk": [4, 7, 11, 110, 114], "50gb": [4, 11, 110], "bit": [4, 11, 110, 114], "k3": [4, 6, 11, 110], "expect": 4, "most": [4, 22, 106], "modern": [4, 20, 21, 22], "oss": 4, "raspbian": 4, "buster": 4, "step": [4, 5, 7, 9, 11, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102, 104, 105, 108, 110, 111], "legaci": [4, 12, 20, 22], "iptabl": 4, "alpin": 4, "setup": [4, 13, 19, 103], "red": [4, 111], "hat": 4, "cento": 4, "enterpris": 4, "command": [4, 6, 7, 11, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 101, 102, 105, 108, 110, 111], "curl": [4, 6, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 110, 111], "sfl": [4, 6, 11, 110, 111], "http": [4, 5, 6, 7, 11, 15, 17, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 102, 110, 111], "ai": [4, 6, 7, 11, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 100, 110, 111], "sh": [4, 6, 11, 110, 111], "abl": [4, 11, 15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 106, 110], "log": [4, 9, 15, 21, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "script": 4, "doe": [4, 6], "cert": [4, 7], "helm": [4, 6, 9, 10, 11, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 110, 111], "chart": [4, 6, 9, 10, 11, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 110, 114], "In": [4, 5, 11, 13, 15, 21, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 100, 103, 104, 105, 106, 108, 110, 111, 113], "order": [4, 5, 9, 15, 19, 21, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 103], "ingress": [4, 5, 115], "ctl": [4, 11, 110, 111], "hostnam": [4, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 110], "argument": [4, 102], "com": [4, 5, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 110], "sign": 4, "certif": [4, 11, 110], "out": [4, 6, 14, 15, 19, 22, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 102, 103, 111], "box": [4, 12, 14, 16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "instruct": [4, 6, 11, 15, 19, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 110], "pass": [4, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "io": [4, 5], "cluster": [4, 6, 11, 15, 21, 25, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 110], "valu": [4, 5, 14, 15, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 107, 108], "clusterissu": 4, "selfsign": 4, "letsencrypt": [4, 110], "acm": 4, "http01": 4, "challeng": 4, "valid": [4, 19, 100, 102], "proce": [4, 104], "successfulli": [4, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 111], "cname": 4, "record": [4, 11, 110], "domain": [4, 5, 11, 13, 20, 103, 110], "subdomain": 4, "exist": [4, 6, 9, 16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "prior": 4, "must": [4, 5, 14, 101, 102, 110], "simplest": 4, "issu": 4, "behind": [4, 19, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 104, 115], "fqdn": [4, 21], "public": [4, 11, 19, 21, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 103, 104, 109, 110, 114], "common": [4, 13, 14, 15, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 103, 106], "approach": [4, 12], "privat": [4, 7, 14, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 103, 106], "deploy": [4, 6, 8, 11, 21, 110], "dns01": 4, "familiar": [4, 102, 110], "specifi": [4, 14, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 115], "yaml": [4, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "suitabl": 4, "kubectl": [4, 5, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "f": [4, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 87, 91, 96], "my": [4, 5, 11, 19, 102, 110, 114], "rerun": 4, "latest": [4, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "version": [4, 15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 102, 111], "simpli": [4, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 111, 112], "newer": 4, "updat": [4, 5, 19, 21, 100, 111], "few": [4, 15, 19, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 104, 105, 106, 108], "minut": [4, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 104, 105, 108, 111], "usr": 4, "local": [4, 10, 15, 18, 19, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 84, 86, 87, 89, 91, 92, 94, 96, 97, 102], "bin": [4, 100], "store": [4, 18, 102], "data": [4, 8, 9, 18, 19, 20, 21, 22, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 102, 103, 114], "mariadb": [4, 111], "highli": [4, 18, 111], "cronjob": 4, "mysqldump": [4, 111], "take": [4, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 111], "databas": [4, 18, 111, 113], "snapshot": [4, 111], "n": [4, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "exec": [4, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93, 111], "bash": [4, 111], "c": [4, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 111], "u": [4, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 111], "mariadb_us": [4, 111], "mariadb_password": [4, 111], "mariadb_databas": [4, 111], "db": [4, 111], "m": [4, 11, 17, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 106, 110], "h": 4, "sql": [4, 111], "find": [4, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 79, 83, 88, 93, 103, 106], "yyyi": 4, "mm": 4, "dd": 4, "hh": 4, "ss": 4, "current": [4, 14, 15, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 79, 81, 86, 91, 96, 100, 103, 111, 114, 115], "directori": [4, 9, 111], "contain": [4, 6, 15, 18, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 111], "cp": [4, 111], "opt": [4, 111], "mysql": [4, 111], "root": [4, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 105, 108, 111], "mariadb_root_password": [4, 111], "1": [5, 6, 7, 15, 21, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 98, 100, 102, 109, 110, 111, 113], "12": [5, 27, 97], "pv": 5, "provision": 5, "underli": [5, 103], "infrastructur": [5, 9], "repositori": [5, 7, 100], "netrisai": [5, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "github": [5, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "namespac": [5, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "app": [5, 111], "releas": [5, 9, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 103, 115], "readm": 5, "about": [5, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 106], "paramet": [5, 19, 102, 107, 108, 115], "offer": [6, 98], "simplifi": 6, "model": [6, 19], "who": [6, 103, 115], "want": [6, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 106, 114], "quickli": [6, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "shortest": 6, "amount": 6, "streamlin": 6, "wish": 6, "instead": [6, 12, 15, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "w": [6, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "login": [6, 7, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 110], "web": [6, 11, 15, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 104, 105, 106, 107, 108, 110, 111, 112], "ui": [6, 15, 19, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 102], "credenti": [6, 7, 11, 15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 102, 110], "ipam": [6, 14, 17, 19, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 84, 87, 89, 92, 94, 97, 102, 106, 115], "new": [6, 15, 17, 19, 21, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 101, 102, 103, 104, 107, 108, 109, 113, 114, 115], "subnet": [6, 8, 9, 13, 19, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 102, 103, 106, 113], "softgat": [6, 9, 13, 14, 15, 17, 19, 21, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 81, 82, 84, 87, 89, 92, 94, 97, 102, 104, 106, 109, 111, 113], "plan": [6, 9, 103], "192": [6, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 102], "168": [6, 15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97], "100": [6, 18, 19, 34, 37, 102], "upper": 6, "fill": [6, 14, 19, 22, 103], "selector": 6, "congratul": [6, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "now": [6, 15, 21, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102, 104, 106, 107, 108, 114], "100gb": 7, "qemu": 7, "apt": 7, "virt": 7, "download": [7, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 102], "imag": [7, 19, 100], "contact": 7, "permiss": [7, 8, 9], "cd": 7, "var": 7, "lib": 7, "libvirt": 7, "wget": 7, "img": 7, "controller3": 7, "qcow2": 7, "definit": [7, 9, 103], "xml": 7, "virsh": 7, "bind": [7, 19], "br": 7, "mgmt": [7, 102], "correct": [7, 111], "bridg": [7, 115], "ifreload": [7, 100], "autostart": 7, "By": [7, 13, 22, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 102, 103, 114], "obtain": 7, "dhcp": [7, 100], "how": [7, 17, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94, 102, 104, 106], "consol": [7, 11, 100, 104, 105, 106, 107, 108, 110, 112], "newnet0p": [7, 8, 11, 100, 110], "forget": 7, "passwd": 7, "primari": [7, 100], "eth0": [7, 100], "dn": [7, 11, 13, 21, 100, 103, 110], "nameserv": [7, 100], "reload": 7, "make": [7, 15, 18, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 81, 83, 86, 88, 91, 93, 96, 102, 106, 111, 115], "sure": [7, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "browser": [7, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "nginx": [7, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "pem": 7, "kei": [7, 15, 19, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 107, 108], "restart": [7, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 111], "systemctl": [7, 101, 111], "nomenclatur": 8, "understand": [8, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "account": [8, 9, 11, 110], "restapi": [8, 18], "role": [8, 9, 14, 106], "separ": [8, 102, 106], "center": [8, 9, 18, 19, 20, 22, 103], "unit": [8, 19, 101, 103], "come": [8, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102], "preconfigur": 8, "global": [8, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 103, 113], "AS": [8, 15, 19, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 103], "acl": [8, 9, 25, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 106, 113], "mesh": [8, 15, 18, 19, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "vpn": [8, 18, 19, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "bgp": [8, 9, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 102, 106, 109, 114, 115], "extern": [8, 15, 18, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 114], "peer": [8, 9, 15, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 104], "ibgp": 8, "ebgp": 8, "automat": [9, 12, 15, 17, 18, 19, 21, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 100, 101, 102, 103, 104, 106, 114, 115], "platform": [9, 101, 102, 113], "like": [9, 11, 12, 14, 15, 19, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103, 104, 106, 110], "experi": [9, 14, 15], "engin": [9, 18], "upgrad": [9, 10, 105, 108, 109], "rollback": [9, 109], "procedur": [9, 109], "vpc": [9, 104, 106], "equinix": [9, 106], "metal": [9, 22, 106], "api": [9, 15, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 109, 113], "integr": [9, 17, 18, 21, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 109], "provis": [9, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 100, 102, 106, 108, 109, 113], "project": [9, 106, 107, 108], "nat": [9, 14, 18, 21, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 103, 104, 109, 113], "v": [9, 14, 19, 21, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 101, 109, 114], "load": [9, 12, 14, 15, 18, 19, 21, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 104, 109, 113, 114], "balanc": [9, 12, 14, 15, 18, 19, 21, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 104, 109, 114], "pool": [9, 14, 17, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 103, 104, 109], "isol": [9, 109], "demand": [9, 15, 17, 21, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 109], "elast": [9, 109], "l4": [9, 15, 21, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 109], "method": [9, 19, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 112, 114], "regular": [9, 18, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "manifest": [9, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "loadbalanc": [9, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "introduct": 9, "l4lb": [9, 14, 21], "reclaim": 9, "calico": [9, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "cni": [9, 21, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "terraform": [9, 106, 112], "prepar": 9, "architectur": [9, 19, 22], "unmanag": [9, 20], "scalabl": 9, "prerequisit": 9, "minimum": 9, "pro": [9, 105, 108], "bio": 9, "alloc": [9, 17, 19, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97, 102], "profil": [9, 21, 22], "hairpin": [9, 21], "cumulu": [9, 101], "basic": [9, 102, 106], "advanc": [9, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "map": 9, "rule": [9, 13, 21, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 79, 83, 84, 88, 89, 93, 94, 98, 103, 106], "sitemesh": 9, "glass": [9, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94], "l3": [9, 21, 22, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 82, 85, 87, 90, 92, 95, 97, 102, 103], "anycast": [9, 21, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 82, 85, 87, 90, 92, 95, 97], "lb": [9, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 114], "consum": [9, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106, 112], "approv": [9, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "workflow": 9, "process": [9, 11, 15, 18, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 103, 110, 111], "roh": [9, 14, 16, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 82, 84, 87, 89, 92, 94, 97, 103], "visibl": [9, 19], "graph": 9, "board": 9, "note": [9, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 101, 106, 111, 113], "vm": [10, 18, 19, 22, 106, 113], "deploi": [10, 11, 14, 15, 24, 25, 27, 29, 30, 32, 34, 35, 37, 39, 40, 42, 44, 45, 47, 49, 50, 52, 54, 55, 57, 59, 60, 62, 64, 65, 67, 69, 70, 72, 74, 75, 77, 80, 82, 84, 85, 87, 89, 90, 92, 94, 95, 97, 102, 110], "applic": [10, 15, 16, 18, 21, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 103, 111], "cloud": [10, 12, 14, 15, 17, 18, 21, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 112], "move": [10, 11, 19, 103, 104, 110], "prem": [10, 14, 18, 21], "anytim": 10, "ONE": 10, "uninstal": [10, 100], "restor": [10, 111], "repo": 10, "info": [10, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 104], "almost": [11, 110], "mai": [11, 15, 22, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 106, 110], "fact": [11, 110], "multipl": [11, 16, 17, 18, 101, 110, 113, 114, 115], "doesn": [11, 22, 103, 110], "t": [11, 14, 15, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 103, 106, 110, 115], "matter": [11, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 103, 110], "am": 11, "aw": 11, "machin": [11, 19, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 110], "ec2": 11, "got": [11, 21, 110], "54": [11, 110], "219": [11, 110], "211": [11, 110], "71": [11, 110], "easier": [11, 110], "potenti": [11, 103, 110, 111], "somewher": [11, 110], "cloudflar": [11, 110], "dev": [11, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 100, 110], "point": [11, 19, 102, 105, 106, 108, 110, 113], "ensur": [11, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 106, 110, 111], "inde": [11, 110], "resolv": [11, 100, 110], "liner": [11, 105, 108, 110, 111], "stand": [11, 110], "top": [11, 12, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 110, 114], "let": [11, 12, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102, 110], "encrypt": [11, 18, 19, 21, 110], "ssl": [11, 110], "That": [11, 106, 110, 113], "why": [11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 104, 110], "finish": [11, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 105, 108, 110, 111], "consonl": 11, "immedi": [11, 110], "someth": [11, 106, 110], "strong": [11, 110], "whitelist": [11, 110], "restrict": [11, 110], "softwar": [12, 18, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "continu": [12, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 100], "monitor": [12, 103, 114], "remedi": [12, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "inform": [12, 18, 19, 102], "necessari": [12, 18, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 83, 84, 88, 89, 93, 94], "human": 12, "intervent": 12, "abstract": 12, "awai": 12, "complex": [12, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "effici": 12, "down": [12, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 101, 114], "allow": [13, 14, 15, 19, 22, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 78, 79, 81, 84, 86, 89, 91, 94, 96, 103, 115], "harden": [13, 103], "devic": [13, 18, 19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 101, 103, 111], "flow": [13, 103], "As": [13, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103], "includ": [13, 14, 17, 19, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 79, 82, 87, 92, 97, 103, 113], "ntp": [13, 21, 103], "free": [13, 15, 19, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 101, 103, 104], "text": [13, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 103], "timezon": [13, 21, 103], "track": 14, "design": [14, 15, 18, 21, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "tree": [14, 15, 22], "opportun": 14, "kind": [14, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "purpos": [14, 17, 19, 21, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 102, 103, 106], "There": [14, 15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 102, 103], "main": [14, 18, 21, 102, 103], "organ": [14, 19, 21, 115], "rir": 14, "lir": 14, "alwai": [14, 19, 106], "child": 14, "parent": 14, "bottom": [14, 16, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94], "overlap": 14, "owner": [14, 15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 101, 103, 115], "ust": 14, "band": [14, 19, 103], "itself": 14, "inact": 14, "reserv": [14, 18, 21], "futur": [14, 105, 106, 108, 115], "kube": [15, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "complement": [15, 21], "between": [15, 16, 18, 19, 21, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 103, 104, 106, 111, 113, 115], "accomplish": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "master": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "v1": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "secret": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "nnetri": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "cred": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "liter": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "inspect": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "pod": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "l": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "output": [15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 111], "demonstr": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96], "success": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 102, 103], "level": [15, 18, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 114], "1629994653": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "6441543": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "logger": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "msg": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "worker": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "reconcilergroup": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "k8": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97], "reconcilerkind": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "count": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 98, 103, 114], "plane": [15, 18, 21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 115], "scenario": [15, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 113], "we": [15, 18, 19, 21, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 102, 103, 104, 105, 106, 108, 113], "simpl": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96], "podinfo": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "k": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "stefanprodan": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "kustom": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "po": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "svc": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "clusterip": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "readi": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 111], "statu": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 103, 111], "ag": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "576d5bf6bd": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "7z9jl": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "49": [15, 26, 31, 36, 41, 46, 51, 56, 61, 64, 66, 71, 76, 81, 86, 91, 96], "nhlmh": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "33": [15, 26, 31, 36, 41, 46, 51, 56, 59, 61, 66, 71, 76, 81, 86, 91, 96], "172": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 87, 91, 96, 102], "21": [15, 23, 26, 27, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 81, 83, 86, 88, 91, 93, 96, 98], "65": [15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 67, 71, 76, 81, 86, 91, 96], "106": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 92, 96], "none": [15, 17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "9898": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "9999": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "50": [15, 24, 26, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 59, 61, 64, 66, 69, 71, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97], "outsid": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 113, 115], "patch": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "spec": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "again": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "pend": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "32584": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "30365": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "8m57": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "real": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 103], "becaus": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "xxxxxxxx": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "been": [15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 101, 103, 111], "inject": [15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "117": [15, 24, 29, 31, 32, 33, 34, 36, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 59, 64, 69, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97], "59": [15, 24, 29, 31, 32, 59, 64, 69, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97], "202": 15, "9m17": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "creation": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "previou": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 113], "detect": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "them": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 106, 114], "frontend": [15, 17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 114], "66d44feb": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "0278": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "412a": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "a32d": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "73afe011f2c6": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "nyc": [15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97], "33m": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "32m": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "recreat": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "wa": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96], "origin": [15, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96], "being": [15, 17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 111], "nativ": [15, 17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 112], "schema": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "our": [15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 112], "backend": [15, 17, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 114], "srv04": [15, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97], "srv05": [15, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97], "80": [15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 69, 71, 72, 76, 81, 86, 91, 96], "These": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "index": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "html": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96], "cat": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "eof": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "apivers": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "v1alpha1": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "metadata": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 104, 113], "ownerten": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "45": [15, 23, 24, 26, 27, 28, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97], "46": [15, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97], "timeout": [15, 17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "3000": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "And": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102, 106], "d07acd0f": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "51ea": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "429a": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "89dd": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "8e4c1d6d0a86": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "2m17": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "3m47": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "203": [15, 102], "try": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "displai": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 102], "l2": [15, 20, 103], "segment": [15, 22, 115], "vnet": [15, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 102], "guestten": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "switchport": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "swp2": [15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102], "sw22": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "7": [15, 67, 81, 98, 100], "incom": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "isp2": [15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "softgate2": [15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "neighbora": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "65007": [15, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97], "transport": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "swp14": [15, 79, 81], "sw02": [15, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 78, 79, 81, 84, 86, 87, 89, 91, 92, 94, 96, 97], "vlanid": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "1092": [15, 94, 96, 97], "localip": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "118": [15, 34, 36, 37, 96, 97], "30": [15, 22, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 102, 106], "remoteip": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "prefixlistoutbound": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "28": [15, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 106], "le": [15, 19, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 84, 86, 87, 89, 91, 92, 94, 96, 97, 102], "32": [15, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 84, 86, 87, 89, 91, 92, 94, 96, 97, 102], "session": [15, 16, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 104, 111, 114], "neighbor": [15, 19, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97], "remot": [15, 19, 24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 84, 86, 87, 89, 91, 92, 94, 96, 97, 113, 115], "15": [15, 26, 31, 32, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 98], "similar": [15, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 111], "00": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "51": [15, 19, 81, 102], "2m3": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "feel": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97], "annot": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "previous": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 111], "true": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "otherwis": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 115], "complain": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "thei": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 102, 104, 113, 115], "won": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "put": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96], "back": [15, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "declar": [15, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "suppos": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 106], "prevent": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "reclaimpolici": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "retain": [15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "Or": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "ll": [15, 22, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 115], "leaf": [15, 16, 20, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97], "tor": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "clean": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "doc": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102, 106, 110], "veri": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "bgpconfigur": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "nodetonodemeshen": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "asnumb": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "64512": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "projectcalico": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "org": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "logseverityscreen": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "06": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "7m59": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "sandbox9": [15, 93, 94, 96, 97], "srv06": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "110": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97], "66": [15, 19, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 67, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "4200070000": [15, 81], "26": [15, 26, 31, 36, 41, 46, 51, 56, 61, 62, 66, 71, 76, 81, 86, 91, 96, 102], "srv07": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "67": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "4200070001": [15, 81], "srv08": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "68": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "4200070002": [15, 81], "might": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "notic": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 105, 108], "07": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "48": [15, 26, 31, 36, 41, 46, 51, 56, 61, 64, 66, 67, 71, 76, 81, 86, 91, 96], "8m41": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "44": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 86, 87, 91, 92, 96, 97], "19": [15, 26, 28, 29, 31, 32, 36, 41, 42, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 98], "still": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 111], "mean": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 113], "fals": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "final": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 102], "earlier": [15, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96], "ye": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "mfpdt": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "revis": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "34577c": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "logo": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "raw": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "githubusercont": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "gh": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "page": [15, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "cuddle_clap": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "gif": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "messag": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "greet": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "v6": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "goo": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "goarch": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "amd64": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "runtim": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "go1": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "num_goroutin": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "num_cpu": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "leverag": [16, 17, 22], "ecmp": [16, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "hash": [16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "capabl": [16, 22, 106], "spine": [16, 20, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 78, 82, 87, 92, 97, 103], "deliv": 16, "besid": [16, 114], "advertis": [16, 19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 82, 84, 87, 89, 92, 94, 97, 104, 106], "unicast": 16, "toward": [16, 19, 22, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "pair": [16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "lifetim": [16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "identifi": [16, 113], "failur": [16, 103], "rerout": 16, "case": [16, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 100, 106, 111, 115], "outag": 16, "instanc": [16, 22, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "ellips": 16, "extra": 16, "router": [17, 18, 19, 22, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 83, 84, 87, 88, 89, 92, 93, 94, 97, 104], "layer": [17, 18, 22, 113, 115], "least": 17, "well": [17, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 112, 113], "guid": 17, "kubenet": [17, 21], "requestor": 17, "space": [17, 18, 79], "administr": [17, 19], "expos": [17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "next": [17, 19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 100, 102, 103, 104, 105, 106, 108], "altern": [17, 19], "manual": [17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 101, 103], "probe": 17, "uncondition": 17, "millisecond": 17, "path": [17, 19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "compos": 18, "element": [18, 102], "statist": [18, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 114], "analyt": 18, "modul": 18, "diagram": [18, 24, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 80, 84, 85, 89, 90, 94, 95], "high": [18, 19, 20, 22, 105, 108, 115], "more": [18, 22, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 102, 106, 110], "replic": [18, 106], "singl": [18, 20, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 114, 115], "unreach": [18, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "toler": [18, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "stat": 18, "collect": [18, 111], "unavail": [18, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 111], "howev": 18, "affect": [18, 111], "NOS": [18, 100, 103], "grpc": [18, 111], "border": [18, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 104], "translat": [18, 19, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "x86": [18, 105], "dpdk": [18, 21, 105, 108], "enter": [18, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 102], "bypass": 18, "kernel": [18, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "travel": 18, "pcie": 18, "bu": 18, "closest": [18, 19], "last": 18, "cach": [18, 19], "header": 18, "rewrit": 18, "mac": [18, 19, 103, 114], "vlan": [18, 19, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 101, 113, 115], "id": [18, 19, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 84, 87, 89, 92, 94, 97, 101, 102, 107, 108, 113, 115], "One": 18, "frr": [18, 19, 21, 22, 104], "standard": [18, 20, 106], "wireguard": [18, 19, 21], "tunnel": [18, 19, 21], "dynam": [18, 19, 21, 103], "program": 19, "meet": 19, "tabl": [19, 22], "termin": [19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 102], "autonom": 19, "without": [19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 114], "upstream": [19, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 80, 83, 85, 88, 90, 93, 95, 103], "cabl": [19, 21, 102, 103], "ixp": 19, "ggc": 19, "googl": 19, "tag": [19, 105, 106, 108, 115], "untag": [19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 115], "example2": 19, "png": 19, "align": 19, "class": 19, "shadow": 19, "fine": 19, "expand": [19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 115], "aka": 19, "multihop": 19, "choos": 19, "speaker": 19, "allowa": 19, "occurr": 19, "nlri": 19, "normal": [19, 114], "inbound": [19, 22, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 84, 87, 89, 92, 94, 97, 104], "max": [19, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "exce": 19, "maximum": [19, 101], "1000": 19, "handl": [19, 104, 106], "million": 19, "outbound": [19, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 84, 87, 89, 92, 94, 97, 106], "prefer": 19, "weight": 19, "prepend": 19, "mani": [19, 22, 102, 115], "referenc": 19, "extend": [19, 101], "larg": [19, 106], "consist": 19, "possibl": [19, 106, 115], "mandatori": [19, 103], "len": 19, "ge": 19, "keyword": 19, "string": [19, 100], "aa": 19, "nn": 19, "65535": 19, "known": [19, 22], "export": [19, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "associ": [19, 106], "sequenc": 19, "drag": [19, 103], "claus": 19, "hop": 19, "med": 19, "met": 19, "whether": [19, 102], "manipul": 19, "isp": [19, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 102, 114], "situat": [19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "dual": [19, 115], "home": [19, 115], "temporari": 19, "interconnect": [19, 21, 103], "old": [19, 22, 115], "migrat": 19, "hypervisor": [19, 22], "intern": [19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 79, 83, 84, 88, 89, 93, 94], "scope": 19, "null": 19, "254": [19, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 102, 113], "leaf1": 19, "spine1": 19, "snat": [19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 79, 83, 84, 88, 89, 93, 94, 98, 103, 106], "dnat": [19, 98, 103, 106], "roll": [19, 103], "larger": [19, 105], "scale": 19, "carrier": 19, "grade": 19, "overload": 19, "198": [19, 102], "1080": 19, "particip": [19, 21, 115], "henc": 19, "learn": [19, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 102], "reach": [19, 24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 82, 84, 87, 89, 92, 94, 97, 104, 111], "hub": 19, "carri": 19, "transit": 19, "major": [19, 20], "spoke": 19, "small": [19, 105], "offic": 19, "maintain": 19, "measur": 19, "underneath": 19, "mathemat": 19, "randomli": 19, "converg": 19, "status": [19, 103, 114], "tool": [19, 21, 112], "perspect": 19, "left": [19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 106, 114], "dropdown": [19, 101, 106, 107, 108, 114], "famili": [19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "summari": [19, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 114], "adjac": [19, 22, 78], "queri": 19, "lookup": 19, "rib": 19, "tracerout": 19, "conduct": 19, "determin": 19, "ping": [19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "evpn": [19, 115], "brief": 19, "vni": 19, "distinguish": 19, "fall": 20, "four": [20, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "pattern": 20, "collaps": 20, "throughput": 20, "redund": 20, "distribut": [20, 21], "higher": [21, 22, 103], "27mpp": [21, 98], "100gbp": 21, "12mpp": [21, 98], "built": [21, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "andoth": 21, "comprehens": [21, 114], "call": [21, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 114], "sent": [21, 114], "abil": [21, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 114], "search": [21, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 114], "sort": [21, 114], "column": [21, 114], "filter": [21, 114], "andautomat": 21, "switchdev": 21, "loop": [21, 103], "entir": [21, 106, 114, 115], "On": [21, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 82, 85, 90, 95], "kvm": [21, 113], "construct": 21, "instal": [21, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 105, 108, 109, 111], "easi": 21, "authent": 21, "improv": 21, "becom": [21, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "place": [21, 103], "hide": 21, "comfort": 21, "resili": 22, "ecosystem": 22, "daemon": [22, 101, 104], "bare": [22, 106, 113], "even": [22, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 114], "suit": 22, "don": [22, 100, 106], "ethernet": [22, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "span": [22, 115], "stp": 22, "miniz": 22, "reliabl": 22, "increas": 22, "interim": 22, "proxmox": 22, "Will": 22, "aggreg": [22, 101], "inherit": 22, "zero": [22, 100], "msft": 22, "anywher": [22, 115], "sometim": 22, "autodetect": [22, 115], "1gbp": [22, 115], "10gpb": [22, 115], "speed": [22, 101, 115], "bulk": [22, 101, 115], "certain": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 101], "pre": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97], "explor": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "interact": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 102], "yourself": [23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96], "visit": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "sandbox1": [23, 24, 26, 27], "examin": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "indic": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "sw21": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "2607": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 83, 87, 88, 92, 93, 97], "f358": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 83, 87, 88, 92, 93, 97], "11": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 83, 87, 88, 92, 93, 97, 98], "ffc1": [23, 27], "respect": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "properli": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "remain": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "unchang": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "label": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "submit": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94], "result": [23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 98], "thank": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "vrf": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "vrf_netri": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "56": [23, 28, 33, 38, 43, 48, 53, 58, 63, 64, 67, 68, 73, 78, 83, 88, 93], "84": [23, 28, 33, 38, 43, 44, 47, 48, 53, 58, 63, 68, 69, 72, 73, 78, 83, 88, 93], "byte": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "icmp_seq": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "ttl": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "562": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "745": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "690": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "737": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "666": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "transmit": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "loss": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "4092m": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "rtt": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "min": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "avg": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "mdev": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "680": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "065": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "interest": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "found": [23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 111], "asid": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "softgate1": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 105, 108], "iri": [23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97], "isp1": [23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97], "sandbox": [23, 24, 25, 26, 28, 29, 30, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96], "fault": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "observ": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 83, 84, 88, 89, 93, 94], "g": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 79, 83, 84, 88, 89, 93, 94], "demo": [23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97], "166": [23, 24, 27, 28, 29, 32, 44, 46, 47, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97], "88": [23, 24, 27, 28, 29, 32, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97], "17": [23, 24, 27, 28, 29, 32, 37, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 98], "30064": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97], "introductori": [23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96], "ping4": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 83, 88, 93], "repli": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "62": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 66, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "wan": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "impos": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "granular": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 106, 113], "implement": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "would": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 115], "short": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "exercis": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "conjunct": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "intellig": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "automag": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "across": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 114], "give": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 106], "30065": [24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 82, 84, 87, 89, 92, 94, 97], "eth1": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "proto": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "onlink": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "fulli": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "reachabl": [24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 103], "judg": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "corner": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 114], "mark": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 114], "wire": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "55": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "world": [24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 82, 84, 87, 89, 92, 94, 97], "swp16": [24, 26, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 84, 86, 87, 89, 91, 92, 94, 96, 97, 102], "easili": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "uncheck": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "1012": [24, 26, 27], "126": [24, 29, 31, 32, 39, 41, 42, 59, 64, 69, 74, 84, 89, 94], "125": [24, 29, 31, 32, 39, 41, 42, 59, 64, 69, 74, 84, 89, 94], "38": [24, 26, 27, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72], "161": [24, 26, 27, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 84], "green": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 111], "glean": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "insight": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "leav": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 107, 108], "face": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "aim": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "toggl": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 101], "23": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 98], "were": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "uc": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "longer": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "pop": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "sync": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 101, 103], "member": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 114], "resum": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94], "srv01": [24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 82, 84, 87, 89, 92, 94, 97], "srv02": [24, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 82, 84, 87, 89, 92, 94, 97], "webpag": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "appear": [24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 101], "216": [24, 29, 32, 59, 64, 69, 74, 84, 89, 94], "repeat": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 103, 105, 108], "had": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "tab": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "bar": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "calcul": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "factor": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "websit": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "land": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "trigger": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "direct": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "could": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "around": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "simul": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "mainten": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "vip": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94, 114], "name_50": [24, 29, 59, 64, 69, 74, 84, 89, 94], "opposit": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "al3lb": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "content": [25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "welcom": [25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "vxlan": [25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 115], "exterior": [25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "l3lb": [25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 85, 90, 95], "intro": [25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95], "environ": [26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 102], "kubesprai": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "dedic": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 103], "apiserv": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "kubeconfig": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "ve": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "6443": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "debug": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "diagnos": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "problem": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "dump": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "13": [26, 97, 98], "rememb": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "develop": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "decid": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "json": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "op": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "ic": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "coffe": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "14": [26, 32], "shown": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "didn": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "behavior": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "round": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "robin": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "intermitt": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "went": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "btw": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "prefixlistinbound": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 86, 91, 96, 102], "160": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 84, 86, 87, 91, 96], "01": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 86, 91, 96, 102, 113], "27": [26, 31, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 61, 66, 71, 76, 86, 91, 96], "4230000000": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 86, 91, 96], "4230000001": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 86, 91, 96], "4230000002": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 86, 91, 96], "beer": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "depend": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 115], "question": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "slack": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 111], "five": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "integrat": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "routabl": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "srv03": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "30061": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "30062": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "30063": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "softage1": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "sw01": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 78, 87, 92, 97], "1011": 27, "ffc0": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "127": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97, 102], "softage2": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "___": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "sandbox10": [28, 29, 31, 32], "ffca": [28, 32], "1102": [29, 31, 32], "208": [29, 31, 32], "209": [29, 57], "212": [29, 32], "221": 31, "222": 31, "1101": 32, "122": [32, 42], "121": [32, 42], "220": 32, "sandbox11": [33, 34, 36, 37], "ffcb": [33, 37], "82": [33, 34, 37, 77], "1112": [34, 36, 37], "96": [34, 36, 37], "97": [34, 87], "104": [34, 37], "name_45": [34, 39, 44, 49, 54], "109": [36, 91, 92], "1111": 37, "114": [37, 97], "113": [37, 97, 102], "108": 37, "sandbox12": [38, 39, 41, 42], "ffcc": [38, 42], "83": [38, 39, 42], "1122": [39, 41, 42], "129": [39, 74], "132": [39, 42, 74, 77], "136": [39, 42, 74, 77], "141": [41, 76], "142": [41, 76], "1121": 42, "140": [42, 77], "sandbox13": [43, 44, 46, 47], "ffcd": [43, 47], "1132": [44, 46, 47], "165": [44, 46, 47], "144": [44, 46, 47, 79, 81, 82], "145": 44, "148": [44, 47], "152": [44, 47], "157": 46, "158": 46, "1131": 47, "162": 47, "1b": 47, "1a": 47, "156": 47, "sandbox14": [48, 49, 52], "ffce": [48, 52], "85": [48, 49, 52, 76, 77], "1142": [49, 51, 52], "174": [49, 51, 52, 86], "173": [49, 51, 52, 86], "176": [49, 51, 52, 89, 91, 92], "177": [49, 89], "180": [49, 52, 89, 92], "184": [49, 52, 89, 92], "sandbox15": [51, 53, 54, 56, 57], "189": [51, 63, 64, 67, 91], "190": [51, 58, 59, 62, 91], "1141": 52, "170": 52, "169": 52, "1d": 52, "1c": 52, "188": [52, 68, 69, 72, 92], "ffcf": [53, 57], "86": [53, 54, 57, 76, 77], "1152": [54, 56, 57], "214": [54, 56, 57], "213": [54, 56, 57], "193": [54, 94], "196": [54, 57, 94, 97], "200": [54, 57, 94, 97], "205": [56, 96], "206": [56, 96], "1151": 57, "210": 57, "1f": 57, "1e": 57, "204": [57, 97], "sandbox2": [58, 59, 61, 62], "ffc2": [58, 62], "1022": [59, 61, 62], "36": [59, 62], "40": [59, 62], "29": [61, 62, 88, 89, 92], "1021": 62, "25": [62, 102], "sandbox3": [63, 64, 66, 67], "ffc3": [63, 67], "1032": [64, 66, 67], "52": [64, 67], "61": 66, "70": [66, 67], "69": [66, 67], "1031": 67, "60": 67, "sandbox4": [68, 69, 71, 72], "ffc4": [68, 72], "1042": [69, 71, 72], "81": [69, 77], "93": [71, 79, 81, 82], "94": [71, 79, 81, 82], "78": [71, 72], "77": [71, 72], "1041": 72, "74": 72, "73": 72, "9": 72, "92": 72, "sandbox5": [73, 74, 76, 77], "ffc5": [73, 77], "187": [73, 74, 77], "1052": [74, 76, 77], "1051": 77, "b": 77, "sandbox6": [78, 79, 81, 82], "radio": 78, "row": 78, "choic": 78, "543": 78, "436": 78, "445": 78, "354": 78, "345": 78, "4074m": 78, "424": 78, "075": 78, "insid": [78, 104, 115], "screen": 78, "pubblic": 78, "186": [78, 79, 82], "disallow": 78, "nfv": [79, 103], "unselect": 79, "1062": [79, 81, 82], "150": 79, "durat": 79, "v0": 81, "154": 81, "155": 81, "1061": 82, "90": 82, "89": 82, "sandbox7": [83, 84, 86, 87], "ffc7": [83, 87], "185": [83, 84, 87], "1072": [84, 86, 87], "164": [84, 87], "102": [86, 87], "101": [86, 87], "1071": 87, "98": 87, "sandbox8": [88, 89, 91, 92], "ffc8": [88, 92], "1082": [89, 91, 92], "1081": 92, "105": 92, "ffc9": [93, 97], "1091": 97, "help": 98, "mpp": 98, "unmatch": 98, "e5": 98, "2660": 98, "fw": 98, "schedul": 98, "kni": 98, "17mpp": 98, "7mpp": 98, "9mpp": 98, "2mpp": 98, "gold": 98, "6130": 98, "odd": 98, "31mpp": 98, "22mpp": 98, "15mpp": 98, "11mpp": 98, "3mpp": 98, "3m": 98, "conntrack": 98, "20k": 98, "38mpp": 98, "28mpp": 98, "19mpp": 98, "fresh": 100, "cl": 100, "past": [100, 105, 107, 108, 111], "press": [100, 102], "ctrl": 100, "grub": 100, "oni": 100, "stop": [100, 111], "discoveri": 100, "addr": 100, "echo": 100, "conf": 100, "x86_64": 100, "mlnx_x86": 100, "r0": 100, "nos": [100, 102], "sonic_20211125_074752_ec202012_227": 100, "yourpassword": 100, "touch": 100, "ztp": 100, "patient": 100, "whom": 101, "breakout": 101, "split": 101, "switchd": 101, "mtu": 101, "transmiss": 101, "autoneg": 101, "autonegoti": 101, "duplex": 101, "extens": [101, 102], "quick": 101, "detach": 101, "hand": 102, "tutori": 102, "hashicorp": 102, "tf": 102, "At": [102, 105, 106, 108], "begin": 102, "required_provid": 102, "netris_address": 102, "netris_login": 102, "netris_password": 102, "init": 102, "folder": 102, "properti": 102, "Such": 102, "netris_sit": 102, "re": 102, "clear": 102, "tenantid": 102, "netris_ten": 102, "netris_alloc": 102, "netris_subnet": 102, "defaultgatewai": 102, "siteid": 102, "depends_on": 102, "With": [102, 103], "netris_softg": 102, "mainip": 102, "mgmtip": 102, "netris_switch": 102, "cumulus_linux": 102, "portcount": 102, "netris_link": 102, "sg": [102, 111], "sw": [102, 111], "swp1": 102, "netris_vnet": 102, "1050": 102, "swp3": 102, "10th": 102, "netris_port": 102, "swp10_my_switch": 102, "swp10": 102, "netris_bgp": 102, "23456": 102, "portid": 102, "done": [102, 105, 106, 108], "error": [102, 114], "confirm": 102, "progress": 102, "destroi": 102, "resid": 103, "preliminari": 103, "regist": 103, "administ": [103, 106], "chapter": [103, 104, 106, 113], "fail": 103, "watch": 103, "against": 103, "long": 103, "capac": 103, "lift": 103, "v2": 103, "know": [103, 104], "visual": [103, 114], "scene": [104, 115], "pull": 104, "bring": [104, 105, 108], "happen": [105, 108], "acceler": [105, 108], "signal": [105, 106, 108, 114], "crit": [105, 108], "sens": [105, 106, 108], "heart": [105, 108], "beat": [105, 108], "too": [105, 108], "although": 106, "intend": [106, 115], "crush": 106, "smaller": 106, "worri": 106, "tenanc": 106, "word": 106, "colleagu": 106, "later": [106, 114], "repurpos": 106, "deal": [107, 108], "write": [107, 108], "smallest": 108, "flavor": 108, "block": 109, "issuer": 110, "hint": 110, "due": 111, "structur": 111, "unlik": 111, "event": 111, "sshing": 111, "check_ag": 111, "impact": 111, "product": 111, "afterward": 111, "svclb": 111, "haproxi": 111, "6tkgj": 111, "38d": 111, "bcb944b7c": 111, "qcbf8": 111, "13d": 111, "squid": 111, "7f6fdc6cf9": 111, "7fdx8": 111, "58rnp": 111, "graphit": 111, "mongodb": 111, "redi": 111, "smtp": 111, "76778cf85f": 111, "lw5v5": 111, "10d": 111, "8b9dbbcd8": 111, "8snhd": 111, "notifi": 111, "647975848f": 111, "fs5dn": 111, "b9b8d8f8d": 111, "4ssqb": 111, "987669fb9": 111, "jjskp": 111, "777c98c5d9": 111, "mqwl6": 111, "lqmq7": 111, "20h": 111, "introduc": 111, "advers": 111, "taken": 111, "downgrad": 111, "031": 111, "meanwhil": 112, "vmware": 113, "comput": 113, "02": 113, "03": 113, "launch": 113, "subinterfac": 113, "serv": 113, "sinc": 113, "live": 113, "sum": 114, "open": 114, "titl": 114, "bp": 114, "pp": 114, "optic": 114, "histori": 114, "iris1": 114, "iris2": 114, "explan": 114, "summar": 114, "icon": 114, "pie": 114, "suppli": 114, "fan": 114, "temperatur": 114, "sensor": 114, "synchron": 114, "alarm": 114, "unrout": 115, "collabor": 115, "unnumb": 115, "underlai": 115, "awar": 115, "rare": 115, "guest": 115, "circuit": 115, "backbon": 115, "alia": 115, "swp": 115, "signific": 115, "egress": 115, "lag": 115, "standbi": 115, "assum": 115, "svi": 115, "nose": 115}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"softgat": [0, 1, 18, 98, 99, 103, 105, 108], "pro": 0, "instal": [0, 1, 4, 5, 6, 7, 9, 10, 11, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 100, 102, 110], "minimum": [0, 1], "hardwar": [0, 1], "requir": [0, 1, 4, 5, 7], "bio": 0, "configur": [0, 5, 9, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 100, 102], "netri": [0, 1, 4, 7, 9, 11, 12, 15, 18, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 99, 100, 102, 105, 108, 109, 110, 111], "agent": [0, 1, 18, 100], "account": 2, "user": 2, "tenant": 2, "permiss": 2, "group": 2, "role": 2, "access": [3, 7, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "control": [3, 4, 7, 10, 11, 15, 18, 23, 24, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 110], "list": [3, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "acl": [3, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "default": 3, "polici": [3, 9, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "rule": [3, 19], "approv": 3, "workflow": 3, "process": [3, 6], "order": 3, "gener": [4, 109], "linux": [4, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 100], "host": [4, 22], "specif": 4, "name": 4, "let": 4, "": 4, "encrypt": 4, "ssl": [4, 7], "custom": [4, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "issuer": 4, "upgrad": [4, 111], "uninstal": [4, 5], "backup": 4, "restor": 4, "helm": [5, 15], "chart": [5, 15], "get": 5, "repo": 5, "info": 5, "quickstart": 6, "virtual": [7, 113], "machin": 7, "kvm": 7, "hypervisor": 7, "vm": 7, "replac": 7, "certif": 7, "definit": 8, "welcom": [9, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "document": 9, "tutori": [9, 109], "cloud": 9, "nativ": 9, "tool": 9, "fundament": 9, "detail": 9, "switch": [9, 18, 99, 100, 101, 103], "fabric": 9, "network": [9, 20, 23, 24, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 99, 113], "servic": [9, 17, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 106, 109, 112, 113], "oper": [9, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "updat": 9, "introduct": [12, 15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "inventori": [13, 103], "profil": [13, 103], "field": [13, 14, 19, 103, 115], "ip": [14, 100, 106], "address": [14, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 100, 106], "manag": [14, 99, 100, 103], "alloc": 14, "subnet": 14, "add": [14, 103], "an": [14, 16, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "kubernet": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97], "integr": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 107, 108], "method": 15, "regular": 15, "manifest": 15, "us": [15, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 112, 113], "type": 15, "loadbalanc": 15, "resourc": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 102], "l4lb": [15, 17, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "v": [15, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 106, 113, 115], "net": [15, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 106, 113, 115], "bgp": [15, 19, 23, 24, 26, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 104], "import": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "exist": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "from": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "reclaim": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "calico": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "cni": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "disabl": [15, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "l3": [16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "load": [16, 17, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 106, 112], "balanc": [16, 17, 24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 81, 84, 86, 89, 91, 94, 96, 106, 112], "anycast": [16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "lb": 16, "creat": [16, 24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 102], "l4": [17, 112], "enabl": [17, 19, 106, 107, 108, 109], "consum": [17, 109], "architectur": [18, 99], "basic": 19, "ad": [19, 22, 103, 108], "peer": 19, "advanc": 19, "object": 19, "ipv4": 19, "prefix": 19, "ipv6": 19, "commun": 19, "rout": [19, 22], "map": 19, "static": 19, "nat": [19, 23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94, 106], "defin": 19, "sitemesh": 19, "look": 19, "glass": 19, "refer": [20, 99], "design": 20, "releas": 21, "note": 21, "roh": 22, "provid": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 102], "exampl": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93], "ethernet": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "vlan": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "vxlan": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "e": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "exterior": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "border": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "gatewai": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "protocol": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "translat": [23, 24, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 88, 89, 93, 94], "learn": [24, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96], "l3lb": [24, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 84, 89, 94], "sandbox1": 25, "intro": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "deploi": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "applic": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "On": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "demand": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 106, 112], "mileston": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "1": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 106], "vnet": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96], "2": [26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 106], "sandbox": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "topologi": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 103], "diagram": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "server": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "cluster": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "upstream": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "isp": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "sandbox10": 30, "sandbox11": 35, "sandbox12": 40, "sandbox13": 45, "sandbox14": 50, "sandbox15": 55, "sandbox2": 60, "sandbox3": 65, "sandbox4": 70, "sandbox5": 75, "sandbox6": 80, "gui": 82, "sandbox7": 85, "sandbox8": 90, "sandbox9": 95, "perform": 98, "unmanag": 99, "ha": 99, "scalabl": 99, "data": 99, "center": 99, "prerequisit": 100, "step": 100, "nvidia": [100, 103], "cumulu": [100, 103], "devic": 100, "oob": 100, "licens": 100, "ubuntu": 100, "switchdev": 100, "edgecor": 100, "sonic": 100, "port": 101, "terraform": 102, "To": 102, "your": 102, "first": 102, "directori": 102, "file": 102, "prepar": 102, "infrastructur": 102, "plan": 102, "delet": 102, "view": 103, "link": 103, "hairpin": 103, "onli": 103, "activ": 104, "equinix": [104, 105, 107, 108, 109, 113], "metal": [104, 105, 107, 108, 109, 113], "project": [104, 105, 109, 113], "provis": 105, "node": [105, 108], "pool": 106, "request": 106, "new": 106, "public": 106, "block": 106, "elast": [106, 112], "3": 106, "4": 106, "api": [107, 108, 114], "vpc": 109, "rollback": 111, "procedur": 111, "isol": 113, "visibl": 114, "telescop": 114, "graph": 114, "board": 114, "log": 114, "dashboard": 114}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.todo": 2, "sphinx.ext.viewcode": 1, "sphinx": 57}, "alltitles": {"SoftGate PRO Installation": [[0, "softgate-pro-installation"]], "Minimum Hardware Requirements": [[0, "minimum-hardware-requirements"], [1, "minimum-hardware-requirements"]], "BIOS Configuration": [[0, "bios-configuration"]], "Install the Netris Agent": [[0, "install-the-netris-agent"], [1, "install-the-netris-agent"], [100, "install-the-netris-agent"]], "SoftGate Installation": [[1, "softgate-installation"]], "Accounts": [[2, "accounts"]], "Users": [[2, "users"]], "Tenants": [[2, "tenants"]], "Permission Groups": [[2, "permission-groups"]], "User Roles": [[2, "user-roles"]], "Access Control Lists (ACL)": [[3, "access-control-lists-acl"]], "ACL Default Policy": [[3, "acl-default-policy"]], "ACL Rules": [[3, "acl-rules"]], "ACL Approval Workflow": [[3, "acl-approval-workflow"]], "ACL Processing Order": [[3, "acl-processing-order"]], "Netris Controller installation on a generic Linux host": [[4, "netris-controller-installation-on-a-generic-linux-host"]], "Linux Host requirements": [[4, "linux-host-requirements"]], "Installation": [[4, "installation"]], "Installation with the specific host name": [[4, "installation-with-the-specific-host-name"]], "Installation with the Let\u2019s Encrypt SSL": [[4, "installation-with-the-let-s-encrypt-ssl"]], "Installation with the Custom SSL Issuer": [[4, "installation-with-the-custom-ssl-issuer"]], "Upgrading": [[4, "upgrading"]], "Uninstalling": [[4, "uninstalling"]], "Backup and Restore": [[4, "backup-and-restore"]], "Backup": [[4, "backup"]], "Restore": [[4, "restore"]], "Helm Chart Installation": [[5, "helm-chart-installation"]], "Requirements": [[5, "requirements"], [7, "requirements"]], "Get Repo Info": [[5, "get-repo-info"]], "Installing the Chart": [[5, "installing-the-chart"]], "Uninstalling the Chart": [[5, "uninstalling-the-chart"]], "Chart Configuration": [[5, "chart-configuration"]], "Quickstart Installation": [[6, "quickstart-installation"]], "Quickstart Process": [[6, "quickstart-process"]], "Virtual Machine Installation": [[7, "virtual-machine-installation"]], "KVM Hypervisor Installation": [[7, "kvm-hypervisor-installation"]], "VM Controller Installation": [[7, "vm-controller-installation"]], "Accessing the Netris Controller": [[7, "accessing-the-netris-controller"]], "Replacing the SSL certificate": [[7, "replacing-the-ssl-certificate"]], "Definitions": [[8, "definitions"]], "Welcome to Netris Documentation": [[9, "welcome-to-netris-documentation"]], "Tutorials": [[9, null]], "Cloud Native Tools": [[9, null]], "Netris Fundamentals": [[9, null]], "Detailed Installation": [[9, null]], "Switch-fabric Configuration": [[9, null]], "Network Policies": [[9, null]], "Network Services": [[9, null]], "Operations": [[9, null]], "Updates": [[9, null]], "Controller Installation": [[10, "controller-installation"], [10, null]], "Installing a Netris Controller": [[11, "installing-a-netris-controller"], [110, "installing-a-netris-controller"]], "Introduction to Netris": [[12, "introduction-to-netris"]], "Inventory Profiles": [[13, "inventory-profiles"], [103, "inventory-profiles"]], "Inventory Profile Fields": [[13, "id1"], [103, "id1"]], "IP Address Management": [[14, "ip-address-management"]], "Allocations and Subnets": [[14, "allocations-and-subnets"]], "Add an Allocation": [[14, "add-an-allocation"]], "Allocation Fields": [[14, "id1"]], "Add a Subnet": [[14, "add-a-subnet"]], "Subnet fields": [[14, "id2"]], "Kubernetes Integration": [[15, "kubernetes-integration"]], "Install Netris Operator": [[15, "install-netris-operator"], [26, "install-netris-operator"], [31, "install-netris-operator"], [36, "install-netris-operator"], [41, "install-netris-operator"], [46, "install-netris-operator"], [51, "install-netris-operator"], [56, "install-netris-operator"], [61, "install-netris-operator"], [66, "install-netris-operator"], [71, "install-netris-operator"], [76, "install-netris-operator"], [81, "install-netris-operator"], [86, "install-netris-operator"], [91, "install-netris-operator"], [96, "install-netris-operator"]], "Helm Chart Method": [[15, "helm-chart-method"]], "Regular Manifest Method": [[15, "regular-manifest-method"]], "Using Type \u2018LoadBalancer\u2019": [[15, "using-type-loadbalancer"]], "Using Netris Custom Resources": [[15, "using-netris-custom-resources"], [26, "using-netris-custom-resources"], [31, "using-netris-custom-resources"], [36, "using-netris-custom-resources"], [41, "using-netris-custom-resources"], [46, "using-netris-custom-resources"], [51, "using-netris-custom-resources"], [56, "using-netris-custom-resources"], [61, "using-netris-custom-resources"], [66, "using-netris-custom-resources"], [71, "using-netris-custom-resources"], [76, "using-netris-custom-resources"], [81, "using-netris-custom-resources"], [86, "using-netris-custom-resources"], [91, "using-netris-custom-resources"], [96, "using-netris-custom-resources"]], "Introduction to Netris Custom Resources": [[15, "introduction-to-netris-custom-resources"], [26, "introduction-to-netris-custom-resources"], [31, "introduction-to-netris-custom-resources"], [36, "introduction-to-netris-custom-resources"], [41, "introduction-to-netris-custom-resources"], [46, "introduction-to-netris-custom-resources"], [51, "introduction-to-netris-custom-resources"], [56, "introduction-to-netris-custom-resources"], [61, "introduction-to-netris-custom-resources"], [66, "introduction-to-netris-custom-resources"], [71, "introduction-to-netris-custom-resources"], [76, "introduction-to-netris-custom-resources"], [81, "introduction-to-netris-custom-resources"], [86, "introduction-to-netris-custom-resources"], [91, "introduction-to-netris-custom-resources"], [96, "introduction-to-netris-custom-resources"]], "L4LB Custom Resource": [[15, "l4lb-custom-resource"], [26, "l4lb-custom-resource"], [31, "l4lb-custom-resource"], [36, "l4lb-custom-resource"], [41, "l4lb-custom-resource"], [46, "l4lb-custom-resource"], [51, "l4lb-custom-resource"], [56, "l4lb-custom-resource"], [61, "l4lb-custom-resource"], [66, "l4lb-custom-resource"], [71, "l4lb-custom-resource"], [76, "l4lb-custom-resource"], [81, "l4lb-custom-resource"], [86, "l4lb-custom-resource"], [91, "l4lb-custom-resource"], [96, "l4lb-custom-resource"]], "V-Net Custom Resource": [[15, "v-net-custom-resource"]], "BGP Custom Resource": [[15, "bgp-custom-resource"], [26, "bgp-custom-resource"], [31, "bgp-custom-resource"], [36, "bgp-custom-resource"], [41, "bgp-custom-resource"], [46, "bgp-custom-resource"], [51, "bgp-custom-resource"], [56, "bgp-custom-resource"], [61, "bgp-custom-resource"], [66, "bgp-custom-resource"], [71, "bgp-custom-resource"], [76, "bgp-custom-resource"], [81, "bgp-custom-resource"], [86, "bgp-custom-resource"], [91, "bgp-custom-resource"], [96, "bgp-custom-resource"]], "Importing existing resources from Netris Controller to Kubernetes": [[15, "importing-existing-resources-from-netris-controller-to-kubernetes"], [26, "importing-existing-resources-from-netris-controller-to-kubernetes"], [31, "importing-existing-resources-from-netris-controller-to-kubernetes"], [36, "importing-existing-resources-from-netris-controller-to-kubernetes"], [41, "importing-existing-resources-from-netris-controller-to-kubernetes"], [46, "importing-existing-resources-from-netris-controller-to-kubernetes"], [51, "importing-existing-resources-from-netris-controller-to-kubernetes"], [56, "importing-existing-resources-from-netris-controller-to-kubernetes"], [61, "importing-existing-resources-from-netris-controller-to-kubernetes"], [66, "importing-existing-resources-from-netris-controller-to-kubernetes"], [71, "importing-existing-resources-from-netris-controller-to-kubernetes"], [76, "importing-existing-resources-from-netris-controller-to-kubernetes"], [81, "importing-existing-resources-from-netris-controller-to-kubernetes"], [86, "importing-existing-resources-from-netris-controller-to-kubernetes"], [91, "importing-existing-resources-from-netris-controller-to-kubernetes"], [96, "importing-existing-resources-from-netris-controller-to-kubernetes"]], "Reclaim Policy": [[15, "reclaim-policy"], [26, "reclaim-policy"], [31, "reclaim-policy"], [36, "reclaim-policy"], [41, "reclaim-policy"], [46, "reclaim-policy"], [51, "reclaim-policy"], [56, "reclaim-policy"], [61, "reclaim-policy"], [66, "reclaim-policy"], [71, "reclaim-policy"], [76, "reclaim-policy"], [81, "reclaim-policy"], [86, "reclaim-policy"], [91, "reclaim-policy"], [96, "reclaim-policy"]], "Calico CNI Integration": [[15, "calico-cni-integration"]], "Disabling Netris-Calico Integration": [[15, "disabling-netris-calico-integration"], [26, "disabling-netris-calico-integration"], [31, "disabling-netris-calico-integration"], [36, "disabling-netris-calico-integration"], [41, "disabling-netris-calico-integration"], [46, "disabling-netris-calico-integration"], [51, "disabling-netris-calico-integration"], [56, "disabling-netris-calico-integration"], [61, "disabling-netris-calico-integration"], [66, "disabling-netris-calico-integration"], [71, "disabling-netris-calico-integration"], [76, "disabling-netris-calico-integration"], [81, "disabling-netris-calico-integration"], [86, "disabling-netris-calico-integration"], [91, "disabling-netris-calico-integration"], [96, "disabling-netris-calico-integration"]], "L3 Load Balancer (Anycast LB)": [[16, "l3-load-balancer-anycast-lb"]], "Creating an L3 Load Balancer": [[16, "creating-an-l3-load-balancer"]], "L4 Load Balancer (L4LB)": [[17, "l4-load-balancer-l4lb"]], "Enabling L4LB service": [[17, "enabling-l4lb-service"]], "Consuming L4LB service": [[17, "consuming-l4lb-service"]], "Netris Architecture": [[18, "netris-architecture"]], "Netris Controller": [[18, "netris-controller"], [27, "netris-controller"], [32, "netris-controller"], [37, "netris-controller"], [42, "netris-controller"], [47, "netris-controller"], [52, "netris-controller"], [57, "netris-controller"], [62, "netris-controller"], [67, "netris-controller"], [72, "netris-controller"], [77, "netris-controller"], [87, "netris-controller"], [92, "netris-controller"], [97, "netris-controller"]], "Netris Switch Agent": [[18, "netris-switch-agent"]], "Netris SoftGate": [[18, "netris-softgate"]], "Basic BGP": [[19, "basic-bgp"]], "Adding BGP Peers": [[19, "adding-bgp-peers"]], "BGP Peer Fields": [[19, "id1"]], "Advanced BGP": [[19, "advanced-bgp"]], "BGP Peer Fields - Advanced": [[19, "id2"]], "BGP Objects": [[19, "bgp-objects"]], "IPv4 Prefix": [[19, "ipv4-prefix"]], "IPv6 Prefix": [[19, "ipv6-prefix"]], "Community": [[19, "community"]], "BGP route-maps": [[19, "bgp-route-maps"]], "Static Routing": [[19, "static-routing"]], "NAT": [[19, "nat"]], "Enabling NAT": [[19, "enabling-nat"]], "Defining NAT rules": [[19, "defining-nat-rules"]], "SiteMesh": [[19, "sitemesh"]], "Looking Glass": [[19, "looking-glass"]], "Network Reference Designs": [[20, "network-reference-designs"]], "Release notes": [[21, "release-notes"]], "ROH (Routing on the Host)": [[22, "roh-routing-on-the-host"]], "Adding ROH Hosts": [[22, "adding-roh-hosts"]], "Provided Example Configurations": [[23, "provided-example-configurations"], [28, "provided-example-configurations"], [33, "provided-example-configurations"], [38, "provided-example-configurations"], [43, "provided-example-configurations"], [48, "provided-example-configurations"], [53, "provided-example-configurations"], [58, "provided-example-configurations"], [63, "provided-example-configurations"], [68, "provided-example-configurations"], [73, "provided-example-configurations"], [78, "provided-example-configurations"], [83, "provided-example-configurations"], [88, "provided-example-configurations"], [93, "provided-example-configurations"]], "V-Net (Ethernet/Vlan/VXlan) Example": [[23, "v-net-ethernet-vlan-vxlan-example"], [28, "v-net-ethernet-vlan-vxlan-example"], [33, "v-net-ethernet-vlan-vxlan-example"], [38, "v-net-ethernet-vlan-vxlan-example"], [43, "v-net-ethernet-vlan-vxlan-example"], [48, "v-net-ethernet-vlan-vxlan-example"], [53, "v-net-ethernet-vlan-vxlan-example"], [58, "v-net-ethernet-vlan-vxlan-example"], [63, "v-net-ethernet-vlan-vxlan-example"], [68, "v-net-ethernet-vlan-vxlan-example"], [73, "v-net-ethernet-vlan-vxlan-example"], [78, "v-net-ethernet-vlan-vxlan-example"], [83, "v-net-ethernet-vlan-vxlan-example"], [88, "v-net-ethernet-vlan-vxlan-example"], [93, "v-net-ethernet-vlan-vxlan-example"]], "E-BGP (Exterior Border Gateway Protocol) Example": [[23, "e-bgp-exterior-border-gateway-protocol-example"], [28, "e-bgp-exterior-border-gateway-protocol-example"], [33, "e-bgp-exterior-border-gateway-protocol-example"], [38, "e-bgp-exterior-border-gateway-protocol-example"], [43, "e-bgp-exterior-border-gateway-protocol-example"], [48, "e-bgp-exterior-border-gateway-protocol-example"], [53, "e-bgp-exterior-border-gateway-protocol-example"], [58, "e-bgp-exterior-border-gateway-protocol-example"], [63, "e-bgp-exterior-border-gateway-protocol-example"], [68, "e-bgp-exterior-border-gateway-protocol-example"], [73, "e-bgp-exterior-border-gateway-protocol-example"], [78, "e-bgp-exterior-border-gateway-protocol-example"], [83, "e-bgp-exterior-border-gateway-protocol-example"], [88, "e-bgp-exterior-border-gateway-protocol-example"], [93, "e-bgp-exterior-border-gateway-protocol-example"]], "NAT (Network Address Translation) Example": [[23, "nat-network-address-translation-example"], [28, "nat-network-address-translation-example"], [33, "nat-network-address-translation-example"], [38, "nat-network-address-translation-example"], [43, "nat-network-address-translation-example"], [48, "nat-network-address-translation-example"], [53, "nat-network-address-translation-example"], [58, "nat-network-address-translation-example"], [63, "nat-network-address-translation-example"], [68, "nat-network-address-translation-example"], [73, "nat-network-address-translation-example"], [78, "nat-network-address-translation-example"], [83, "nat-network-address-translation-example"], [88, "nat-network-address-translation-example"], [93, "nat-network-address-translation-example"]], "ACL (Access Control List) Example": [[23, "acl-access-control-list-example"], [28, "acl-access-control-list-example"], [33, "acl-access-control-list-example"], [38, "acl-access-control-list-example"], [43, "acl-access-control-list-example"], [48, "acl-access-control-list-example"], [53, "acl-access-control-list-example"], [58, "acl-access-control-list-example"], [63, "acl-access-control-list-example"], [68, "acl-access-control-list-example"], [73, "acl-access-control-list-example"], [78, "acl-access-control-list-example"], [83, "acl-access-control-list-example"], [88, "acl-access-control-list-example"], [93, "acl-access-control-list-example"]], "Learn by Creating Services": [[24, "learn-by-creating-services"], [29, "learn-by-creating-services"], [34, "learn-by-creating-services"], [39, "learn-by-creating-services"], [44, "learn-by-creating-services"], [49, "learn-by-creating-services"], [54, "learn-by-creating-services"], [59, "learn-by-creating-services"], [64, "learn-by-creating-services"], [69, "learn-by-creating-services"], [74, "learn-by-creating-services"], [79, "learn-by-creating-services"], [84, "learn-by-creating-services"], [89, "learn-by-creating-services"], [94, "learn-by-creating-services"]], "V-Net (Ethernet/Vlan/VXlan)": [[24, "v-net-ethernet-vlan-vxlan"], [29, "v-net-ethernet-vlan-vxlan"], [34, "v-net-ethernet-vlan-vxlan"], [39, "v-net-ethernet-vlan-vxlan"], [44, "v-net-ethernet-vlan-vxlan"], [49, "v-net-ethernet-vlan-vxlan"], [54, "v-net-ethernet-vlan-vxlan"], [59, "v-net-ethernet-vlan-vxlan"], [64, "v-net-ethernet-vlan-vxlan"], [69, "v-net-ethernet-vlan-vxlan"], [74, "v-net-ethernet-vlan-vxlan"], [79, "v-net-ethernet-vlan-vxlan"], [84, "v-net-ethernet-vlan-vxlan"], [89, "v-net-ethernet-vlan-vxlan"], [94, "v-net-ethernet-vlan-vxlan"]], "E-BGP (Exterior Border Gateway Protocol)": [[24, "e-bgp-exterior-border-gateway-protocol"], [29, "e-bgp-exterior-border-gateway-protocol"], [34, "e-bgp-exterior-border-gateway-protocol"], [39, "e-bgp-exterior-border-gateway-protocol"], [44, "e-bgp-exterior-border-gateway-protocol"], [49, "e-bgp-exterior-border-gateway-protocol"], [54, "e-bgp-exterior-border-gateway-protocol"], [59, "e-bgp-exterior-border-gateway-protocol"], [64, "e-bgp-exterior-border-gateway-protocol"], [69, "e-bgp-exterior-border-gateway-protocol"], [74, "e-bgp-exterior-border-gateway-protocol"], [79, "e-bgp-exterior-border-gateway-protocol"], [84, "e-bgp-exterior-border-gateway-protocol"], [89, "e-bgp-exterior-border-gateway-protocol"], [94, "e-bgp-exterior-border-gateway-protocol"]], "NAT (Network Address Translation)": [[24, "nat-network-address-translation"], [29, "nat-network-address-translation"], [34, "nat-network-address-translation"], [39, "nat-network-address-translation"], [44, "nat-network-address-translation"], [49, "nat-network-address-translation"], [54, "nat-network-address-translation"], [59, "nat-network-address-translation"], [64, "nat-network-address-translation"], [69, "nat-network-address-translation"], [74, "nat-network-address-translation"], [79, "nat-network-address-translation"], [84, "nat-network-address-translation"], [89, "nat-network-address-translation"], [94, "nat-network-address-translation"]], "ACL (Access Control List)": [[24, "acl-access-control-list"], [29, "acl-access-control-list"], [34, "acl-access-control-list"], [39, "acl-access-control-list"], [44, "acl-access-control-list"], [49, "acl-access-control-list"], [54, "acl-access-control-list"], [59, "acl-access-control-list"], [64, "acl-access-control-list"], [69, "acl-access-control-list"], [74, "acl-access-control-list"], [79, "acl-access-control-list"], [84, "acl-access-control-list"], [89, "acl-access-control-list"], [94, "acl-access-control-list"]], "L3LB (Anycast L3 load balancer)": [[24, "l3lb-anycast-l3-load-balancer"], [29, "l3lb-anycast-l3-load-balancer"], [34, "l3lb-anycast-l3-load-balancer"], [39, "l3lb-anycast-l3-load-balancer"], [44, "l3lb-anycast-l3-load-balancer"], [49, "l3lb-anycast-l3-load-balancer"], [54, "l3lb-anycast-l3-load-balancer"], [59, "l3lb-anycast-l3-load-balancer"], [64, "l3lb-anycast-l3-load-balancer"], [69, "l3lb-anycast-l3-load-balancer"], [74, "l3lb-anycast-l3-load-balancer"], [84, "l3lb-anycast-l3-load-balancer"], [89, "l3lb-anycast-l3-load-balancer"], [94, "l3lb-anycast-l3-load-balancer"]], "Sandbox1": [[25, "sandbox1"]], "Learn Netris operations with Kubernetes": [[26, "learn-netris-operations-with-kubernetes"], [31, "learn-netris-operations-with-kubernetes"], [36, "learn-netris-operations-with-kubernetes"], [41, "learn-netris-operations-with-kubernetes"], [46, "learn-netris-operations-with-kubernetes"], [51, "learn-netris-operations-with-kubernetes"], [56, "learn-netris-operations-with-kubernetes"], [61, "learn-netris-operations-with-kubernetes"], [66, "learn-netris-operations-with-kubernetes"], [71, "learn-netris-operations-with-kubernetes"], [76, "learn-netris-operations-with-kubernetes"], [81, "learn-netris-operations-with-kubernetes"], [86, "learn-netris-operations-with-kubernetes"], [91, "learn-netris-operations-with-kubernetes"], [96, "learn-netris-operations-with-kubernetes"]], "Intro": [[26, "intro"], [31, "intro"], [36, "intro"], [41, "intro"], [46, "intro"], [51, "intro"], [56, "intro"], [61, "intro"], [66, "intro"], [71, "intro"], [76, "intro"], [81, "intro"], [86, "intro"], [91, "intro"], [96, "intro"]], "Deploy an Application with an On-Demand Netris Load Balancer": [[26, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [31, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [36, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [41, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [46, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [51, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [56, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [61, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [66, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [71, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [76, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [81, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [86, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [91, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [96, "deploy-an-application-with-an-on-demand-netris-load-balancer"]], "Milestone 1": [[26, null], [31, null], [36, null], [41, null], [46, null], [51, null], [56, null], [61, null], [66, null], [71, null], [76, null], [81, null], [86, null], [91, null], [96, null]], "VNet Custom Resource": [[26, "vnet-custom-resource"], [31, "vnet-custom-resource"], [36, "vnet-custom-resource"], [41, "vnet-custom-resource"], [46, "vnet-custom-resource"], [51, "vnet-custom-resource"], [56, "vnet-custom-resource"], [61, "vnet-custom-resource"], [66, "vnet-custom-resource"], [71, "vnet-custom-resource"], [76, "vnet-custom-resource"], [81, "vnet-custom-resource"], [86, "vnet-custom-resource"], [91, "vnet-custom-resource"], [96, "vnet-custom-resource"]], "Netris Calico CNI Integration": [[26, "netris-calico-cni-integration"], [31, "netris-calico-cni-integration"], [36, "netris-calico-cni-integration"], [41, "netris-calico-cni-integration"], [46, "netris-calico-cni-integration"], [51, "netris-calico-cni-integration"], [56, "netris-calico-cni-integration"], [61, "netris-calico-cni-integration"], [66, "netris-calico-cni-integration"], [71, "netris-calico-cni-integration"], [76, "netris-calico-cni-integration"], [81, "netris-calico-cni-integration"], [86, "netris-calico-cni-integration"], [91, "netris-calico-cni-integration"], [96, "netris-calico-cni-integration"]], "Milestone 2": [[26, null], [31, null], [36, null], [41, null], [46, null], [51, null], [56, null], [61, null], [66, null], [71, null], [76, null], [81, null], [86, null], [91, null], [96, null]], "Welcome to Netris Sandbox": [[27, "welcome-to-netris-sandbox"], [32, "welcome-to-netris-sandbox"], [37, "welcome-to-netris-sandbox"], [42, "welcome-to-netris-sandbox"], [47, "welcome-to-netris-sandbox"], [52, "welcome-to-netris-sandbox"], [57, "welcome-to-netris-sandbox"], [62, "welcome-to-netris-sandbox"], [67, "welcome-to-netris-sandbox"], [72, "welcome-to-netris-sandbox"], [77, "welcome-to-netris-sandbox"], [82, "welcome-to-netris-sandbox"], [87, "welcome-to-netris-sandbox"], [92, "welcome-to-netris-sandbox"], [97, "welcome-to-netris-sandbox"]], "Topology diagram": [[27, "topology-diagram"], [32, "topology-diagram"], [37, "topology-diagram"], [42, "topology-diagram"], [47, "topology-diagram"], [52, "topology-diagram"], [57, "topology-diagram"], [62, "topology-diagram"], [67, "topology-diagram"], [72, "topology-diagram"], [77, "topology-diagram"], [82, "topology-diagram"], [87, "topology-diagram"], [92, "topology-diagram"], [97, "topology-diagram"]], "Linux servers": [[27, "linux-servers"], [32, "linux-servers"], [37, "linux-servers"], [42, "linux-servers"], [47, "linux-servers"], [52, "linux-servers"], [57, "linux-servers"], [62, "linux-servers"], [67, "linux-servers"], [72, "linux-servers"], [77, "linux-servers"], [82, "linux-servers"], [87, "linux-servers"], [92, "linux-servers"], [97, "linux-servers"]], "Kubernetes cluster": [[27, "kubernetes-cluster"], [32, "kubernetes-cluster"], [37, "kubernetes-cluster"], [42, "kubernetes-cluster"], [47, "kubernetes-cluster"], [52, "kubernetes-cluster"], [57, "kubernetes-cluster"], [62, "kubernetes-cluster"], [67, "kubernetes-cluster"], [72, "kubernetes-cluster"], [77, "kubernetes-cluster"], [82, "kubernetes-cluster"], [87, "kubernetes-cluster"], [92, "kubernetes-cluster"], [97, "kubernetes-cluster"]], "Upstream ISP": [[27, "upstream-isp"], [32, "upstream-isp"], [37, "upstream-isp"], [42, "upstream-isp"], [47, "upstream-isp"], [52, "upstream-isp"], [57, "upstream-isp"], [62, "upstream-isp"], [67, "upstream-isp"], [72, "upstream-isp"], [77, "upstream-isp"], [82, "upstream-isp"], [87, "upstream-isp"], [92, "upstream-isp"], [97, "upstream-isp"]], "Networks Used": [[27, "networks-used"], [32, "networks-used"], [37, "networks-used"], [42, "networks-used"], [47, "networks-used"], [52, "networks-used"], [57, "networks-used"], [62, "networks-used"], [67, "networks-used"], [72, "networks-used"], [77, "networks-used"], [87, "networks-used"], [92, "networks-used"], [97, "networks-used"]], "Sandbox10": [[30, "sandbox10"]], "Sandbox11": [[35, "sandbox11"]], "Sandbox12": [[40, "sandbox12"]], "Sandbox13": [[45, "sandbox13"]], "Sandbox14": [[50, "sandbox14"]], "Sandbox15": [[55, "sandbox15"]], "Sandbox2": [[60, "sandbox2"]], "Sandbox3": [[65, "sandbox3"]], "Sandbox4": [[70, "sandbox4"]], "Sandbox5": [[75, "sandbox5"]], "Sandbox6": [[80, "sandbox6"]], "Netris GUI": [[82, "netris-gui"]], "Networks used": [[82, "networks-used"]], "Sandbox7": [[85, "sandbox7"]], "Sandbox8": [[90, "sandbox8"]], "Sandbox9": [[95, "sandbox9"]], "SoftGate Performance": [[98, "softgate-performance"], [98, "id1"]], "Reference Network Architectures": [[99, "reference-network-architectures"]], "Unmanaged Switch & Netris SoftGate": [[99, "unmanaged-switch-netris-softgate"]], "Unmanaged Switch & SoftGate (HA)": [[99, "unmanaged-switch-softgate-ha"]], "Netris Managed Switch & SoftGate (HA)": [[99, "netris-managed-switch-softgate-ha"]], "Netris Managed Switch & SoftGate scalable data center (HA)": [[99, "netris-managed-switch-softgate-scalable-data-center-ha"]], "Switch Agent Installation": [[100, "id1"]], "Prerequisite Steps": [[100, "prerequisite-steps"]], "Nvidia Cumulus Linux Devices": [[100, "nvidia-cumulus-linux-devices"]], "Configure the OOB Management IP address": [[100, "configure-the-oob-management-ip-address"], [100, "id2"], [100, "id3"]], "Configure Nvidia Cumulus Linux License": [[100, "configure-nvidia-cumulus-linux-license"]], "Ubuntu SwitchDev Devices": [[100, "ubuntu-switchdev-devices"]], "EdgeCore SONiC Devices": [[100, "edgecore-sonic-devices"]], "Switch Ports": [[101, "switch-ports"]], "Terraform: Netris provider": [[102, "terraform-netris-provider"]], "To create your first Terraform configuration:": [[102, "to-create-your-first-terraform-configuration"]], "Install Terraform": [[102, "install-terraform"]], "Create a directory for Terraform files": [[102, "create-a-directory-for-terraform-files"]], "Configure a provider": [[102, "configure-a-provider"]], "Prepare an infrastructure plan": [[102, "prepare-an-infrastructure-plan"]], "Create resources": [[102, "create-resources"]], "Delete resources": [[102, "delete-resources"]], "Inventory": [[103, "inventory"]], "Adding Switches": [[103, "adding-switches"]], "Add Inventory Fields - Switch": [[103, "id2"]], "Adding SoftGates": [[103, "adding-softgates"]], "Add Inventory Fields - SoftGate": [[103, "id3"]], "Viewing Inventory": [[103, "viewing-inventory"]], "Topology Manager": [[103, "topology-manager"]], "Adding Links": [[103, "adding-links"]], "Hairpin Links (Nvidia Cumulus only)": [[103, "hairpin-links-nvidia-cumulus-only"]], "Activating BGP on Equinix Metal Project": [[104, "activating-bgp-on-equinix-metal-project"]], "Provisioning Netris SoftGate nodes in Equinix Metal Project": [[105, "provisioning-netris-softgate-nodes-in-equinix-metal-project"]], "Enabling services (NAT, V-Net, Load Balancer, IP pools)": [[106, "enabling-services-nat-v-net-load-balancer-ip-pools"]], "1) Requesting new Public IP address block": [[106, "requesting-new-public-ip-address-block"]], "2) Enable on-demand (elastic) Load Balancer": [[106, "enable-on-demand-elastic-load-balancer"]], "3) Enable V-Net": [[106, "enable-v-net"]], "4) Enable NAT": [[106, "enable-nat"]], "Equinix Metal API integration enablement": [[107, "equinix-metal-api-integration-enablement"], [108, "equinix-metal-api-integration-enablement"]], "Adding Netris SoftGate nodes": [[108, "adding-netris-softgate-nodes"]], "Generic Netris Tutorials": [[109, "generic-netris-tutorials"]], "Netris VPC for Equinix Metal Tutorials": [[109, "netris-vpc-for-equinix-metal-tutorials"]], "Enabling Netris VPC for Equinix Metal Project": [[109, null]], "Consuming Netris VPC services for Equinix Metal Project": [[109, null]], "Netris Upgrade and Rollback Procedures": [[111, "netris-upgrade-and-rollback-procedures"]], "Upgrade Procedure": [[111, "upgrade-procedure"]], "Rollback Procedure": [[111, "rollback-procedure"]], "Using on-demand (elastic) L4 Load Balancer service": [[112, "using-on-demand-elastic-l4-load-balancer-service"]], "Using V-Net (isolated virtual network) services in Equinix Metal Project": [[113, "using-v-net-isolated-virtual-network-services-in-equinix-metal-project"]], "Visibility (Telescope)": [[114, "visibility-telescope"]], "Graph Boards": [[114, "graph-boards"]], "API Logs": [[114, "api-logs"]], "Dashboard": [[114, "dashboard"]], "V-Net": [[115, "v-net"]], "V-Net Fields": [[115, "v-net-fields"]]}, "indexentries": {}}) \ No newline at end of file diff --git a/en/3.0/softgate-performance.html b/en/3.0/softgate-performance.html new file mode 100644 index 0000000000..c7fc60dc48 --- /dev/null +++ b/en/3.0/softgate-performance.html @@ -0,0 +1,592 @@ + + + + + + + + + + + SoftGate Performance — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

SoftGate Performance

+

The following tested results are offered to help properly size the hardware needed for a SoftGate with various types of services:

+ + +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SoftGate Performance

Routing(mpps)

Routing with unmatched DNAT rules (mpps)

Routing with unmatched DNAT and SNAT rules (mpps)

DNAT

SNAT

DNAT with unmatched SNAT rules

Xeon E5-2660 CPU cores: (6 core for fw) Scheduler 0 Forward 1-7 KNI 8-10 OS 11-15

~27mpps

~17mpps

7mpps

9mpps

~2mpps

Xeon Gold 6130 CPU cores: (6 core for fw) Scheduler 0 Forward 3-13(odd numbers) KNI 15 17 19

~31mpps

~22mpps

~15mpps

~11mpps

~3mpps(3M conntrack count) ~7mpps(20k conntrack count)

~11mpps

Xeon Gold 6130 CPU cores: (8 core for fw) Scheduler 0 Forward 3-17(odd numbers) KNI 19 21 23

~38mpps

~28mpps

19mpps

12mpps

~12mpps

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/supported-networks.html b/en/3.0/supported-networks.html new file mode 100644 index 0000000000..e2ea528130 --- /dev/null +++ b/en/3.0/supported-networks.html @@ -0,0 +1,569 @@ + + + + + + + + + + + Reference Network Architectures — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Reference Network Architectures
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Reference Network Architectures

+
+

Unmanaged Switch & Netris SoftGate

+Unmanaged Switch & SoftGate +
+
+
+

Unmanaged Switch & SoftGate (HA)

+Unmanaged Switch & SoftGate (HA) +
+
+
+

Netris Managed Switch & SoftGate (HA)

+Netris Managed Switch & SoftGate small data center (HA) +
+
+
+

Netris Managed Switch & SoftGate scalable data center (HA)

+Netris Managed Switch & SoftGate scalable data center (HA) +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/switch-agent-installation.html b/en/3.0/switch-agent-installation.html new file mode 100644 index 0000000000..67002b64ec --- /dev/null +++ b/en/3.0/switch-agent-installation.html @@ -0,0 +1,729 @@ + + + + + + + + + + + Switch Agent Installation — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Switch Agent Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Switch Agent Installation

+
+

Prerequisite Steps

+
+

Nvidia Cumulus Linux Devices

+

Requirements: +* Fresh install of Cumulus Linux v. 3.7.(x) - Cumulus 4.X is in the process of validation and will be supported in the next Netris release.

+
+

Configure the OOB Management IP address

+

Configure internet connectivity via management port.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <management IP address/prefix length>
+        gateway <gateway of management network>
+        dns-nameserver <dns server>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
+
+

Configure Nvidia Cumulus Linux License

+
sudo cl-license -i
+
+
+

Copy/paste the Cumulus Linux license string then press ctrl-d.

+

Continue to “Install the Netris Agent” section.

+
+
+
+

Ubuntu SwitchDev Devices

+
+

Note

+

Further installation requires a Console and Internet connectivity via management port!

+
+
    +
  1. NOS Uninstall

  2. +
+

Uninstall current NOS using Uninstall OS from grub menu:

+_images/uninstallOS.png +

Once the uninstallation is completed, the switch will reboot automatically.

+
    +
  1. Update ONIE

  2. +
+

Select Update ONIE from grub menu:

+_images/updateONIE.png +

In case you don’t have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually:

+
onie-discovery-stop
+ip addr add <management IP address/prefix> dev eth0
+ip route add default via <gateway of management network>
+echo "nameserver <dns server>" > /etc/resolv.conf
+
+
+

Update ONIE to the supported version.

+
+

Note

+

ONIE image available for Mellanox switches only!

+
+
onie-self-update http://downloads.netris.ai/onie-updater-x86_64-mlnx_x86-r0
+
+
+
    +
  1. NOS Install

  2. +
+

Select Install OS from grub menu:

+_images/installOS.png +

In case you don’t have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually:

+
onie-discovery-stop
+ip addr add <management IP address/prefix> dev eth0
+ip route add default via <gateway of management network>
+echo "nameserver <dns server>" > /etc/resolv.conf
+
+
+

Install Ubuntu-SwitchDev from the Netris custom image:

+
onie-nos-install http://downloads.netris.ai/netris-ubuntu-18.04.1.bin
+
+
+

Default username/password

+

netris/newNet0ps

+
+

Configure the OOB Management IP address

+

Configure internet connectivity via management port.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <management IP address/prefix length>
+        gateway <gateway of management network>
+        dns-nameserver <dns server>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+

Continue to “Install the Netris Agent” section.

+
+
+
+

EdgeCore SONiC Devices

+
+

Note

+

Further installation requires a Console and Internet connectivity via management port!

+
+
    +
  1. NOS Uninstall

  2. +
+

Uninstall current NOS using Uninstall OS from grub menu:

+_images/uninstallOS.png +

Once the uninstallation is completed, the switch will reboot automatically.

+
    +
  1. NOS Install

  2. +
+

Select Install OS from grub menu:

+_images/installOS.png +

In case you don’t have DHCP in the management network, then stop ONIE discovery service and configure IP address and default gateway manually:

+
onie-discovery-stop
+ip addr add <management IP address/prefix> dev eth0
+ip route add default via <gateway of management network>
+echo "nameserver <dns server>" > /etc/resolv.conf
+
+
+

Install EdgeCore SONiC image from the Netris repository:

+
onie-nos-install http://downloads.netris.ai/Edgecore-SONiC_20211125_074752_ec202012_227.bin
+
+
+

Default username/password

+

admin/YourPaSsWoRd

+
+

Configure the OOB Management IP address

+

Disable Zero Touch Provisioning for time being.

+
ztp disable -y
+
+
+
+

Note

+

This will take some time, please be patient.

+
+

Configure internet connectivity via management port.

+
ip addr add <management IP address/prefix> dev eth0
+ip route add default via <gateway of management network>
+echo "nameserver <dns server>" > /etc/resolv.conf
+
+
+

Continue to “Install the Netris Agent” section.

+
+
+
+
+

Install the Netris Agent

+
    +
  1. Add the Switch in the controller Inventory. Detailed configuration documentation is available here: “Adding Switches”

  2. +
  3. Once the Switch is created in the Inventory, click on three vertical dots (⋮) on the right side on the Switch and select the Install Agent option

  4. +
  5. Copy the agent install command to your clipboard and run the command on the Switch

  6. +
  7. Reboot the Switch when the installation completes

  8. +
+
sudo reboot
+
+
+

Once the switch boots up you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and switch color will reflect its health in Net→Topology

+

Screenshot: Net→Inventory

+_images/inventory_heartbeat.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/switch-ports.html b/en/3.0/switch-ports.html new file mode 100644 index 0000000000..cdada739c0 --- /dev/null +++ b/en/3.0/switch-ports.html @@ -0,0 +1,571 @@ + + + + + + + + + + + Switch Ports — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Switch Ports

+

Switch ports can be directly managed in the Switch Port UI section. Both physical and virtual ports (extended, aggregate, etc…) will appear in this section once they have been added to inventory. The Netris Controller will automatically sync the list of available ports that appear on each device.

+

The following options are available for editing on each port:

+
    +
  • Description - Description of the port.

  • +
  • Tenant - Tenant to whom the port is assigned, by default it is the owner tenant of the device to whom the port belongs to.

  • +
  • Breakout - Available only for physical switch ports, used to split physical ports into multiple physical ports. When there is a need to use other supported option supported by switch not shown in the dropdown list user must set breakout to “Manual” and configure breakout manually on the switch. For certain platforms some ports need to be disabled to support breakout into other ports, for that option use “Disable” mode of breakout. For Cumulus, after configuration, user must manually restart switchd daemon on the switch via command “systemctl restart switchd”.

  • +
  • MTU - Maximum transmission unit of the port.

  • +
  • Autoneg - Toggle autonegotiation. Available only for physical ports.

  • +
  • Speed - Toggle speed. Available only for physical ports.

  • +
  • Duplex - Toggle duplex. Available only for physical ports.

  • +
  • Extension - Create extension ports. Available for physical and aggregate ports.

  • +
  • Extension Name - Name for new extension.

  • +
  • VLAN Range - VLAN id range for new extension port.

  • +
+_images/edit-port.png +

Example: Edit physical port

+

Quick action menu provides following actions for ports (note that Bulk Action also available for multiple ports:

+

Edit - Edit the port. +Admin UP/Down - Toggle admin status of the port. +Add to V-net - Add selected port(s) to a V-net.

+_images/add-to-lag-port.png +

Free Up Port - Detach port from all resources.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/terraform-integration.html b/en/3.0/terraform-integration.html new file mode 100644 index 0000000000..6fb7b82408 --- /dev/null +++ b/en/3.0/terraform-integration.html @@ -0,0 +1,834 @@ + + + + + + + + + + + Terraform: Netris provider — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Terraform: Netris provider
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Terraform: Netris provider

+

Use Netris provider to interact with the many resources supported by Netris. You must configure the provider with the proper credentials before you can use it. +To learn the basics of Terraform using this provider, follow the hands-on get started tutorials on HashiCorp’s Learn platform.

+_images/diagrams_terraform.png +

When you make changes in the Terraform files and apply them, Terraform automatically decides which part of your configuration is already deployed into Netris controller and what should be added or removed.

+ +
+

Install Terraform

+

Download and install the Terraform

+
+
+

Create a directory for Terraform files

+
    +
  1. Create a directory with any name, for example, netris-terraform. It stores the configuration files and saved states for Terraform and your infrastructure.

  2. +
  3. Create a configuration file with the .tf extension in this directory, such as main.tf.

  4. +
+
+
+

Configure a provider

+
    +
  1. At the beginning of the configuration file, specify the provider settings.

  2. +
+
terraform {
+  required_providers {
+    netris = {
+      source  = "netrisai/netris"
+      version = "1.1.0"
+    }
+  }
+}
+
+provider "netris" {
+  address = "<controller address>"
+  login = "<controller login>"
+  password = "<controller password>"
+}
+
+
+

Specify the provider required arguments:

+
    +
  • address - This is your Netris-Controller address (http://example.com). This can also be specified with the NETRIS_ADDRESS environment variable.

  • +
  • login - This is your Netris-Controller login. This can also be specified with the NETRIS_LOGIN environment variable.

  • +
  • password - This is your Netris-Controller password. This can also be specified with the NETRIS_PASSWORD environment variable.

  • +
+
    +
  1. Execute the command terraform init in the folder with the configuration file. This command initializes the providers specified in the configuration files and lets you work with the provider resources and data sources.

  2. +
+
+
+

Prepare an infrastructure plan

+

By using Netris Terraform Provider, you can create all kinds of resources, such as Sites, IPAMs, Topology, Inventory, etc. +To create a resource, specify a set of required and optional parameters that define the resource properties. Such resource descriptions make up an infrastructure plan.

+

Infrastructure provisioning in Netris starts with Site resources. The Netris-Controller comes with the initial site Default. You can use it in your Terraform configuration files by getting its ID with the Terraform Data source element.

+

Let’s create a separate file for site resource, and get its ID via Terraform Data source element.

+
cat << EOF > site.tf
+data "netris_site" "default" {
+  name = "Default"
+}
+EOF
+
+
+

Or, you can create a new Site resource, here is the detailed documentation with examples.

+

Now, when we’re clear on Site resource usage, let’s define our IPAM. There are two types of IPAM resources in the Netris-Controller it’s Allocation and Subnet. +IPAM resources only require tenantid field, let’s get our default Admin tenant ID with the Data source element.

+
cat << EOF > tenant.tf
+data "netris_tenant" "admin"{
+  name = "Admin"
+}
+EOF
+
+
+

Then, when we have the tenantid, we can create IPAM resources.

+
cat << EOF > ipam.tf
+resource "netris_allocation" "my-allocation-mgmt" {
+  name = "my-allocation-mgmt"
+  prefix = "192.0.2.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_allocation" "my-allocation-loopback" {
+  name = "my-allocation-loopback"
+  prefix = "198.51.100.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_allocation" "my-allocation-common" {
+  name = "my-allocation-common"
+  prefix = "203.0.113.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_subnet" "my-subnet-mgmt" {
+  name = "my-subnet-mgmt"
+  prefix = "192.0.2.0/24"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "management"
+  defaultgateway = "192.0.2.254"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-mgmt,
+  ]
+}
+
+resource "netris_subnet" "my-subnet-loopback" {
+  name = "my-subnet-loopback"
+  prefix = "198.51.100.0/24"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "loopback"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-loopback,
+  ]
+}
+
+resource "netris_subnet" "my-subnet-common" {
+  name = "my-subnet-common"
+  prefix = "203.0.113.0/25"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "common"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-common,
+  ]
+}
+EOF
+
+
+

With the command above, we’ve defined 6 resources, 3 of the type of Allocation, 3 of the type of Subnet, each Subnet resource has a different purpose. +For more details, get familiar with the IPAM docs.

+

Now, when we have all the required resources let’s define our Inventory. +We’re going to create 1 SoftGate, 1 switch and connect them with a link.

+
cat << EOF > inventory.tf
+resource "netris_softgate" "my-softgate" {
+  name = "my-softgate"
+  tenantid = data.netris_tenant.admin.id
+  siteid = data.netris_site.default.id
+  description = "Softgate 1"
+  mainip = "auto"
+  mgmtip = "auto"
+  depends_on = [
+    netris_subnet.my-subnet-mgmt,
+    netris_subnet.my-subnet-loopback,
+  ]
+}
+
+resource "netris_switch" "my-switch" {
+  name = "my-switch"
+  tenantid = data.netris_tenant.admin.id
+  siteid = data.netris_site.default.id
+  description = "Switch 01"
+  nos = "cumulus_linux"
+  asnumber = "auto"
+  mainip = "auto"
+  mgmtip = "auto"
+  portcount = 16
+  depends_on = [
+    netris_subnet.my-subnet-mgmt,
+    netris_subnet.my-subnet-loopback,
+  ]
+}
+
+resource "netris_link" "sg-to-sw" {
+  ports = [
+    "swp1@my-softgate",
+    "swp16@my-switch"
+  ]
+  depends_on = [
+    netris_softgate.my-softgate,
+    netris_switch.my-switch,
+  ]
+}
+EOF
+
+
+

Next, let’s define a local L3 network for our servers, suppose we want to connect 3 servers to our switch first 3 ports

+
cat << EOF > vnet.tf
+resource "netris_vnet" "my-vnet" {
+  name = "my-vnet"
+  tenantid = data.netris_tenant.admin.id
+  state = "active"
+  sites{
+    id = data.netris_site.default.id
+    gateways {
+      prefix = "203.0.113.1/25"
+    }
+    ports {
+      name = "swp1@my-switch"
+      vlanid = 1050
+    }
+    ports {
+      name = "swp2@my-switch"
+      vlanid = 1050
+    }
+    ports {
+      name = "swp3@my-switch"
+    }
+  }
+  depends_on = [
+    netris_switch.my-switch,
+    netris_subnet.my-subnet-common,
+  ]
+}
+EOF
+
+
+

And finally, we have to provide internet connectivity to our fabric, for that we’ll define BGP resource. Suppose we’re going to connect our ISP cable to the 10th port of our switch, and want to establish the BGP session on our Softgate.

+
cat << EOF > bgp.tf
+data "netris_port" "swp10_my_switch"{
+  name = "swp10@my-switch"
+  depends_on = [netris_switch.my-switch]
+}
+
+resource "netris_bgp" "my-bgp" {
+  name = "my-bgp"
+  siteid = data.netris_site.default.id
+  hardware = "my-softgate"
+  neighboras = 23456
+  portid = data.netris_port.swp10_my_switch.id
+  vlanid = 3000
+  localip = "172.16.0.2/30"
+  remoteip = "172.16.0.1/30"
+  description = "My First BGP"
+  prefixlistinbound = ["deny 127.0.0.0/8 le 32", "permit 0.0.0.0/0 le 24"]
+  prefixlistoutbound = ["permit 192.0.2.0/24", "permit 198.51.100.0/24 le 25", "permit 203.0.113.0/24 le 26"]
+  depends_on = [netris_link.sg-to-sw]
+}
+EOF
+
+
+
+

Note

+

For more information about all resources, how to create and manage them in Terraform, see the provider’s documentation.

+
+

Now, when we’ve done with the configuration files, let’s check whether they are valid

+
terraform validate
+
+
+

If the configuration is valid, the following message is returned:

+
Success! The configuration is valid.
+
+
+
+
+

Create resources

+
    +
  1. After preparing and checking the configuration, run the command:

  2. +
+
terraform plan
+
+
+

The terminal will display a list of resources with parameters. This is a test step. No resources are created. If there are errors in the configuration, Terraform points them out.

+
    +
  1. To create resources, run the command:

  2. +
+
terraform apply
+
+
+
    +
  1. Confirm the resource creation: type yes in the terminal and press Enter.

  2. +
+

Terraform will create all the required resources and the terminal will display the progress. After creation, you can check resource availability and their settings in the Netris-Controller UI.

+
+
+

Delete resources

+
    +
  1. To delete resources created using Terraform:

  2. +
+

Run the command:

+
terraform destroy
+
+
+

After the command is executed, the terminal will display a list of resources to be deleted.

+
    +
  1. Type yes to confirm their deletion and press Enter.

  2. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/topology-management.html b/en/3.0/topology-management.html new file mode 100644 index 0000000000..dd8c36bbf8 --- /dev/null +++ b/en/3.0/topology-management.html @@ -0,0 +1,769 @@ + + + + + + + + + + + Inventory — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Inventory

+

The Inventory section allows you to add/edit/delete network switches and SoftGates. Initial setup of a Netris managed network is a three part process:

+
    +
  1. Create Inventory Profiles

  2. +
  3. Adding Switches

  4. +
  5. Adding Softgates

  6. +
+
+

Inventory Profiles

+

Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/SoftGate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except Netris-defined and user-defined custom flows. Generated rules include:

+
    +
  • SSH from user defined subnets

  • +
  • NTP from user defined ntp services

  • +
  • DNS from user defined DNS servers

  • +
  • Custom user defined rules

  • +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + +
Inventory Profile Fields

Name

Profile name

Description

Free text description

Allow SSH from IPv4

List of IPv4 subnets allowed to ssh (one address per line)

Allow SSH from IPv6

List of IPv6 subnets allowed to ssh (one address per line)

Timezone

Devices using this inventory profile will adjust their system time to the selected timezone.

NTP servers

List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate.

DNS servers

List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate.

+

Example: In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup).

+_images/inventory-profile.png +
+
+

Adding Switches

+

Every switch needs to be added to the Netris Controller inventory. You can add new devices with the following process:

+
    +
  1. Navigate to Net→Inventory

  2. +
  3. Click the Add button

  4. +
  5. Fill in the fields as described below

  6. +
  7. Click the Add button

  8. +
+
+
+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Add Inventory Fields - Switch

Name

Name of the device

Owner Tenant

Owner tenant of the device

Description

Description of the device

Type

There are 3 types of devices that users can add/edit - Switch/Softgate/Controller. Other types are added automatically when creating services like ROH

NOS

Operating system of the device; applicable for switches only

Site

Site where the devices reside

AS Number

Private AS number of the device; applicable for switches only; recommended to be assigned automatically

Profile

Inventory profile for current device. Profiles are used for security hardening the devices

Main IP address

Main loopback IP address for the device. Can be configured manually or assigned automatically from subnet with loopback purpose defined for current site.

Management IP address

Management IP address for the device. Can be configured manually or assigned automatically from subnet with management purpose defined for current site. This IP address is configured on the out-of-band management interface of the device.

MAC address

MAC address of the device; applicable for switches only

Preliminary port count

Used for definition of topology. When the device registers with the controller the real ports are synced with inventory

Add Link

Provides functionality to define the connections between devices; mandatory for Switch and Softgate physical interconnections

+
+

Example: Add a new Switch.

+
+
_images/add-new-hardware.png +
+
+

Note

+

Repeat this process to define all your switches.

+
+
+
+

Adding SoftGates

+

Every SoftGate node needs to be added to the Netris Controller inventory. To add a SoftGate node:

+
    +
  1. Navigate to Net→Topology

  2. +
  3. Click Add

  4. +
  5. Fill in the fields as described below

  6. +
  7. Click the Add button

  8. +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Add Inventory Fields - SoftGate

Name

Descriptive name

Owner Tenant

Tenant(typically Admin); who administers this node

Description

Free text description

Hardware Type

Select SoftGate

Site

The data center where the current SoftGate node belongs.

Inventory Profile

Profile describing the timezone; DNS; NTP; and Security features

IP Address

IPv4 address for the loopback interface

Management IP address

IPv4 address for the out of band management interface

NFV Node Port

A physical port on a spine switch where the SoftGate node’s first SmartNIC port is connected. Typically each spine switch has one SoftGate node connected to it.

+NAT address

Public IP addresses to be used as global IP for SNAT/DNAT. (check Enabling NAT section of Network Policies chapter)

+NAT address pool

Public IP address subnets to be used as rolling global IP addresses for SNAT. (check Enabling NAT section of Network Policies chapter)

+

Example: Adding a SoftGate Node to Topology.

+_images/add-softgate.png +
+
+

Viewing Inventory

+

Inventory Listing shows also Heartbeat and monitoring statuses of each device.

+

Heartbeat - Shows the status of device reachability. +Health - Shows number of successful and failed checks on the device.

+
+
_images/inventory-listing.png +
+
+

Note

+

You can also add new devices in the Topology view.

+
+
+
+
+

Topology Manager

+

The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents will configure the underlying network devices according to this topology dynamically and start watching against potential failures.

+ + +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/activating-bgp-on-equinix-metal-project.html b/en/3.0/tutorials/activating-bgp-on-equinix-metal-project.html new file mode 100644 index 0000000000..7eea8d96ee --- /dev/null +++ b/en/3.0/tutorials/activating-bgp-on-equinix-metal-project.html @@ -0,0 +1,561 @@ + + + + + + + + + + Activating BGP on Equinix Metal Project — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Activating BGP on Equinix Metal Project

+

Why use BGP with Equinix Metal? +SoftGate nodes are like border routers to your VPC, they are routing traffic between hosts inside your project and the Internet. We are going to establish 2 BGP sessions between SoftGate nodes and Equinix Metal. So there will be 4 BGP sessions total.

+

We need these BGP sessions for moving further. In the next chapters we are going to request pools of public IP addresses, that Netris will automatically advertise to Equinix Metal, so inbound traffic “knows” how to reach Load Balancer, NAT, and other services that you will use within your VPC.

+../_images/equinix-metal-bgp-diagram.png +

You only need to activate BGP on the Equinix Metal Project. Netris will handle the rest. +In the Equinix Metal web console go to IPs & Networks → BGP then click Activate BGP on This Project. (see below screenshots)

+../_images/equinix-metal-activate-bgp.png +

Netris will handle the rest behind the scenes automatically. Netris will enable BGP peering on the Equinix Metal side, Netris will pull the metadata with the BGP info, and will automatically configure FRR (Free Range Routing BGP daemon) on both SoftGate nodes to bring up the BGP sessions up.

+

After a few minutes you should see 4 new BGP sessions in your Netris web console under Net → E-BGP. (example screenshot below).

+../_images/equinix-metal-netris-bgp-up.png +

Now your Netris VPC has established BGP sessions with Equinix Metal Project, and you can proceed to the next step.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html b/en/3.0/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html new file mode 100644 index 0000000000..e5e4bf4f97 --- /dev/null +++ b/en/3.0/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html @@ -0,0 +1,574 @@ + + + + + + + + + + Provisioning Netris SoftGate nodes in Equinix Metal Project — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Provisioning Netris SoftGate nodes in Equinix Metal Project

+

For SoftGate nodes you can start with two c3.small.x86 or larger servers. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one.

+

Request two servers(c3.small.x86) from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned.

+
    +
  1. At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory

  2. +
+../_images/softgate-nodes-created-in-equinix.png +
    +
  1. When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”.

    +
    +

    Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location).

    +
    +
  2. +
+

Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+../_images/softgate-nodes-recognized-in-netris.png +
    +
  1. Provision SoftGate nodes.

  2. +
+

Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there.

+../_images/softgate-one-liner-provisioning.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/softgate-green.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/enable-services-on-equinix-metal-project.html b/en/3.0/tutorials/enable-services-on-equinix-metal-project.html new file mode 100644 index 0000000000..64620535a3 --- /dev/null +++ b/en/3.0/tutorials/enable-services-on-equinix-metal-project.html @@ -0,0 +1,584 @@ + + + + + + + + + + Enabling services (NAT, V-Net, Load Balancer, IP pools) — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Enabling services (NAT, V-Net, Load Balancer, IP pools)

+

Although bare metal servers in Equinix Metal Project get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities.

+

Both NAT and on-demand Load Balancer services need public IP addresses.

+
+

1) Requesting new Public IP address block

+

Go to Equinix Metal web console and click on IPs & Networks → IPs (see the screenshot below)

+

In this example, I’m requesting two IP address blocks, one /30 (4 IPs) for NAT and one /28 (16 IPs) for Load Balancer.

+

It’s important to tag IP blocks as “netris”. This is a signal for Netris Controller that this block is intended for Netris VPC.

+

You can always request more IP address blocks in the future. Also it is possible to request a large block and then use Netris IPAM for crushing it into smaller blocks. You can read more about Netris IPAM in Netris docs.

+../_images/equinix-metal-request-ip-block.png +

Once IP address blocks are provisioned on Equinix Metal Project you should be able to find them automatically replicated in Netris web console under Net → IPAM

+../_images/equinix-metal-netris-ipam-synced.png +

You don’t need to worry about advertising them over BGP, Netris will handle that automatically when that makes sense (associated with any service).

+
+
+

2) Enable on-demand (elastic) Load Balancer

+

To Enable on-demand (elastic) Load Balancer you only need to change the “purpose” field of appropriate IP address block from “common” into “load-balancer”

+

Click on the 3 dots menu (in this example of /28 IP address block), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field.

+../_images/netris-enable-elb.png +

Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform.

+

Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it.

+
+
+

3) Enable V-Net

+

V-Net is a service for virtual private networks. You need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources.

+../_images/netris-create-common-subnets.png +
+
+

4) Enable NAT

+

To enable NAT, you need to repurpose a block of IP addresses for NAT. In the below example I’m repurposing the newly requested /30 subnet for NAT.

+../_images/netris-ipam-nat.png +

Then You need to create a NAT rule. In the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. In this example I’m enabling SNAT for the entire 10.0.0.0/8 private network, so basically I just want to ensure that VMs that will get IPs from private networks will get outbound Internet access through NAT. You can always have more granular control either through NAT rule or using Services → ACLs.

+../_images/netris-create-nat-rule.png +

At this point the minimal configuration of Netris VPC networking is done, next chapters will describe how to consume the VPC, how to request resources and services.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/equinix-metal-api-integration-enablement.html b/en/3.0/tutorials/equinix-metal-api-integration-enablement.html new file mode 100644 index 0000000000..ced6b9cca2 --- /dev/null +++ b/en/3.0/tutorials/equinix-metal-api-integration-enablement.html @@ -0,0 +1,588 @@ + + + + + + + + + + + Equinix Metal API integration enablement — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Equinix Metal API integration enablement

+

For each Equinix Metal Project+location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “Equinix Metal” from the dropdown menu.

Name

Type a descriptive name for your Equinix Metal Project+location.

Equinix Project ID

Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID.

Equinix Project API key

Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here.

Equinix Location

Select your equinix location from the dropdown menu.

+

Equinix Metal Project ID

+../_images/equinix-metal-project-id.png +

Equinix Metal Project API key

+../_images/equinix-metal-project-api-keys.png +

Netris Create New Site

+../_images/netris-create-equinix-metal-site.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/getting-started-for-equinix-metal.html b/en/3.0/tutorials/getting-started-for-equinix-metal.html new file mode 100644 index 0000000000..ba33e8d0c4 --- /dev/null +++ b/en/3.0/tutorials/getting-started-for-equinix-metal.html @@ -0,0 +1,598 @@ + + + + + + + + + + + Equinix Metal API integration enablement — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Equinix Metal API integration enablement
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Equinix Metal API integration enablement

+

For each Equinix Metal Project+location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “Equinix Metal” from the dropdown menu.

Name

Type a descriptive name for your Equinix Metal Project+location.

Equinix Project ID

Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID.

Equinix Project API key

Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here.

Equinix Location

Select your equinix location from the dropdown menu.

+

Equinix Metal Project ID

+../_images/equinix-metal-project-id.png +

Equinix Metal Project API key

+../_images/equinix-metal-project-api-keys.png +

Netris Create New Site

+../_images/netris-create-equinix-metal-site.png +
+

Adding Netris SoftGate nodes

+

For SoftGate nodes you can start with two servers of the smallest flavor. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one.

+

Request two servers from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned.

+
    +
  1. At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory

  2. +
+../_images/softgate-nodes-created-in-equinix.png +
    +
  1. When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”.

    +
    +

    Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location).

    +
    +
  2. +
+

Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+../_images/softgate-nodes-recognized-in-netris.png +
    +
  1. Provision SoftGate nodes.

  2. +
+

Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there.

+../_images/softgate-one-liner-provisioning.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/softgate-green.png +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/index.html b/en/3.0/tutorials/index.html new file mode 100644 index 0000000000..5b2dd1ad96 --- /dev/null +++ b/en/3.0/tutorials/index.html @@ -0,0 +1,583 @@ + + + + + + + + + + Generic Netris Tutorials — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Generic Netris Tutorials
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/installing-netris-controller.html b/en/3.0/tutorials/installing-netris-controller.html new file mode 100644 index 0000000000..1851212723 --- /dev/null +++ b/en/3.0/tutorials/installing-netris-controller.html @@ -0,0 +1,571 @@ + + + + + + + + + + + Installing a Netris Controller — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Installing a Netris Controller

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. 1) So you can access the web console. 2) Nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface.

+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

In this example my host has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

I’m using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my host : 54.219.211.71.

+../_images/cloudflare-dns-record1.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+

To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. For more details, get familiar with the controller installation doc.

+
curl -sfL https://get.netris.ai | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+

Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+

Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/upgrading-netris.html b/en/3.0/tutorials/upgrading-netris.html new file mode 100644 index 0000000000..f84f53cded --- /dev/null +++ b/en/3.0/tutorials/upgrading-netris.html @@ -0,0 +1,688 @@ + + + + + + + + + + + Netris Upgrade and Rollback Procedures — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ + +
+

Netris Upgrade and Rollback Procedures

+
+

Upgrade Procedure

+

Due to potential database structural changes between Netris versions, it’s highly recommended to take a backup of the database before upgrading. The backup will be used in the unlikely event of the need to perform a rollback.

+
    +
  1. To create a database backup, run the following command by first SSHing the Controller:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot.sql
+
+
+

Ensure that SQL file db-snapshot.sql is generated and present in the current directory.

+
    +
  1. Stop all Netris agents on devices managed by the controller (switch & SoftGate).

  2. +
+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl stop netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl stop netris-sg
+
+
+

Ensure that all devices in the Net → Inventory section are “red” with the “check_agent” status being “Agent is unavailable”.

+
+

Note

+

A stopped Netris agent has no impact on production traffic through the device.

+
+
    +
  1. Before upgrading the Netris Controller, take a note of the “Netris Version” by navigating to Setting → General in the Controller web interface. This version number will be used in case of the need to perform a rollback procedure.

  2. +
+Netris Version Example +
    +
  1. Start the upgrade of the Netris Controller using the one-liner after SSHing to the Controller.

  2. +
+
curl -sfL https://get.netris.ai | sh -
+
+
+
+

Note

+

This process can take up to 5 minutes

+
+

Afterwards, make sure that all pods have either “Running” or “Completed” status by executing the following command:

+
kubectl -n netris-controller get pods
+
+
+

The output is similar to this:

+
NAME                                                      READY   STATUS      RESTARTS    AGE
+svclb-netris-controller-haproxy-6tkgj                     4/4     Running     0           38d
+netris-controller-haproxy-bcb944b7c-qcbf8                 1/1     Running     0           13d
+netris-controller-squid-7f6fdc6cf9-7fdx8                  1/1     Running     0           38d
+svclb-netris-controller-squid-58rnp                       1/1     Running     0           38d
+netris-controller-graphite-0                              1/1     Running     0           38d
+netris-controller-mongodb-0                               1/1     Running     0           38d
+netris-controller-redis-master-0                          1/1     Running     0           38d
+netris-controller-smtp-76778cf85f-lw5v5                   1/1     Running     0           10d
+netris-controller-mariadb-0                               1/1     Running     0           10d
+netris-controller-web-session-generator-8b9dbbcd8-8snhd   1/1     Running     0           10d
+netris-controller-telescope-notifier-647975848f-fs5dn     1/1     Running     0           10d
+netris-controller-app-b9b8d8f8d-4ssqb                     1/1     Running     0           10d
+netris-controller-grpc-987669fb9-jjskp                    1/1     Running     0           10d
+netris-controller-telescope-777c98c5d9-mqwl6              1/1     Running     0           10d
+helm-install-netris-controller-lqmq7                      0/1     Completed   0           20h
+
+
+
+

Warning

+

If, after 5 minutes, you see pods with a status other than “Running” or “Completed”, please reach out to us via Slack.

+
+

Then verify that the “Netris Version” reflects the version change by navigating to Setting → General in the Controller web interface.

+
    +
  1. Once you have verified that the Netris controller is up-to-date, it is time to update the switch and SoftGate agents.

  2. +
+

Upgrade the switch & SoftGate agents by copying the one-liner from the “Install Agent” option of the device’s 3-dot menu found under the Net → Inventory section and pasting it after SSHing to the corresponding device.

+Install Agent +

After all the agents have finished the upgrade process, make sure all devices in the Net → Inventory section have a “green” status and the Netris version for each device reflects the version change.

+

In the event the “check_agent” status is “Agent is unavailable” after the agent upgrade has finished, perform agent restart on the affected device(s).

+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl restart netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl restart netris-sg
+
+
+
+
+

Rollback Procedure

+

A rollback procedure can be executed in the event the upgrade introduced any adverse impact on the production traffic.

+
    +
  1. Stop all Netris agents on the devices managed by the controller (switch & SoftGate).

  2. +
+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl stop netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl stop netris-sg
+
+
+
    +
  1. Restore the database from the previously taken snapshot.

  2. +
+

Copy the backup file from the controller host system to the MariaDB container by running the following command after SSHing to the Controller:

+
kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql
+
+
+

While still connected to the Controller, restore the database:

+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql'
+
+
+
    +
  1. Downgrade Netris Controller application with the following command.

  2. +
+
+

Note

+

For the version number, use the number collected from step #3 during the upgrade procedure.

+
+

Example:

+
curl -sfL https://get.netris.ai | sh -s -- --ctl-version 3.0.10-031
+
+
+

Afterwards, verify that the version of the “Netris Version” reflects the downgraded version by navigating to Setting → General in the Netris Controller.

+
    +
  1. Once you have verified that the Netris controller has been downgraded to the correct version, it is time to downgrade the switch and SoftGate agents.

  2. +
+

Install the correct and appropriate versions of the switch & SoftGate agents simply by copying the one-liner from the “Install Agent” option of the device’s 3-dot menu found under the Net → Inventory section and pasting it after SSHing to the corresponding device.

+

After all the switches and SoftGates have been successfully downgraded, make sure all the devices in the Net → Inventory section have a “green” status and the Netris version for each device reflects the version downgrade.

+

In case the “check_agent” status is “Agent is unavailable” after agent downgrade, perform agent restart.

+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl restart netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl restart netris-sg
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/using-l4-load-balancer.html b/en/3.0/tutorials/using-l4-load-balancer.html new file mode 100644 index 0000000000..e0c5a81e80 --- /dev/null +++ b/en/3.0/tutorials/using-l4-load-balancer.html @@ -0,0 +1,554 @@ + + + + + + + + + + Using on-demand (elastic) L4 Load Balancer service — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using on-demand (elastic) L4 Load Balancer service

+

Services –> L4 Load Balancer is an on-demand (elastic) server load balancer. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console.

+../_images/netris-l4-load-balancer.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/tutorials/using-vnet-in-equinix-metal-project.html b/en/3.0/tutorials/using-vnet-in-equinix-metal-project.html new file mode 100644 index 0000000000..5d81985e77 --- /dev/null +++ b/en/3.0/tutorials/using-vnet-in-equinix-metal-project.html @@ -0,0 +1,557 @@ + + + + + + + + + + Using V-Net (isolated virtual network) services in Equinix Metal Project — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using V-Net (isolated virtual network) services in Equinix Metal Project

+

V-Net, under Services –> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from Equinix Metal Project into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network.

+

Netris V-Net (in Equinix Metal scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Layer-2 network using Equinix Metal API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Layer-2 network. SoftGate nodes are the default gateway for the V-Net services.

+

You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, Equinix, and your Compute.

+../_images/netris-creating-vnet-for-equinix-metal.png +

In this example, the new V-NET has VLAN ID 2, subnet 10.0.0.0/24, and gateway 10.0.0.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 2, and they should use IP addresses from 10.0.0.2-254 pointing to 10.0.0.1 as the default gateway. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 2 will have Internet access over the NAT.

+../_images/netris-vnet-ready-in-equinix-metal.png +

Note that you can use Services –> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other. remote sites)

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/visibility.html b/en/3.0/visibility.html new file mode 100644 index 0000000000..83a5ab670b --- /dev/null +++ b/en/3.0/visibility.html @@ -0,0 +1,606 @@ + + + + + + + + + + + Visibility (Telescope) — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Visibility (Telescope)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Visibility (Telescope)

+
+

Graph Boards

+

You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view.

+

To start with Graph Boards, first, you need to add a new Graph Board.

+
    +
  1. Navigate to Telescope→Graph Boards, open the dropdown menu in the top left corner, then click +Add board.

  2. +
+_images/telescope.png +
    +
  1. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants.

  2. +
+_images/createboard.png +

Now you can add graphs by clicking +Add graph.

+

Description of +Add graph fields:

+
    +
  • Title - Title for the new graph.

  • +
  • Type - Type of data source.

    +
      +
    • Bps - Traffic bits per second.

    • +
    • Pps - Traffic packets per second.

    • +
    • Errors - Errors per second.

    • +
    • Optical - Optical signal statistics/history.

    • +
    • MAC Count - History of the number of MAC addresses on the port.

    • +
    +
  • +
  • Function - Currently, only summing is supported.

  • +
  • +Member - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port.

  • +
+

Example: Sum of traffic on two ISP(Iris1 + Iris2) links.

+_images/ISP_Iris.png +

Example: Sum of the traffic on all ports under the service called “my V-NET”

+_images/V_NET.png +

Screenshot: Listing of a Graph Board with the explanation of the controls.

+_images/graphboard.png +
+
+

API Logs

+

Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type.

+
+
+

Dashboard

+

Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems.

+

Telescope→Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner.

+

Description of the pie charts.

+
    +
  • Hardware Health - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions.

  • +
  • E-BGP - Statuses of external BGP sessions.

  • +
  • LB VIP - Statuses of Load Balancer frontend / VIP availability.

  • +
  • LB Members - Statuses of Load Balancer backend members.

  • +
+

By clicking on each title you can see the details of the checks on the right side.

+

Screenshot: Dashboard showing details of “Hardware Health.”

+_images/hardware_health.png +

Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state.

+

Screenshot: “Save as normal” on selected ports.

+_images/saveasnormal.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.0/vnet.html b/en/3.0/vnet.html new file mode 100644 index 0000000000..1d48b672e2 --- /dev/null +++ b/en/3.0/vnet.html @@ -0,0 +1,582 @@ + + + + + + + + + + + V-Net — Netris 3.0 Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

V-Net

+

V-Net is a virtual networking service that provide a Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or they can be created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization). +Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes.

+
+

V-Net Fields

+
    +
  • Name - Unique name for the V-Net

  • +
  • Owner - Tenant, who can make any changes to current V-Net

  • +
  • V-Net state - Active/Disable state for entire V-Net

  • +
  • VLAN aware - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible

  • +
  • Guest tenants - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters

  • +
  • Sites - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites).

  • +
  • IPv4 Gateway - IPv4 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site where V-Net is intended to span.

  • +
  • IPv6 Gateway - IPv6 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site or sites where V-Net is intended to span.

  • +
  • Port - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports.

    +
      +
    • Enabled - Enable or disable individual Switch Port under current V-Net

    • +
    • Port Name - Switch Port format: <alias>(swp<number>)@<switch name>

    • +
    • VLAN ID / Untag - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port’s ingress/egress unless VLAN aware mode is used.

    • +
    • LAG Mode - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes).

    • +
    +
  • +
+
+

Tip

+

Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+
+_images/add-vnet.png +

+Example: Adding a new V-Net.

_images/list-vnet.png +

+Example: Listing of V-Nets.

_images/list-vnet-expanded.png +

+Example: Expanded view of a V-Net listing.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.0 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+
+ + +
3.4
+ + + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/.buildinfo b/en/3.4/.buildinfo new file mode 100644 index 0000000000..61eed0fea2 --- /dev/null +++ b/en/3.4/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: 23d507a586dcaee5f9996fdb9eaec278 +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/en/3.4/.doctrees/EdgeCore-SONiC-Switch-initial-setup.doctree b/en/3.4/.doctrees/EdgeCore-SONiC-Switch-initial-setup.doctree new file mode 100644 index 0000000000..a9b12eac36 Binary files /dev/null and b/en/3.4/.doctrees/EdgeCore-SONiC-Switch-initial-setup.doctree differ diff --git a/en/3.4/.doctrees/Nvidia-Cumulus-v3.7-Switch-initial-setup.doctree b/en/3.4/.doctrees/Nvidia-Cumulus-v3.7-Switch-initial-setup.doctree new file mode 100644 index 0000000000..f773db9115 Binary files /dev/null and b/en/3.4/.doctrees/Nvidia-Cumulus-v3.7-Switch-initial-setup.doctree differ diff --git a/en/3.4/.doctrees/Nvidia-Cumulus-v5-Switch-initial-setup.doctree b/en/3.4/.doctrees/Nvidia-Cumulus-v5-Switch-initial-setup.doctree new file mode 100644 index 0000000000..8b570649d3 Binary files /dev/null and b/en/3.4/.doctrees/Nvidia-Cumulus-v5-Switch-initial-setup.doctree differ diff --git a/en/3.4/.doctrees/SoftGate-PRO-installation.doctree b/en/3.4/.doctrees/SoftGate-PRO-installation.doctree new file mode 100644 index 0000000000..1cd5276e13 Binary files /dev/null and b/en/3.4/.doctrees/SoftGate-PRO-installation.doctree differ diff --git a/en/3.4/.doctrees/SoftGate-installation.doctree b/en/3.4/.doctrees/SoftGate-installation.doctree new file mode 100644 index 0000000000..85500bd662 Binary files /dev/null and b/en/3.4/.doctrees/SoftGate-installation.doctree differ diff --git a/en/3.4/.doctrees/Ubuntu-SwitchDev-Switch-initial-setup.doctree b/en/3.4/.doctrees/Ubuntu-SwitchDev-Switch-initial-setup.doctree new file mode 100644 index 0000000000..a0b1d168d6 Binary files /dev/null and b/en/3.4/.doctrees/Ubuntu-SwitchDev-Switch-initial-setup.doctree differ diff --git a/en/3.4/.doctrees/accounts.doctree b/en/3.4/.doctrees/accounts.doctree new file mode 100644 index 0000000000..b1110b9109 Binary files /dev/null and b/en/3.4/.doctrees/accounts.doctree differ diff --git a/en/3.4/.doctrees/acls.doctree b/en/3.4/.doctrees/acls.doctree new file mode 100644 index 0000000000..a542b65b01 Binary files /dev/null and b/en/3.4/.doctrees/acls.doctree differ diff --git a/en/3.4/.doctrees/controller-k3s-installation.doctree b/en/3.4/.doctrees/controller-k3s-installation.doctree new file mode 100644 index 0000000000..4539a768d0 Binary files /dev/null and b/en/3.4/.doctrees/controller-k3s-installation.doctree differ diff --git a/en/3.4/.doctrees/controller-k8s-installation.doctree b/en/3.4/.doctrees/controller-k8s-installation.doctree new file mode 100644 index 0000000000..216acfd9a8 Binary files /dev/null and b/en/3.4/.doctrees/controller-k8s-installation.doctree differ diff --git a/en/3.4/.doctrees/controller-k8s-quickstart.doctree b/en/3.4/.doctrees/controller-k8s-quickstart.doctree new file mode 100644 index 0000000000..2052fba9be Binary files /dev/null and b/en/3.4/.doctrees/controller-k8s-quickstart.doctree differ diff --git a/en/3.4/.doctrees/controller-vm-installation.doctree b/en/3.4/.doctrees/controller-vm-installation.doctree new file mode 100644 index 0000000000..03d887a76a Binary files /dev/null and b/en/3.4/.doctrees/controller-vm-installation.doctree differ diff --git a/en/3.4/.doctrees/definitions.doctree b/en/3.4/.doctrees/definitions.doctree new file mode 100644 index 0000000000..ad9a04c62e Binary files /dev/null and b/en/3.4/.doctrees/definitions.doctree differ diff --git a/en/3.4/.doctrees/environment.pickle b/en/3.4/.doctrees/environment.pickle new file mode 100644 index 0000000000..d429761f82 Binary files /dev/null and b/en/3.4/.doctrees/environment.pickle differ diff --git a/en/3.4/.doctrees/index.doctree b/en/3.4/.doctrees/index.doctree new file mode 100644 index 0000000000..bad2ff7e9a Binary files /dev/null and b/en/3.4/.doctrees/index.doctree differ diff --git a/en/3.4/.doctrees/installation.doctree b/en/3.4/.doctrees/installation.doctree new file mode 100644 index 0000000000..d736b47b21 Binary files /dev/null and b/en/3.4/.doctrees/installation.doctree differ diff --git a/en/3.4/.doctrees/installing-netris-controller.doctree b/en/3.4/.doctrees/installing-netris-controller.doctree new file mode 100644 index 0000000000..b670343cc6 Binary files /dev/null and b/en/3.4/.doctrees/installing-netris-controller.doctree differ diff --git a/en/3.4/.doctrees/introduction.doctree b/en/3.4/.doctrees/introduction.doctree new file mode 100644 index 0000000000..42f08ca00e Binary files /dev/null and b/en/3.4/.doctrees/introduction.doctree differ diff --git a/en/3.4/.doctrees/inventory-profiles.doctree b/en/3.4/.doctrees/inventory-profiles.doctree new file mode 100644 index 0000000000..75abb37c6f Binary files /dev/null and b/en/3.4/.doctrees/inventory-profiles.doctree differ diff --git a/en/3.4/.doctrees/ipam.doctree b/en/3.4/.doctrees/ipam.doctree new file mode 100644 index 0000000000..ceeecb8543 Binary files /dev/null and b/en/3.4/.doctrees/ipam.doctree differ diff --git a/en/3.4/.doctrees/kubernetes-integration.doctree b/en/3.4/.doctrees/kubernetes-integration.doctree new file mode 100644 index 0000000000..a254b0f8a4 Binary files /dev/null and b/en/3.4/.doctrees/kubernetes-integration.doctree differ diff --git a/en/3.4/.doctrees/l3-load-balancer.doctree b/en/3.4/.doctrees/l3-load-balancer.doctree new file mode 100644 index 0000000000..9cc353f6f5 Binary files /dev/null and b/en/3.4/.doctrees/l3-load-balancer.doctree differ diff --git a/en/3.4/.doctrees/l4-load-balancer.doctree b/en/3.4/.doctrees/l4-load-balancer.doctree new file mode 100644 index 0000000000..5f3038dc64 Binary files /dev/null and b/en/3.4/.doctrees/l4-load-balancer.doctree differ diff --git a/en/3.4/.doctrees/netris-architecture.doctree b/en/3.4/.doctrees/netris-architecture.doctree new file mode 100644 index 0000000000..8fcb0a95c3 Binary files /dev/null and b/en/3.4/.doctrees/netris-architecture.doctree differ diff --git a/en/3.4/.doctrees/network-policies.doctree b/en/3.4/.doctrees/network-policies.doctree new file mode 100644 index 0000000000..9685e957fc Binary files /dev/null and b/en/3.4/.doctrees/network-policies.doctree differ diff --git a/en/3.4/.doctrees/reference-designs.doctree b/en/3.4/.doctrees/reference-designs.doctree new file mode 100644 index 0000000000..7242a6ba41 Binary files /dev/null and b/en/3.4/.doctrees/reference-designs.doctree differ diff --git a/en/3.4/.doctrees/release-notes.doctree b/en/3.4/.doctrees/release-notes.doctree new file mode 100644 index 0000000000..0111fa4795 Binary files /dev/null and b/en/3.4/.doctrees/release-notes.doctree differ diff --git a/en/3.4/.doctrees/roh.doctree b/en/3.4/.doctrees/roh.doctree new file mode 100644 index 0000000000..3664e2b3fa Binary files /dev/null and b/en/3.4/.doctrees/roh.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox1/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox1/configurations.doctree new file mode 100644 index 0000000000..8f256d6193 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox1/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox1/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox1/creating-services.doctree new file mode 100644 index 0000000000..a27c8a241b Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox1/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox1/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox1/index.doctree new file mode 100644 index 0000000000..bfbfa4ef91 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox1/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree new file mode 100644 index 0000000000..d1f9ba317f Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox1/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox1/sandbox-info.doctree new file mode 100644 index 0000000000..04a712466c Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox1/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox10/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox10/configurations.doctree new file mode 100644 index 0000000000..8017c6444f Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox10/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox10/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox10/creating-services.doctree new file mode 100644 index 0000000000..ac2db9ed47 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox10/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox10/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox10/index.doctree new file mode 100644 index 0000000000..3959348855 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox10/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree new file mode 100644 index 0000000000..d3e214a7d2 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox10/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox10/sandbox-info.doctree new file mode 100644 index 0000000000..8ee1ec15d5 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox10/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox11/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox11/configurations.doctree new file mode 100644 index 0000000000..69eddb35e3 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox11/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox11/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox11/creating-services.doctree new file mode 100644 index 0000000000..4584b97f71 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox11/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox11/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox11/index.doctree new file mode 100644 index 0000000000..1f9aba3b30 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox11/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree new file mode 100644 index 0000000000..ce98de5250 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox11/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox11/sandbox-info.doctree new file mode 100644 index 0000000000..06b8c8750e Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox11/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox12/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox12/configurations.doctree new file mode 100644 index 0000000000..7699a6c1a9 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox12/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox12/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox12/creating-services.doctree new file mode 100644 index 0000000000..fe16c5a7ab Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox12/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox12/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox12/index.doctree new file mode 100644 index 0000000000..4c3f4ad04b Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox12/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree new file mode 100644 index 0000000000..a460ac44c1 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox12/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox12/sandbox-info.doctree new file mode 100644 index 0000000000..3e45fbc9f3 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox12/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox13/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox13/configurations.doctree new file mode 100644 index 0000000000..76179a0eaf Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox13/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox13/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox13/creating-services.doctree new file mode 100644 index 0000000000..5354b722ab Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox13/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox13/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox13/index.doctree new file mode 100644 index 0000000000..f5affbdf90 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox13/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree new file mode 100644 index 0000000000..9cd35cc725 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox13/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox13/sandbox-info.doctree new file mode 100644 index 0000000000..3e506dc06d Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox13/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox14/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox14/configurations.doctree new file mode 100644 index 0000000000..4533e33877 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox14/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox14/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox14/creating-services.doctree new file mode 100644 index 0000000000..f92a43ac53 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox14/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox14/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox14/index.doctree new file mode 100644 index 0000000000..f2897077c7 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox14/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree new file mode 100644 index 0000000000..fcbac2ae69 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox14/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox14/sandbox-info.doctree new file mode 100644 index 0000000000..ba192859e5 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox14/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox15/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox15/configurations.doctree new file mode 100644 index 0000000000..4088e5f0df Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox15/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox15/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox15/creating-services.doctree new file mode 100644 index 0000000000..6354537171 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox15/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox15/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox15/index.doctree new file mode 100644 index 0000000000..edb1910acf Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox15/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree new file mode 100644 index 0000000000..058d01746c Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox15/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox15/sandbox-info.doctree new file mode 100644 index 0000000000..f5d323180a Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox15/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox2/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox2/configurations.doctree new file mode 100644 index 0000000000..82ed732e8b Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox2/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox2/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox2/creating-services.doctree new file mode 100644 index 0000000000..6775b0fe51 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox2/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox2/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox2/index.doctree new file mode 100644 index 0000000000..838c291c04 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox2/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree new file mode 100644 index 0000000000..1a8e6cac90 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox2/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox2/sandbox-info.doctree new file mode 100644 index 0000000000..6941cfd9f9 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox2/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox3/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox3/configurations.doctree new file mode 100644 index 0000000000..93c85ebb37 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox3/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox3/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox3/creating-services.doctree new file mode 100644 index 0000000000..b354a78f60 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox3/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox3/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox3/index.doctree new file mode 100644 index 0000000000..aadce7ac4d Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox3/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree new file mode 100644 index 0000000000..bcec897de7 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox3/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox3/sandbox-info.doctree new file mode 100644 index 0000000000..667334eb3d Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox3/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox4/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox4/configurations.doctree new file mode 100644 index 0000000000..ab948ed7e7 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox4/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox4/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox4/creating-services.doctree new file mode 100644 index 0000000000..6dc67f2cda Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox4/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox4/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox4/index.doctree new file mode 100644 index 0000000000..14c3139c1e Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox4/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree new file mode 100644 index 0000000000..815f733eb2 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox4/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox4/sandbox-info.doctree new file mode 100644 index 0000000000..45a3960adb Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox4/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox5/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox5/configurations.doctree new file mode 100644 index 0000000000..cbe0b34bbb Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox5/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox5/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox5/creating-services.doctree new file mode 100644 index 0000000000..a147284581 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox5/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox5/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox5/index.doctree new file mode 100644 index 0000000000..77802000ae Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox5/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree new file mode 100644 index 0000000000..7b0d9d7ed5 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox5/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox5/sandbox-info.doctree new file mode 100644 index 0000000000..84c591234d Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox5/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox6/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox6/configurations.doctree new file mode 100644 index 0000000000..641af3079a Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox6/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox6/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox6/creating-services.doctree new file mode 100644 index 0000000000..e76cfedab7 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox6/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox6/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox6/index.doctree new file mode 100644 index 0000000000..64df49b411 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox6/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree new file mode 100644 index 0000000000..050750e1ce Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox6/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox6/sandbox-info.doctree new file mode 100644 index 0000000000..9b3cd57b17 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox6/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox7/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox7/configurations.doctree new file mode 100644 index 0000000000..648732c9d2 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox7/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox7/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox7/creating-services.doctree new file mode 100644 index 0000000000..4cabcee626 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox7/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox7/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox7/index.doctree new file mode 100644 index 0000000000..ea0db683d0 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox7/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree new file mode 100644 index 0000000000..7c996dc270 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox7/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox7/sandbox-info.doctree new file mode 100644 index 0000000000..c0140c6148 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox7/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox8/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox8/configurations.doctree new file mode 100644 index 0000000000..101c05d034 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox8/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox8/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox8/creating-services.doctree new file mode 100644 index 0000000000..730ed796f6 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox8/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox8/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox8/index.doctree new file mode 100644 index 0000000000..236cb72b58 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox8/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree new file mode 100644 index 0000000000..6a427a79d8 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox8/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox8/sandbox-info.doctree new file mode 100644 index 0000000000..db018c6b0b Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox8/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox9/configurations.doctree b/en/3.4/.doctrees/sandbox/Sandbox9/configurations.doctree new file mode 100644 index 0000000000..56860241b3 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox9/configurations.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox9/creating-services.doctree b/en/3.4/.doctrees/sandbox/Sandbox9/creating-services.doctree new file mode 100644 index 0000000000..1fadc4345d Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox9/creating-services.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox9/index.doctree b/en/3.4/.doctrees/sandbox/Sandbox9/index.doctree new file mode 100644 index 0000000000..36793d6a86 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox9/index.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree b/en/3.4/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree new file mode 100644 index 0000000000..9e273f1811 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree differ diff --git a/en/3.4/.doctrees/sandbox/Sandbox9/sandbox-info.doctree b/en/3.4/.doctrees/sandbox/Sandbox9/sandbox-info.doctree new file mode 100644 index 0000000000..8c20228788 Binary files /dev/null and b/en/3.4/.doctrees/sandbox/Sandbox9/sandbox-info.doctree differ diff --git a/en/3.4/.doctrees/softgate-performance.doctree b/en/3.4/.doctrees/softgate-performance.doctree new file mode 100644 index 0000000000..eebf6702f3 Binary files /dev/null and b/en/3.4/.doctrees/softgate-performance.doctree differ diff --git a/en/3.4/.doctrees/supported-networks.doctree b/en/3.4/.doctrees/supported-networks.doctree new file mode 100644 index 0000000000..bf7cfd412a Binary files /dev/null and b/en/3.4/.doctrees/supported-networks.doctree differ diff --git a/en/3.4/.doctrees/switch-agent-installation.doctree b/en/3.4/.doctrees/switch-agent-installation.doctree new file mode 100644 index 0000000000..c753d9df58 Binary files /dev/null and b/en/3.4/.doctrees/switch-agent-installation.doctree differ diff --git a/en/3.4/.doctrees/switch-ports.doctree b/en/3.4/.doctrees/switch-ports.doctree new file mode 100644 index 0000000000..2ce7d5b0d2 Binary files /dev/null and b/en/3.4/.doctrees/switch-ports.doctree differ diff --git a/en/3.4/.doctrees/terraform-integration.doctree b/en/3.4/.doctrees/terraform-integration.doctree new file mode 100644 index 0000000000..1e9b325223 Binary files /dev/null and b/en/3.4/.doctrees/terraform-integration.doctree differ diff --git a/en/3.4/.doctrees/topology-management.doctree b/en/3.4/.doctrees/topology-management.doctree new file mode 100644 index 0000000000..ea92d52cd6 Binary files /dev/null and b/en/3.4/.doctrees/topology-management.doctree differ diff --git a/en/3.4/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree b/en/3.4/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree new file mode 100644 index 0000000000..7846919578 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree differ diff --git a/en/3.4/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree b/en/3.4/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree new file mode 100644 index 0000000000..fbf4ca4132 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree differ diff --git a/en/3.4/.doctrees/tutorials/aws-concept.doctree b/en/3.4/.doctrees/tutorials/aws-concept.doctree new file mode 100644 index 0000000000..bc569a08d9 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/aws-concept.doctree differ diff --git a/en/3.4/.doctrees/tutorials/aws-deploy-softgate.doctree b/en/3.4/.doctrees/tutorials/aws-deploy-softgate.doctree new file mode 100644 index 0000000000..2fddcdf6f9 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/aws-deploy-softgate.doctree differ diff --git a/en/3.4/.doctrees/tutorials/create-interconnection-to-fabric.doctree b/en/3.4/.doctrees/tutorials/create-interconnection-to-fabric.doctree new file mode 100644 index 0000000000..52914f32d5 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/create-interconnection-to-fabric.doctree differ diff --git a/en/3.4/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree b/en/3.4/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree new file mode 100644 index 0000000000..4c58592684 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree differ diff --git a/en/3.4/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree b/en/3.4/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree new file mode 100644 index 0000000000..5b6297e3b2 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree differ diff --git a/en/3.4/.doctrees/tutorials/gcp-concept.doctree b/en/3.4/.doctrees/tutorials/gcp-concept.doctree new file mode 100644 index 0000000000..66f1515ff9 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/gcp-concept.doctree differ diff --git a/en/3.4/.doctrees/tutorials/gcp-deploy-softgate.doctree b/en/3.4/.doctrees/tutorials/gcp-deploy-softgate.doctree new file mode 100644 index 0000000000..033498ef51 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/gcp-deploy-softgate.doctree differ diff --git a/en/3.4/.doctrees/tutorials/getting-started-for-equinix-metal.doctree b/en/3.4/.doctrees/tutorials/getting-started-for-equinix-metal.doctree new file mode 100644 index 0000000000..9ba3cb5bbc Binary files /dev/null and b/en/3.4/.doctrees/tutorials/getting-started-for-equinix-metal.doctree differ diff --git a/en/3.4/.doctrees/tutorials/index-equinix.doctree b/en/3.4/.doctrees/tutorials/index-equinix.doctree new file mode 100644 index 0000000000..c4d25b8901 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/index-equinix.doctree differ diff --git a/en/3.4/.doctrees/tutorials/index-vpc.doctree b/en/3.4/.doctrees/tutorials/index-vpc.doctree new file mode 100644 index 0000000000..0b569938ea Binary files /dev/null and b/en/3.4/.doctrees/tutorials/index-vpc.doctree differ diff --git a/en/3.4/.doctrees/tutorials/index.doctree b/en/3.4/.doctrees/tutorials/index.doctree new file mode 100644 index 0000000000..a0cff748d5 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/index.doctree differ diff --git a/en/3.4/.doctrees/tutorials/install-netris-controller-in-equinix-metal.doctree b/en/3.4/.doctrees/tutorials/install-netris-controller-in-equinix-metal.doctree new file mode 100644 index 0000000000..b8f2b1bd21 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/install-netris-controller-in-equinix-metal.doctree differ diff --git a/en/3.4/.doctrees/tutorials/installing-netris-controller.doctree b/en/3.4/.doctrees/tutorials/installing-netris-controller.doctree new file mode 100644 index 0000000000..628f395ac9 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/installing-netris-controller.doctree differ diff --git a/en/3.4/.doctrees/tutorials/netris-vpc-for-aws.doctree b/en/3.4/.doctrees/tutorials/netris-vpc-for-aws.doctree new file mode 100644 index 0000000000..f512ba0bde Binary files /dev/null and b/en/3.4/.doctrees/tutorials/netris-vpc-for-aws.doctree differ diff --git a/en/3.4/.doctrees/tutorials/netris-vpc-for-equinix-metal.doctree b/en/3.4/.doctrees/tutorials/netris-vpc-for-equinix-metal.doctree new file mode 100644 index 0000000000..93537dddf4 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/netris-vpc-for-equinix-metal.doctree differ diff --git a/en/3.4/.doctrees/tutorials/netris-vpc-for-gcp.doctree b/en/3.4/.doctrees/tutorials/netris-vpc-for-gcp.doctree new file mode 100644 index 0000000000..5d5c87e4b5 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/netris-vpc-for-gcp.doctree differ diff --git a/en/3.4/.doctrees/tutorials/netris-vpc-for-phoenixnap-bmc.doctree b/en/3.4/.doctrees/tutorials/netris-vpc-for-phoenixnap-bmc.doctree new file mode 100644 index 0000000000..1f948b9cdb Binary files /dev/null and b/en/3.4/.doctrees/tutorials/netris-vpc-for-phoenixnap-bmc.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.doctree new file mode 100644 index 0000000000..ed1869712f Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-api-integration-enablement.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-api-integration-enablement.doctree new file mode 100644 index 0000000000..4686b17c8d Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-api-integration-enablement.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-concept.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-concept.doctree new file mode 100644 index 0000000000..5844bcb0e4 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-concept.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-install-netris-controller.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-install-netris-controller.doctree new file mode 100644 index 0000000000..7dfc63763b Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-install-netris-controller.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-ipam-setup.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-ipam-setup.doctree new file mode 100644 index 0000000000..119f6aec72 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-ipam-setup.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-link-to-installation.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-link-to-installation.doctree new file mode 100644 index 0000000000..2cfcbd1694 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-link-to-installation.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-l4lb.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-l4lb.doctree new file mode 100644 index 0000000000..9090cdd09f Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-l4lb.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-nat.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-nat.doctree new file mode 100644 index 0000000000..6a89b46f0b Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-nat.doctree differ diff --git a/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-vnet.doctree b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-vnet.doctree new file mode 100644 index 0000000000..90e7a98dd6 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/phoenixnap-bmc-using-vnet.doctree differ diff --git a/en/3.4/.doctrees/tutorials/upgrading-netris.doctree b/en/3.4/.doctrees/tutorials/upgrading-netris.doctree new file mode 100644 index 0000000000..b0c5854861 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/upgrading-netris.doctree differ diff --git a/en/3.4/.doctrees/tutorials/upgrading-sonic-os.doctree b/en/3.4/.doctrees/tutorials/upgrading-sonic-os.doctree new file mode 100644 index 0000000000..80bb1e5106 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/upgrading-sonic-os.doctree differ diff --git a/en/3.4/.doctrees/tutorials/using-l4-load-balancer.doctree b/en/3.4/.doctrees/tutorials/using-l4-load-balancer.doctree new file mode 100644 index 0000000000..06fb4c0b67 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/using-l4-load-balancer.doctree differ diff --git a/en/3.4/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree b/en/3.4/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree new file mode 100644 index 0000000000..21ef790cd5 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-check-default-site.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-check-default-site.doctree new file mode 100644 index 0000000000..f2430e5d6e Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-check-default-site.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-concept.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-concept.doctree new file mode 100644 index 0000000000..3a5275a491 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-concept.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-controller-installation.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-controller-installation.doctree new file mode 100644 index 0000000000..2e0b08425f Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-controller-installation.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-ipam-setup.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-ipam-setup.doctree new file mode 100644 index 0000000000..197a1d7c21 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-ipam-setup.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-softgate-installation.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-softgate-installation.doctree new file mode 100644 index 0000000000..90d41982be Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-softgate-installation.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-upstream-peering.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-upstream-peering.doctree new file mode 100644 index 0000000000..6aa275f9b0 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-upstream-peering.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-using-l4lb.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-l4lb.doctree new file mode 100644 index 0000000000..e08bf3e098 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-l4lb.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-using-multi-interface-softgate.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-multi-interface-softgate.doctree new file mode 100644 index 0000000000..31e4c99555 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-multi-interface-softgate.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-using-nat.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-nat.doctree new file mode 100644 index 0000000000..d8b2f53b84 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-nat.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere-using-vnet.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-vnet.doctree new file mode 100644 index 0000000000..6240d44665 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere-using-vnet.doctree differ diff --git a/en/3.4/.doctrees/tutorials/vpc-anywhere.doctree b/en/3.4/.doctrees/tutorials/vpc-anywhere.doctree new file mode 100644 index 0000000000..3a96e234d3 Binary files /dev/null and b/en/3.4/.doctrees/tutorials/vpc-anywhere.doctree differ diff --git a/en/3.4/.doctrees/visibility.doctree b/en/3.4/.doctrees/visibility.doctree new file mode 100644 index 0000000000..194d9acf1e Binary files /dev/null and b/en/3.4/.doctrees/visibility.doctree differ diff --git a/en/3.4/.doctrees/vnet.doctree b/en/3.4/.doctrees/vnet.doctree new file mode 100644 index 0000000000..22849a3f0e Binary files /dev/null and b/en/3.4/.doctrees/vnet.doctree differ diff --git a/en/3.4/.nojekyll b/en/3.4/.nojekyll new file mode 100644 index 0000000000..e69de29bb2 diff --git a/en/3.4/EdgeCore-SONiC-Switch-initial-setup.html b/en/3.4/EdgeCore-SONiC-Switch-initial-setup.html new file mode 100644 index 0000000000..2b8bd6ba4d --- /dev/null +++ b/en/3.4/EdgeCore-SONiC-Switch-initial-setup.html @@ -0,0 +1,779 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + EdgeCore SONiC Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

EdgeCore SONiC Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first.

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of EC SONiC.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/Edgecore-SONiC_20211125_074752_ec202012_227.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password:

+

admin/YourPaSsWoRd

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Disable ztp:

+
ztp disable -y
+
+
+

Configure the IP address, default gateway, and DNS to establish Internet connectivity via the management port.

+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of management network>
+
+
+
echo "nameserver <dns server>" > /etc/resolv.conf
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory.png +_images/Switch-agent-installation-oneliner.png +_images/Switch-agent-installation-cli.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/Nvidia-Cumulus-v3.7-Switch-initial-setup.html b/en/3.4/Nvidia-Cumulus-v3.7-Switch-initial-setup.html new file mode 100644 index 0000000000..f2602f26e4 --- /dev/null +++ b/en/3.4/Nvidia-Cumulus-v3.7-Switch-initial-setup.html @@ -0,0 +1,795 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Nvidia Cumulus v3.7 Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Nvidia Cumulus v3.7 Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first.

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/cumulus-linux-3.7.15-mlx-amd64.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password for Cumulus v3.7:

+

cumulus/CumulusLinux!

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Open the network interfaces file and add the IP address and other required details.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+       address <management IP address/prefix length>
+       gateway <gateway of management network>
+
+source /etc/network/interfaces.d/*
+
+
+
echo "nameserver <dns server>" | sudo tee /etc/resolv.conf
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Cumulus Linux license installation.

  2. +
+
sudo cl-license -i
+
+
+

Copy/paste the Cumulus Linux license string, then press ctrl-d.

+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory.png +_images/Switch-agent-installation-oneliner.png +_images/Switch-agent-installation-cli.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/Nvidia-Cumulus-v5-Switch-initial-setup.html b/en/3.4/Nvidia-Cumulus-v5-Switch-initial-setup.html new file mode 100644 index 0000000000..0293f5b6d4 --- /dev/null +++ b/en/3.4/Nvidia-Cumulus-v5-Switch-initial-setup.html @@ -0,0 +1,804 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Nvidia Cumulus v5 Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Nvidia Cumulus v5 Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/cumulus-linux-5.4.0-mlx-amd64.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password for Cumulus v5:

+

cumulus/cumulus

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Upon installation of Cumulus Linux v5, the default Virtual Routing and Forwarding (VRF) is set to ‘mgmt.’ To switch to the default VRF, please refer to the following instructions:

+

Disable ztp:

+
sudo ztp -d
+
+
+
sudo ip vrf exec default bash
+
+
+

Open the network interfaces file, add the IP address and other required details, and ensure that you remove the ‘mgmt’ VRF configuration:

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <management IP address/prefix length>
+        gateway <gateway of management network>
+
+source /etc/network/interfaces.d/*
+
+
+
echo "nameserver <dns server>" | sudo tee /etc/resolv.conf
+
+
+
sudo ifreload -a
+
+
+
+

Note

+

You might see a one-time warning in the output of ifreload, which you can ignore:

+
+
warning: mgmt: cmd '/usr/lib/vrf/vrf-helper delete mgmt 1001' failed: returned 1 (Failed to delete cgroup for vrf mgmt)
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory.png +_images/Switch-agent-installation-oneliner.png +_images/Switch-agent-installation-cli.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/SoftGate-PRO-installation.html b/en/3.4/SoftGate-PRO-installation.html new file mode 100644 index 0000000000..7955ee1b1d --- /dev/null +++ b/en/3.4/SoftGate-PRO-installation.html @@ -0,0 +1,781 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate PRO Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • SoftGate PRO Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

SoftGate PRO Installation

+
+

Minimum Hardware Requirements

+
    +
  • 2 x Intel® Xeon® Silver Processor with 10 physical cores per socket (20 cores total)

  • +
  • 128 GB (64 GB RAM per socket) in multichannel configuration

  • +
  • 300 GB HDD

  • +
  • Nvidia Mellanox Connect-X 5/6 SmartNIC card

  • +
+
+
+

BIOS Configuration

+

The following are some recommendations for BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference:

+
    +
  • Before starting consider resetting all BIOS settings to their defaults

  • +
  • Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report

  • +
  • Select Performance as the CPU Power and Performance policy

  • +
  • Enable Turbo Boost

  • +
  • Set memory frequency to the highest available number, NOT auto

  • +
  • Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d

  • +
  • Disable Hyper-Threading

  • +
+
+
+

Install the Netris Agent

+

Requires freshly installed Ubuntu Linux 18.04 LTS and internet connectivity configured from netplan via management port.

+
    +
  1. Add the SoftGate in the controller Inventory or Topology section. Detailed configuration documentation is available here: “Adding SoftGates”.

  2. +
  3. Once the SoftGate is created, navigate to the Inventory section, click the three vertical dots (⋮) on the right side of the newly created SoftGate and select the Install Agent option.

  4. +
  5. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user.

  6. +
  7. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration).

  8. +
+
+

Note

+

If the Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured.

+
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node.
+    up ip route add <Controller address> via <Management network gateway>
+    gateway <Gateway IP address>
+
+ source /etc/network/interfaces.d/*
+
+
+
    +
  1. If everything seems ok, please remove/comment the Gateway line and save the file.

  2. +
+
+

Note

+

Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent.

+
+
    +
  1. Reboot the SoftGate

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/SoftGate-installation.html b/en/3.4/SoftGate-installation.html new file mode 100644 index 0000000000..9b72cda835 --- /dev/null +++ b/en/3.4/SoftGate-installation.html @@ -0,0 +1,791 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • SoftGate Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

SoftGate Installation

+
+

Minimum Hardware Requirements

+
    +
  • 8 CPU cores

  • +
  • 16 GB RAM

  • +
  • 300 GB HDD

  • +
+
+
+

Provision Netris SoftGate software

+

Requires freshly installed Ubuntu Linux 22.04 LTS and internet connectivity.

+
    +
  1. Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: “Adding SoftGates”.

  2. +
  3. Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the SoftGate node you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

  4. +
+_images/softgate-install-agent.png +
    +
  1. Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node)

  2. +
+_images/softgate-provisioning-cli-output.png +
+

Note

+

Please note that Netris replaces Netplan with regular ifupdown and attempts to migrate any prior configuration to /etc/network/interfaces.

+
+
    +
  1. Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.<xyz>). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as you describe in /etc/network/interfaces.

  2. +
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# Physical port on SoftGate node connected to a TRUNK port of your network
+auto ens<X>
+iface ens<x> inet static
+    address 0.0.0.0/0
+
+# Optionally you can add more physical interfaces under your bond0
+auto ens<Y>
+iface ens<Y> inet static
+    address 0.0.0.0/0
+
+# Bond interface
+auto bond0
+iface bond0 inet static
+    address 0.0.0.0/0
+    # Please replace the ensX/Y with actual interface name(s) below to one(s) present in the OS.
+    bond-slaves ens<X> ens<Y>
+    # Optional, please adjust the bonding mode below according to the desired functionality.
+    bond-mode active-backup
+
+source /etc/network/interfaces.d/*
+
+
+
    +
  1. Ensure that SoftGate node will maintain IP connectivity with Netris Controller after reboot.

  2. +
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    # Please delete or comment the line below if Netris Controller is located in the same network with the SoftGate node.
+    up ip route add <Controller address> via <Management network gateway>
+
+
+
    +
  1. Reboot the SoftGate

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up, you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/Ubuntu-SwitchDev-Switch-initial-setup.html b/en/3.4/Ubuntu-SwitchDev-Switch-initial-setup.html new file mode 100644 index 0000000000..00f8efc827 --- /dev/null +++ b/en/3.4/Ubuntu-SwitchDev-Switch-initial-setup.html @@ -0,0 +1,784 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Ubuntu SwitchDev Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Ubuntu SwitchDev Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first.

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of the Ubuntu SwitchDev.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

Install Ubuntu SwitchDev using the Netris customized image:

+
onie-nos-install http://downloads.netris.ai/netris-ubuntu-18.04.1.bin
+
+
+

Default username/password

+

netris/newNet0ps

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Open the network interfaces file and add the IP address and other required details.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <management IP address/prefix length>
+        gateway <gateway of management network>
+        dns-nameserver <dns server>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory-ubuntu.png +_images/Switch-agent-installation-oneliner-ubuntu.png +_images/Switch-agent-installation-cli-ubuntu.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/_images/ACL_active.png b/en/3.4/_images/ACL_active.png new file mode 100644 index 0000000000..6ba98a50b8 Binary files /dev/null and b/en/3.4/_images/ACL_active.png differ diff --git a/en/3.4/_images/ACL_approval.png b/en/3.4/_images/ACL_approval.png new file mode 100644 index 0000000000..6345dea408 Binary files /dev/null and b/en/3.4/_images/ACL_approval.png differ diff --git a/en/3.4/_images/ACL_rule.png b/en/3.4/_images/ACL_rule.png new file mode 100644 index 0000000000..81c08c11db Binary files /dev/null and b/en/3.4/_images/ACL_rule.png differ diff --git a/en/3.4/_images/BGP_neighbors_listing.png b/en/3.4/_images/BGP_neighbors_listing.png new file mode 100644 index 0000000000..de768abbc4 Binary files /dev/null and b/en/3.4/_images/BGP_neighbors_listing.png differ diff --git a/en/3.4/_images/BGP_route.png b/en/3.4/_images/BGP_route.png new file mode 100644 index 0000000000..b2e702fc7e Binary files /dev/null and b/en/3.4/_images/BGP_route.png differ diff --git a/en/3.4/_images/EVPN_routing.png b/en/3.4/_images/EVPN_routing.png new file mode 100644 index 0000000000..c9196538a8 Binary files /dev/null and b/en/3.4/_images/EVPN_routing.png differ diff --git a/en/3.4/_images/IP-allocation.png b/en/3.4/_images/IP-allocation.png new file mode 100644 index 0000000000..a59909eb2c Binary files /dev/null and b/en/3.4/_images/IP-allocation.png differ diff --git a/en/3.4/_images/IPv4-Prefix.png b/en/3.4/_images/IPv4-Prefix.png new file mode 100644 index 0000000000..4726a74662 Binary files /dev/null and b/en/3.4/_images/IPv4-Prefix.png differ diff --git a/en/3.4/_images/IPv6-Prefix.png b/en/3.4/_images/IPv6-Prefix.png new file mode 100644 index 0000000000..c4ab1a9fb0 Binary files /dev/null and b/en/3.4/_images/IPv6-Prefix.png differ diff --git a/en/3.4/_images/ISP_Iris.png b/en/3.4/_images/ISP_Iris.png new file mode 100644 index 0000000000..638bc3a1df Binary files /dev/null and b/en/3.4/_images/ISP_Iris.png differ diff --git a/en/3.4/_images/MAC_listing.png b/en/3.4/_images/MAC_listing.png new file mode 100644 index 0000000000..abe057a8ac Binary files /dev/null and b/en/3.4/_images/MAC_listing.png differ diff --git a/en/3.4/_images/NATIP-address.png b/en/3.4/_images/NATIP-address.png new file mode 100644 index 0000000000..f6a12b62f9 Binary files /dev/null and b/en/3.4/_images/NATIP-address.png differ diff --git a/en/3.4/_images/Port-Forwarding.png b/en/3.4/_images/Port-Forwarding.png new file mode 100644 index 0000000000..fea266b6b3 Binary files /dev/null and b/en/3.4/_images/Port-Forwarding.png differ diff --git a/en/3.4/_images/ROH-instance.png b/en/3.4/_images/ROH-instance.png new file mode 100644 index 0000000000..ca21d16b10 Binary files /dev/null and b/en/3.4/_images/ROH-instance.png differ diff --git a/en/3.4/_images/ROH-listing.png b/en/3.4/_images/ROH-listing.png new file mode 100644 index 0000000000..0e3c74214c Binary files /dev/null and b/en/3.4/_images/ROH-listing.png differ diff --git a/en/3.4/_images/SiteMesh_listing.png b/en/3.4/_images/SiteMesh_listing.png new file mode 100644 index 0000000000..7142f87e52 Binary files /dev/null and b/en/3.4/_images/SiteMesh_listing.png differ diff --git a/en/3.4/_images/SiteMesh_modes.png b/en/3.4/_images/SiteMesh_modes.png new file mode 100644 index 0000000000..5edfe756f0 Binary files /dev/null and b/en/3.4/_images/SiteMesh_modes.png differ diff --git a/en/3.4/_images/Site_Mesh.png b/en/3.4/_images/Site_Mesh.png new file mode 100644 index 0000000000..f9fb780b4d Binary files /dev/null and b/en/3.4/_images/Site_Mesh.png differ diff --git a/en/3.4/_images/Spine1.png b/en/3.4/_images/Spine1.png new file mode 100644 index 0000000000..9871609ae1 Binary files /dev/null and b/en/3.4/_images/Spine1.png differ diff --git a/en/3.4/_images/Switch-agent-installation-Inventory-ubuntu.png b/en/3.4/_images/Switch-agent-installation-Inventory-ubuntu.png new file mode 100644 index 0000000000..17818408a8 Binary files /dev/null and b/en/3.4/_images/Switch-agent-installation-Inventory-ubuntu.png differ diff --git a/en/3.4/_images/Switch-agent-installation-Inventory.png b/en/3.4/_images/Switch-agent-installation-Inventory.png new file mode 100644 index 0000000000..cf99dd239a Binary files /dev/null and b/en/3.4/_images/Switch-agent-installation-Inventory.png differ diff --git a/en/3.4/_images/Switch-agent-installation-cli-ubuntu.png b/en/3.4/_images/Switch-agent-installation-cli-ubuntu.png new file mode 100644 index 0000000000..366ddaa85e Binary files /dev/null and b/en/3.4/_images/Switch-agent-installation-cli-ubuntu.png differ diff --git a/en/3.4/_images/Switch-agent-installation-cli.png b/en/3.4/_images/Switch-agent-installation-cli.png new file mode 100644 index 0000000000..da9b8b4ff1 Binary files /dev/null and b/en/3.4/_images/Switch-agent-installation-cli.png differ diff --git a/en/3.4/_images/Switch-agent-installation-oneliner-ubuntu.png b/en/3.4/_images/Switch-agent-installation-oneliner-ubuntu.png new file mode 100644 index 0000000000..ce08901584 Binary files /dev/null and b/en/3.4/_images/Switch-agent-installation-oneliner-ubuntu.png differ diff --git a/en/3.4/_images/Switch-agent-installation-oneliner.png b/en/3.4/_images/Switch-agent-installation-oneliner.png new file mode 100644 index 0000000000..aae208da96 Binary files /dev/null and b/en/3.4/_images/Switch-agent-installation-oneliner.png differ diff --git a/en/3.4/_images/TCAM.png b/en/3.4/_images/TCAM.png new file mode 100644 index 0000000000..0bba84f117 Binary files /dev/null and b/en/3.4/_images/TCAM.png differ diff --git a/en/3.4/_images/V_NET.png b/en/3.4/_images/V_NET.png new file mode 100644 index 0000000000..74733e071a Binary files /dev/null and b/en/3.4/_images/V_NET.png differ diff --git a/en/3.4/_images/action_permit.png b/en/3.4/_images/action_permit.png new file mode 100644 index 0000000000..c478300133 Binary files /dev/null and b/en/3.4/_images/action_permit.png differ diff --git a/en/3.4/_images/add-allocation.png b/en/3.4/_images/add-allocation.png new file mode 100644 index 0000000000..0d4784990e Binary files /dev/null and b/en/3.4/_images/add-allocation.png differ diff --git a/en/3.4/_images/add-bgp-basic-2.png b/en/3.4/_images/add-bgp-basic-2.png new file mode 100644 index 0000000000..1480062765 Binary files /dev/null and b/en/3.4/_images/add-bgp-basic-2.png differ diff --git a/en/3.4/_images/add-bgp-basic.png b/en/3.4/_images/add-bgp-basic.png new file mode 100644 index 0000000000..be55737f4a Binary files /dev/null and b/en/3.4/_images/add-bgp-basic.png differ diff --git a/en/3.4/_images/add-equinix-bgp-primary.png b/en/3.4/_images/add-equinix-bgp-primary.png new file mode 100644 index 0000000000..2b343ada9b Binary files /dev/null and b/en/3.4/_images/add-equinix-bgp-primary.png differ diff --git a/en/3.4/_images/add-interconnection-request.png b/en/3.4/_images/add-interconnection-request.png new file mode 100644 index 0000000000..de0be61993 Binary files /dev/null and b/en/3.4/_images/add-interconnection-request.png differ diff --git a/en/3.4/_images/add-l3-lb.png b/en/3.4/_images/add-l3-lb.png new file mode 100644 index 0000000000..6760093ee5 Binary files /dev/null and b/en/3.4/_images/add-l3-lb.png differ diff --git a/en/3.4/_images/add-new-ebgp-form.png b/en/3.4/_images/add-new-ebgp-form.png new file mode 100644 index 0000000000..8b771a90c8 Binary files /dev/null and b/en/3.4/_images/add-new-ebgp-form.png differ diff --git a/en/3.4/_images/add-new-hardware.png b/en/3.4/_images/add-new-hardware.png new file mode 100644 index 0000000000..5e00df64a3 Binary files /dev/null and b/en/3.4/_images/add-new-hardware.png differ diff --git a/en/3.4/_images/add-new-vlan-equinix.png b/en/3.4/_images/add-new-vlan-equinix.png new file mode 100644 index 0000000000..241878136e Binary files /dev/null and b/en/3.4/_images/add-new-vlan-equinix.png differ diff --git a/en/3.4/_images/add-softgate.png b/en/3.4/_images/add-softgate.png new file mode 100644 index 0000000000..0ea2b859ac Binary files /dev/null and b/en/3.4/_images/add-softgate.png differ diff --git a/en/3.4/_images/add-subnet.png b/en/3.4/_images/add-subnet.png new file mode 100644 index 0000000000..54586a7099 Binary files /dev/null and b/en/3.4/_images/add-subnet.png differ diff --git a/en/3.4/_images/add-to-lag-port.png b/en/3.4/_images/add-to-lag-port.png new file mode 100644 index 0000000000..479e36897e Binary files /dev/null and b/en/3.4/_images/add-to-lag-port.png differ diff --git a/en/3.4/_images/add-vnet.png b/en/3.4/_images/add-vnet.png new file mode 100644 index 0000000000..4321573d7c Binary files /dev/null and b/en/3.4/_images/add-vnet.png differ diff --git a/en/3.4/_images/approve_reject.png b/en/3.4/_images/approve_reject.png new file mode 100644 index 0000000000..a3054f9912 Binary files /dev/null and b/en/3.4/_images/approve_reject.png differ diff --git a/en/3.4/_images/aws-concept-traffic-flows.png b/en/3.4/_images/aws-concept-traffic-flows.png new file mode 100644 index 0000000000..6f29b595ca Binary files /dev/null and b/en/3.4/_images/aws-concept-traffic-flows.png differ diff --git a/en/3.4/_images/aws-ec2-stop-fwd-check.png b/en/3.4/_images/aws-ec2-stop-fwd-check.png new file mode 100644 index 0000000000..773981a438 Binary files /dev/null and b/en/3.4/_images/aws-ec2-stop-fwd-check.png differ diff --git a/en/3.4/_images/aws-netris-create-sg.png b/en/3.4/_images/aws-netris-create-sg.png new file mode 100644 index 0000000000..83692f9c28 Binary files /dev/null and b/en/3.4/_images/aws-netris-create-sg.png differ diff --git a/en/3.4/_images/aws-netris-enable-site-mesh.png b/en/3.4/_images/aws-netris-enable-site-mesh.png new file mode 100644 index 0000000000..7b72dc4012 Binary files /dev/null and b/en/3.4/_images/aws-netris-enable-site-mesh.png differ diff --git a/en/3.4/_images/aws-netris-ipam-lo.png b/en/3.4/_images/aws-netris-ipam-lo.png new file mode 100644 index 0000000000..4ee3b3ed88 Binary files /dev/null and b/en/3.4/_images/aws-netris-ipam-lo.png differ diff --git a/en/3.4/_images/aws-netris-ipam-mgmt.png b/en/3.4/_images/aws-netris-ipam-mgmt.png new file mode 100644 index 0000000000..a43bd9d8e1 Binary files /dev/null and b/en/3.4/_images/aws-netris-ipam-mgmt.png differ diff --git a/en/3.4/_images/aws-netris-provision-sg.png b/en/3.4/_images/aws-netris-provision-sg.png new file mode 100644 index 0000000000..71d018f61d Binary files /dev/null and b/en/3.4/_images/aws-netris-provision-sg.png differ diff --git a/en/3.4/_images/aws-netris-site-create.png b/en/3.4/_images/aws-netris-site-create.png new file mode 100644 index 0000000000..641f1ef4e2 Binary files /dev/null and b/en/3.4/_images/aws-netris-site-create.png differ diff --git a/en/3.4/_images/aws-netris-site-mesh-status.png b/en/3.4/_images/aws-netris-site-mesh-status.png new file mode 100644 index 0000000000..03d25a1383 Binary files /dev/null and b/en/3.4/_images/aws-netris-site-mesh-status.png differ diff --git a/en/3.4/_images/aws-netris-static-route.png b/en/3.4/_images/aws-netris-static-route.png new file mode 100644 index 0000000000..97bee607d2 Binary files /dev/null and b/en/3.4/_images/aws-netris-static-route.png differ diff --git a/en/3.4/_images/aws-security-group.png b/en/3.4/_images/aws-security-group.png new file mode 100644 index 0000000000..ea71ffdef6 Binary files /dev/null and b/en/3.4/_images/aws-security-group.png differ diff --git a/en/3.4/_images/aws-softgate-deployed.png b/en/3.4/_images/aws-softgate-deployed.png new file mode 100644 index 0000000000..0ae61a94f7 Binary files /dev/null and b/en/3.4/_images/aws-softgate-deployed.png differ diff --git a/en/3.4/_images/aws-vpc-cidr-to-netris.png b/en/3.4/_images/aws-vpc-cidr-to-netris.png new file mode 100644 index 0000000000..31849fd37b Binary files /dev/null and b/en/3.4/_images/aws-vpc-cidr-to-netris.png differ diff --git a/en/3.4/_images/aws-vpc-routes-created.png b/en/3.4/_images/aws-vpc-routes-created.png new file mode 100644 index 0000000000..b355e602ab Binary files /dev/null and b/en/3.4/_images/aws-vpc-routes-created.png differ diff --git a/en/3.4/_images/bgp-listing.png b/en/3.4/_images/bgp-listing.png new file mode 100644 index 0000000000..2e06af3d0d Binary files /dev/null and b/en/3.4/_images/bgp-listing.png differ diff --git a/en/3.4/_images/bgp-looking-glass.png b/en/3.4/_images/bgp-looking-glass.png new file mode 100644 index 0000000000..b2cef4aaca Binary files /dev/null and b/en/3.4/_images/bgp-looking-glass.png differ diff --git a/en/3.4/_images/change-password.png b/en/3.4/_images/change-password.png new file mode 100644 index 0000000000..ad093aa331 Binary files /dev/null and b/en/3.4/_images/change-password.png differ diff --git a/en/3.4/_images/cloudflare-dns-record.png b/en/3.4/_images/cloudflare-dns-record.png new file mode 100644 index 0000000000..a0d7586c2b Binary files /dev/null and b/en/3.4/_images/cloudflare-dns-record.png differ diff --git a/en/3.4/_images/community.png b/en/3.4/_images/community.png new file mode 100644 index 0000000000..53fd553d3f Binary files /dev/null and b/en/3.4/_images/community.png differ diff --git a/en/3.4/_images/create-new-admin-user.png b/en/3.4/_images/create-new-admin-user.png new file mode 100644 index 0000000000..4dcde0ba0b Binary files /dev/null and b/en/3.4/_images/create-new-admin-user.png differ diff --git a/en/3.4/_images/create_link.png b/en/3.4/_images/create_link.png new file mode 100644 index 0000000000..572869cb6f Binary files /dev/null and b/en/3.4/_images/create_link.png differ diff --git a/en/3.4/_images/createboard.png b/en/3.4/_images/createboard.png new file mode 100644 index 0000000000..8040723586 Binary files /dev/null and b/en/3.4/_images/createboard.png differ diff --git a/en/3.4/_images/credentials.png b/en/3.4/_images/credentials.png new file mode 100644 index 0000000000..edbc78148f Binary files /dev/null and b/en/3.4/_images/credentials.png differ diff --git a/en/3.4/_images/defaultroute.png b/en/3.4/_images/defaultroute.png new file mode 100644 index 0000000000..1a4d407173 Binary files /dev/null and b/en/3.4/_images/defaultroute.png differ diff --git a/en/3.4/_images/diagrams_terraform.png b/en/3.4/_images/diagrams_terraform.png new file mode 100644 index 0000000000..33f33920f0 Binary files /dev/null and b/en/3.4/_images/diagrams_terraform.png differ diff --git a/en/3.4/_images/dns-cloudflare-equinix-ip.png b/en/3.4/_images/dns-cloudflare-equinix-ip.png new file mode 100644 index 0000000000..212d1af7d0 Binary files /dev/null and b/en/3.4/_images/dns-cloudflare-equinix-ip.png differ diff --git a/en/3.4/_images/dns-record-netrisctl.png b/en/3.4/_images/dns-record-netrisctl.png new file mode 100644 index 0000000000..1d04c129b5 Binary files /dev/null and b/en/3.4/_images/dns-record-netrisctl.png differ diff --git a/en/3.4/_images/edit-port.png b/en/3.4/_images/edit-port.png new file mode 100644 index 0000000000..eed82b6388 Binary files /dev/null and b/en/3.4/_images/edit-port.png differ diff --git a/en/3.4/_images/edit_switch_port.png b/en/3.4/_images/edit_switch_port.png new file mode 100644 index 0000000000..711980ab9e Binary files /dev/null and b/en/3.4/_images/edit_switch_port.png differ diff --git a/en/3.4/_images/equinix-ebgp-links-up.png b/en/3.4/_images/equinix-ebgp-links-up.png new file mode 100644 index 0000000000..decce718f9 Binary files /dev/null and b/en/3.4/_images/equinix-ebgp-links-up.png differ diff --git a/en/3.4/_images/equinix-metal-activate-bgp.png b/en/3.4/_images/equinix-metal-activate-bgp.png new file mode 100644 index 0000000000..b03c10dd20 Binary files /dev/null and b/en/3.4/_images/equinix-metal-activate-bgp.png differ diff --git a/en/3.4/_images/equinix-metal-bgp-diagram.png b/en/3.4/_images/equinix-metal-bgp-diagram.png new file mode 100644 index 0000000000..50080c16ad Binary files /dev/null and b/en/3.4/_images/equinix-metal-bgp-diagram.png differ diff --git a/en/3.4/_images/equinix-metal-netris-bgp-up.png b/en/3.4/_images/equinix-metal-netris-bgp-up.png new file mode 100644 index 0000000000..8c993d3a97 Binary files /dev/null and b/en/3.4/_images/equinix-metal-netris-bgp-up.png differ diff --git a/en/3.4/_images/equinix-metal-netris-ipam-synced.png b/en/3.4/_images/equinix-metal-netris-ipam-synced.png new file mode 100644 index 0000000000..7095db4a4c Binary files /dev/null and b/en/3.4/_images/equinix-metal-netris-ipam-synced.png differ diff --git a/en/3.4/_images/equinix-metal-project-api-keys.png b/en/3.4/_images/equinix-metal-project-api-keys.png new file mode 100644 index 0000000000..1988e1e503 Binary files /dev/null and b/en/3.4/_images/equinix-metal-project-api-keys.png differ diff --git a/en/3.4/_images/equinix-metal-project-id.png b/en/3.4/_images/equinix-metal-project-id.png new file mode 100644 index 0000000000..de83c5d6c0 Binary files /dev/null and b/en/3.4/_images/equinix-metal-project-id.png differ diff --git a/en/3.4/_images/equinix-metal-request-ip-block.png b/en/3.4/_images/equinix-metal-request-ip-block.png new file mode 100644 index 0000000000..9a39600bac Binary files /dev/null and b/en/3.4/_images/equinix-metal-request-ip-block.png differ diff --git a/en/3.4/_images/equinix-metal-vnet-with-tag-terraform.png b/en/3.4/_images/equinix-metal-vnet-with-tag-terraform.png new file mode 100644 index 0000000000..e40e8c93ce Binary files /dev/null and b/en/3.4/_images/equinix-metal-vnet-with-tag-terraform.png differ diff --git a/en/3.4/_images/equinix-metal-vnet-with-tag.png b/en/3.4/_images/equinix-metal-vnet-with-tag.png new file mode 100644 index 0000000000..714e80e4f5 Binary files /dev/null and b/en/3.4/_images/equinix-metal-vnet-with-tag.png differ diff --git a/en/3.4/_images/equinix-request-c3-small-server.png b/en/3.4/_images/equinix-request-c3-small-server.png new file mode 100644 index 0000000000..18c5129342 Binary files /dev/null and b/en/3.4/_images/equinix-request-c3-small-server.png differ diff --git a/en/3.4/_images/gcp-concept-traffic-flows.png b/en/3.4/_images/gcp-concept-traffic-flows.png new file mode 100644 index 0000000000..cc8e5a03a0 Binary files /dev/null and b/en/3.4/_images/gcp-concept-traffic-flows.png differ diff --git a/en/3.4/_images/gcp-firewall-rule.png b/en/3.4/_images/gcp-firewall-rule.png new file mode 100644 index 0000000000..8b7c9585a0 Binary files /dev/null and b/en/3.4/_images/gcp-firewall-rule.png differ diff --git a/en/3.4/_images/gcp-netris-create-sg.png b/en/3.4/_images/gcp-netris-create-sg.png new file mode 100644 index 0000000000..8203548be0 Binary files /dev/null and b/en/3.4/_images/gcp-netris-create-sg.png differ diff --git a/en/3.4/_images/gcp-netris-enable-site-mesh.png b/en/3.4/_images/gcp-netris-enable-site-mesh.png new file mode 100644 index 0000000000..9b7c2b6086 Binary files /dev/null and b/en/3.4/_images/gcp-netris-enable-site-mesh.png differ diff --git a/en/3.4/_images/gcp-netris-ipam-lo.png b/en/3.4/_images/gcp-netris-ipam-lo.png new file mode 100644 index 0000000000..06bfeddbb2 Binary files /dev/null and b/en/3.4/_images/gcp-netris-ipam-lo.png differ diff --git a/en/3.4/_images/gcp-netris-ipam-mgmt.png b/en/3.4/_images/gcp-netris-ipam-mgmt.png new file mode 100644 index 0000000000..be87b2782a Binary files /dev/null and b/en/3.4/_images/gcp-netris-ipam-mgmt.png differ diff --git a/en/3.4/_images/gcp-netris-provision-sg.png b/en/3.4/_images/gcp-netris-provision-sg.png new file mode 100644 index 0000000000..8c73863859 Binary files /dev/null and b/en/3.4/_images/gcp-netris-provision-sg.png differ diff --git a/en/3.4/_images/gcp-netris-site-create.png b/en/3.4/_images/gcp-netris-site-create.png new file mode 100644 index 0000000000..19b2f73242 Binary files /dev/null and b/en/3.4/_images/gcp-netris-site-create.png differ diff --git a/en/3.4/_images/gcp-netris-site-mesh-status.png b/en/3.4/_images/gcp-netris-site-mesh-status.png new file mode 100644 index 0000000000..6f23791ec7 Binary files /dev/null and b/en/3.4/_images/gcp-netris-site-mesh-status.png differ diff --git a/en/3.4/_images/gcp-netris-static-route.png b/en/3.4/_images/gcp-netris-static-route.png new file mode 100644 index 0000000000..e2ec8aa8b4 Binary files /dev/null and b/en/3.4/_images/gcp-netris-static-route.png differ diff --git a/en/3.4/_images/gcp-softgate-deployed.png b/en/3.4/_images/gcp-softgate-deployed.png new file mode 100644 index 0000000000..1db913283e Binary files /dev/null and b/en/3.4/_images/gcp-softgate-deployed.png differ diff --git a/en/3.4/_images/gcp-vpc-routes-created.png b/en/3.4/_images/gcp-vpc-routes-created.png new file mode 100644 index 0000000000..f574abbf27 Binary files /dev/null and b/en/3.4/_images/gcp-vpc-routes-created.png differ diff --git a/en/3.4/_images/gcp-vpc-subnet-to-netris.png b/en/3.4/_images/gcp-vpc-subnet-to-netris.png new file mode 100644 index 0000000000..db551ea3f1 Binary files /dev/null and b/en/3.4/_images/gcp-vpc-subnet-to-netris.png differ diff --git a/en/3.4/_images/globalIP.png b/en/3.4/_images/globalIP.png new file mode 100644 index 0000000000..bae8a860ce Binary files /dev/null and b/en/3.4/_images/globalIP.png differ diff --git a/en/3.4/_images/graphboard.png b/en/3.4/_images/graphboard.png new file mode 100644 index 0000000000..b777f11b69 Binary files /dev/null and b/en/3.4/_images/graphboard.png differ diff --git a/en/3.4/_images/hairpin.png b/en/3.4/_images/hairpin.png new file mode 100644 index 0000000000..8bfa508087 Binary files /dev/null and b/en/3.4/_images/hairpin.png differ diff --git a/en/3.4/_images/hairpin_topology.png b/en/3.4/_images/hairpin_topology.png new file mode 100644 index 0000000000..95b3c4fe5d Binary files /dev/null and b/en/3.4/_images/hairpin_topology.png differ diff --git a/en/3.4/_images/hardware_health.png b/en/3.4/_images/hardware_health.png new file mode 100644 index 0000000000..df70fc2ecc Binary files /dev/null and b/en/3.4/_images/hardware_health.png differ diff --git a/en/3.4/_images/install_agent.gif b/en/3.4/_images/install_agent.gif new file mode 100644 index 0000000000..0928829c40 Binary files /dev/null and b/en/3.4/_images/install_agent.gif differ diff --git a/en/3.4/_images/inventory-listing.png b/en/3.4/_images/inventory-listing.png new file mode 100644 index 0000000000..caa35c8f66 Binary files /dev/null and b/en/3.4/_images/inventory-listing.png differ diff --git a/en/3.4/_images/inventory-profile.png b/en/3.4/_images/inventory-profile.png new file mode 100644 index 0000000000..e58ef725b7 Binary files /dev/null and b/en/3.4/_images/inventory-profile.png differ diff --git a/en/3.4/_images/l3lb_srv01.png b/en/3.4/_images/l3lb_srv01.png new file mode 100644 index 0000000000..f2fe0bf3bb Binary files /dev/null and b/en/3.4/_images/l3lb_srv01.png differ diff --git a/en/3.4/_images/l3lb_srv02.png b/en/3.4/_images/l3lb_srv02.png new file mode 100644 index 0000000000..6d3a5bd423 Binary files /dev/null and b/en/3.4/_images/l3lb_srv02.png differ diff --git a/en/3.4/_images/leaf1_spine1.png b/en/3.4/_images/leaf1_spine1.png new file mode 100644 index 0000000000..8b9b4a3ff0 Binary files /dev/null and b/en/3.4/_images/leaf1_spine1.png differ diff --git a/en/3.4/_images/list-l3-lb-detail.png b/en/3.4/_images/list-l3-lb-detail.png new file mode 100644 index 0000000000..d9a57ae5f1 Binary files /dev/null and b/en/3.4/_images/list-l3-lb-detail.png differ diff --git a/en/3.4/_images/list-l3-lb.png b/en/3.4/_images/list-l3-lb.png new file mode 100644 index 0000000000..5f33c2ad02 Binary files /dev/null and b/en/3.4/_images/list-l3-lb.png differ diff --git a/en/3.4/_images/list-l4-load-balancers.png b/en/3.4/_images/list-l4-load-balancers.png new file mode 100644 index 0000000000..b100218c79 Binary files /dev/null and b/en/3.4/_images/list-l4-load-balancers.png differ diff --git a/en/3.4/_images/list-subnets.png b/en/3.4/_images/list-subnets.png new file mode 100644 index 0000000000..a3e9d60c73 Binary files /dev/null and b/en/3.4/_images/list-subnets.png differ diff --git a/en/3.4/_images/list-vnet-expanded.png b/en/3.4/_images/list-vnet-expanded.png new file mode 100644 index 0000000000..fdc5f16998 Binary files /dev/null and b/en/3.4/_images/list-vnet-expanded.png differ diff --git a/en/3.4/_images/list-vnet.png b/en/3.4/_images/list-vnet.png new file mode 100644 index 0000000000..2ad4b6fa49 Binary files /dev/null and b/en/3.4/_images/list-vnet.png differ diff --git a/en/3.4/_images/local-public-asn.png b/en/3.4/_images/local-public-asn.png new file mode 100644 index 0000000000..22ffb71e17 Binary files /dev/null and b/en/3.4/_images/local-public-asn.png differ diff --git a/en/3.4/_images/manage-vnet.gif b/en/3.4/_images/manage-vnet.gif new file mode 100644 index 0000000000..f8e7e8620c Binary files /dev/null and b/en/3.4/_images/manage-vnet.gif differ diff --git a/en/3.4/_images/netris-architecture.png b/en/3.4/_images/netris-architecture.png new file mode 100644 index 0000000000..8c36dc9ac8 Binary files /dev/null and b/en/3.4/_images/netris-architecture.png differ diff --git a/en/3.4/_images/netris-controller-installed.png b/en/3.4/_images/netris-controller-installed.png new file mode 100644 index 0000000000..a6851e2b71 Binary files /dev/null and b/en/3.4/_images/netris-controller-installed.png differ diff --git a/en/3.4/_images/netris-create-common-subnets.png b/en/3.4/_images/netris-create-common-subnets.png new file mode 100644 index 0000000000..b47c8e952c Binary files /dev/null and b/en/3.4/_images/netris-create-common-subnets.png differ diff --git a/en/3.4/_images/netris-create-equinix-metal-site.png b/en/3.4/_images/netris-create-equinix-metal-site.png new file mode 100644 index 0000000000..39734075f1 Binary files /dev/null and b/en/3.4/_images/netris-create-equinix-metal-site.png differ diff --git a/en/3.4/_images/netris-create-nat-rule.png b/en/3.4/_images/netris-create-nat-rule.png new file mode 100644 index 0000000000..28c5913965 Binary files /dev/null and b/en/3.4/_images/netris-create-nat-rule.png differ diff --git a/en/3.4/_images/netris-creating-vnet-for-equinix-metal.png b/en/3.4/_images/netris-creating-vnet-for-equinix-metal.png new file mode 100644 index 0000000000..98fdbd910d Binary files /dev/null and b/en/3.4/_images/netris-creating-vnet-for-equinix-metal.png differ diff --git a/en/3.4/_images/netris-enable-elb.png b/en/3.4/_images/netris-enable-elb.png new file mode 100644 index 0000000000..36e7566443 Binary files /dev/null and b/en/3.4/_images/netris-enable-elb.png differ diff --git a/en/3.4/_images/netris-ipam-nat.png b/en/3.4/_images/netris-ipam-nat.png new file mode 100644 index 0000000000..d7ff2449c7 Binary files /dev/null and b/en/3.4/_images/netris-ipam-nat.png differ diff --git a/en/3.4/_images/netris-l4-load-balancer.png b/en/3.4/_images/netris-l4-load-balancer.png new file mode 100644 index 0000000000..e6e466f622 Binary files /dev/null and b/en/3.4/_images/netris-l4-load-balancer.png differ diff --git a/en/3.4/_images/netris-vnet-ready-in-equinix-metal.png b/en/3.4/_images/netris-vnet-ready-in-equinix-metal.png new file mode 100644 index 0000000000..cf7274bb86 Binary files /dev/null and b/en/3.4/_images/netris-vnet-ready-in-equinix-metal.png differ diff --git a/en/3.4/_images/netris_controller_diagram.png b/en/3.4/_images/netris_controller_diagram.png new file mode 100644 index 0000000000..610a8ae396 Binary files /dev/null and b/en/3.4/_images/netris_controller_diagram.png differ diff --git a/en/3.4/_images/netris_version_example.png b/en/3.4/_images/netris_version_example.png new file mode 100644 index 0000000000..55e23d8df9 Binary files /dev/null and b/en/3.4/_images/netris_version_example.png differ diff --git a/en/3.4/_images/password.png b/en/3.4/_images/password.png new file mode 100644 index 0000000000..a27b7725a9 Binary files /dev/null and b/en/3.4/_images/password.png differ diff --git a/en/3.4/_images/permission_group.png b/en/3.4/_images/permission_group.png new file mode 100644 index 0000000000..e4d1ad6ab0 Binary files /dev/null and b/en/3.4/_images/permission_group.png differ diff --git a/en/3.4/_images/phoenixnap-api-credential.png b/en/3.4/_images/phoenixnap-api-credential.png new file mode 100644 index 0000000000..23cdf89bfa Binary files /dev/null and b/en/3.4/_images/phoenixnap-api-credential.png differ diff --git a/en/3.4/_images/phoenixnap-concept-public-network.png b/en/3.4/_images/phoenixnap-concept-public-network.png new file mode 100644 index 0000000000..eb2ee8c35c Binary files /dev/null and b/en/3.4/_images/phoenixnap-concept-public-network.png differ diff --git a/en/3.4/_images/phoenixnap-concept-solution-traffic-flows.png b/en/3.4/_images/phoenixnap-concept-solution-traffic-flows.png new file mode 100644 index 0000000000..0e58af3d47 Binary files /dev/null and b/en/3.4/_images/phoenixnap-concept-solution-traffic-flows.png differ diff --git a/en/3.4/_images/phoenixnap-dns-cloudflare.png b/en/3.4/_images/phoenixnap-dns-cloudflare.png new file mode 100644 index 0000000000..2d45337dad Binary files /dev/null and b/en/3.4/_images/phoenixnap-dns-cloudflare.png differ diff --git a/en/3.4/_images/phoenixnap-l4lb.png b/en/3.4/_images/phoenixnap-l4lb.png new file mode 100644 index 0000000000..e445fd0e0b Binary files /dev/null and b/en/3.4/_images/phoenixnap-l4lb.png differ diff --git a/en/3.4/_images/phoenixnap-nat-dnat.png b/en/3.4/_images/phoenixnap-nat-dnat.png new file mode 100644 index 0000000000..5c2c8678db Binary files /dev/null and b/en/3.4/_images/phoenixnap-nat-dnat.png differ diff --git a/en/3.4/_images/phoenixnap-nat-masquerade.png b/en/3.4/_images/phoenixnap-nat-masquerade.png new file mode 100644 index 0000000000..8a882dcc30 Binary files /dev/null and b/en/3.4/_images/phoenixnap-nat-masquerade.png differ diff --git a/en/3.4/_images/phoenixnap-nat-snat.png b/en/3.4/_images/phoenixnap-nat-snat.png new file mode 100644 index 0000000000..f83b410439 Binary files /dev/null and b/en/3.4/_images/phoenixnap-nat-snat.png differ diff --git a/en/3.4/_images/phoenixnap-netris-create-common-subnets.png b/en/3.4/_images/phoenixnap-netris-create-common-subnets.png new file mode 100644 index 0000000000..0b25e81e28 Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-create-common-subnets.png differ diff --git a/en/3.4/_images/phoenixnap-netris-creating-vnet.png b/en/3.4/_images/phoenixnap-netris-creating-vnet.png new file mode 100644 index 0000000000..9554bbfc75 Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-creating-vnet.png differ diff --git a/en/3.4/_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png b/en/3.4/_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png new file mode 100644 index 0000000000..6eaad541b1 Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png differ diff --git a/en/3.4/_images/phoenixnap-netris-ipam-lb-purpose.png b/en/3.4/_images/phoenixnap-netris-ipam-lb-purpose.png new file mode 100644 index 0000000000..f024de0677 Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-ipam-lb-purpose.png differ diff --git a/en/3.4/_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png b/en/3.4/_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png new file mode 100644 index 0000000000..46ec2cacff Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png differ diff --git a/en/3.4/_images/phoenixnap-netris-ipam-nat-purpose.png b/en/3.4/_images/phoenixnap-netris-ipam-nat-purpose.png new file mode 100644 index 0000000000..e0206cf652 Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-ipam-nat-purpose.png differ diff --git a/en/3.4/_images/phoenixnap-netris-ipam-synced-slash-28.png b/en/3.4/_images/phoenixnap-netris-ipam-synced-slash-28.png new file mode 100644 index 0000000000..8c6fcbfbbf Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-ipam-synced-slash-28.png differ diff --git a/en/3.4/_images/phoenixnap-netris-ipam-synced.png b/en/3.4/_images/phoenixnap-netris-ipam-synced.png new file mode 100644 index 0000000000..a102be885d Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-ipam-synced.png differ diff --git a/en/3.4/_images/phoenixnap-netris-vnet-ready.png b/en/3.4/_images/phoenixnap-netris-vnet-ready.png new file mode 100644 index 0000000000..794a4a1834 Binary files /dev/null and b/en/3.4/_images/phoenixnap-netris-vnet-ready.png differ diff --git a/en/3.4/_images/phoenixnap-request-ctl-server.png b/en/3.4/_images/phoenixnap-request-ctl-server.png new file mode 100644 index 0000000000..9d035e7784 Binary files /dev/null and b/en/3.4/_images/phoenixnap-request-ctl-server.png differ diff --git a/en/3.4/_images/phoenixnap-request-ip-allocation-slash-28.png b/en/3.4/_images/phoenixnap-request-ip-allocation-slash-28.png new file mode 100644 index 0000000000..145e7db0d4 Binary files /dev/null and b/en/3.4/_images/phoenixnap-request-ip-allocation-slash-28.png differ diff --git a/en/3.4/_images/phoenixnap-request-ip-allocation.png b/en/3.4/_images/phoenixnap-request-ip-allocation.png new file mode 100644 index 0000000000..1c4beebb8f Binary files /dev/null and b/en/3.4/_images/phoenixnap-request-ip-allocation.png differ diff --git a/en/3.4/_images/phoenixnap-reserved-ips.png b/en/3.4/_images/phoenixnap-reserved-ips.png new file mode 100644 index 0000000000..815a10d859 Binary files /dev/null and b/en/3.4/_images/phoenixnap-reserved-ips.png differ diff --git a/en/3.4/_images/phoenixnap-site-create.png b/en/3.4/_images/phoenixnap-site-create.png new file mode 100644 index 0000000000..6ce0021bbd Binary files /dev/null and b/en/3.4/_images/phoenixnap-site-create.png differ diff --git a/en/3.4/_images/phoenixnap-softgate-nodes-created.png b/en/3.4/_images/phoenixnap-softgate-nodes-created.png new file mode 100644 index 0000000000..b3f65b8b08 Binary files /dev/null and b/en/3.4/_images/phoenixnap-softgate-nodes-created.png differ diff --git a/en/3.4/_images/phoenixnap-softgate-nodes-creation.png b/en/3.4/_images/phoenixnap-softgate-nodes-creation.png new file mode 100644 index 0000000000..388f903bf5 Binary files /dev/null and b/en/3.4/_images/phoenixnap-softgate-nodes-creation.png differ diff --git a/en/3.4/_images/phoenixnap-softgate-nodes-green.png b/en/3.4/_images/phoenixnap-softgate-nodes-green.png new file mode 100644 index 0000000000..570870bc91 Binary files /dev/null and b/en/3.4/_images/phoenixnap-softgate-nodes-green.png differ diff --git a/en/3.4/_images/phoenixnap-vnet-import-a-new-server.png b/en/3.4/_images/phoenixnap-vnet-import-a-new-server.png new file mode 100644 index 0000000000..4d091b6248 Binary files /dev/null and b/en/3.4/_images/phoenixnap-vnet-import-a-new-server.png differ diff --git a/en/3.4/_images/phoenixnap-vnet-imported-new-server.png b/en/3.4/_images/phoenixnap-vnet-imported-new-server.png new file mode 100644 index 0000000000..2d706c5ecf Binary files /dev/null and b/en/3.4/_images/phoenixnap-vnet-imported-new-server.png differ diff --git a/en/3.4/_images/phoenixnap-vnet-managed-vnet.png b/en/3.4/_images/phoenixnap-vnet-managed-vnet.png new file mode 100644 index 0000000000..d4075cf571 Binary files /dev/null and b/en/3.4/_images/phoenixnap-vnet-managed-vnet.png differ diff --git a/en/3.4/_images/phoenixnap-vnet-ssh-to-server.png b/en/3.4/_images/phoenixnap-vnet-ssh-to-server.png new file mode 100644 index 0000000000..61dce121ac Binary files /dev/null and b/en/3.4/_images/phoenixnap-vnet-ssh-to-server.png differ diff --git a/en/3.4/_images/phoenixnap-vnet-unmanaged-vnet.png b/en/3.4/_images/phoenixnap-vnet-unmanaged-vnet.png new file mode 100644 index 0000000000..3e4b0105d7 Binary files /dev/null and b/en/3.4/_images/phoenixnap-vnet-unmanaged-vnet.png differ diff --git a/en/3.4/_images/phoenixnap-vnet-with-tag-terraform.png b/en/3.4/_images/phoenixnap-vnet-with-tag-terraform.png new file mode 100644 index 0000000000..d0adc87318 Binary files /dev/null and b/en/3.4/_images/phoenixnap-vnet-with-tag-terraform.png differ diff --git a/en/3.4/_images/phoenixnap-vnet-with-tag.png b/en/3.4/_images/phoenixnap-vnet-with-tag.png new file mode 100644 index 0000000000..f8ef08dc88 Binary files /dev/null and b/en/3.4/_images/phoenixnap-vnet-with-tag.png differ diff --git a/en/3.4/_images/ping.png b/en/3.4/_images/ping.png new file mode 100644 index 0000000000..c8e9e64e47 Binary files /dev/null and b/en/3.4/_images/ping.png differ diff --git a/en/3.4/_images/quick-action-ports.png b/en/3.4/_images/quick-action-ports.png new file mode 100644 index 0000000000..347b7bcb66 Binary files /dev/null and b/en/3.4/_images/quick-action-ports.png differ diff --git a/en/3.4/_images/request-L4.png b/en/3.4/_images/request-L4.png new file mode 100644 index 0000000000..8e73cf8be8 Binary files /dev/null and b/en/3.4/_images/request-L4.png differ diff --git a/en/3.4/_images/route-map.png b/en/3.4/_images/route-map.png new file mode 100644 index 0000000000..692cfd63f3 Binary files /dev/null and b/en/3.4/_images/route-map.png differ diff --git a/en/3.4/_images/sandbox-l4lb-kubeapi.png b/en/3.4/_images/sandbox-l4lb-kubeapi.png new file mode 100644 index 0000000000..9faa757631 Binary files /dev/null and b/en/3.4/_images/sandbox-l4lb-kubeapi.png differ diff --git a/en/3.4/_images/sandbox-l4lbs.png b/en/3.4/_images/sandbox-l4lbs.png new file mode 100644 index 0000000000..629b642e88 Binary files /dev/null and b/en/3.4/_images/sandbox-l4lbs.png differ diff --git a/en/3.4/_images/sandbox-podinfo-prov.png b/en/3.4/_images/sandbox-podinfo-prov.png new file mode 100644 index 0000000000..5dec03019c Binary files /dev/null and b/en/3.4/_images/sandbox-podinfo-prov.png differ diff --git a/en/3.4/_images/sandbox-podinfo-ready.png b/en/3.4/_images/sandbox-podinfo-ready.png new file mode 100644 index 0000000000..2bfb5864bb Binary files /dev/null and b/en/3.4/_images/sandbox-podinfo-ready.png differ diff --git a/en/3.4/_images/sandbox_topology.png b/en/3.4/_images/sandbox_topology.png new file mode 100644 index 0000000000..ccb044f313 Binary files /dev/null and b/en/3.4/_images/sandbox_topology.png differ diff --git a/en/3.4/_images/saveasnormal.png b/en/3.4/_images/saveasnormal.png new file mode 100644 index 0000000000..6f5cd5bda3 Binary files /dev/null and b/en/3.4/_images/saveasnormal.png differ diff --git a/en/3.4/_images/siteDefault.png b/en/3.4/_images/siteDefault.png new file mode 100644 index 0000000000..f8b1cf2688 Binary files /dev/null and b/en/3.4/_images/siteDefault.png differ diff --git a/en/3.4/_images/slide-1.png b/en/3.4/_images/slide-1.png new file mode 100644 index 0000000000..b9c94ee198 Binary files /dev/null and b/en/3.4/_images/slide-1.png differ diff --git a/en/3.4/_images/slide-2.png b/en/3.4/_images/slide-2.png new file mode 100644 index 0000000000..99b0ae67e9 Binary files /dev/null and b/en/3.4/_images/slide-2.png differ diff --git a/en/3.4/_images/slide-3.png b/en/3.4/_images/slide-3.png new file mode 100644 index 0000000000..4f17eafc0e Binary files /dev/null and b/en/3.4/_images/slide-3.png differ diff --git a/en/3.4/_images/slide-4.png b/en/3.4/_images/slide-4.png new file mode 100644 index 0000000000..3324e389da Binary files /dev/null and b/en/3.4/_images/slide-4.png differ diff --git a/en/3.4/_images/softgate-green.png b/en/3.4/_images/softgate-green.png new file mode 100644 index 0000000000..5043c928bd Binary files /dev/null and b/en/3.4/_images/softgate-green.png differ diff --git a/en/3.4/_images/softgate-install-agent.png b/en/3.4/_images/softgate-install-agent.png new file mode 100644 index 0000000000..b07f8b551b Binary files /dev/null and b/en/3.4/_images/softgate-install-agent.png differ diff --git a/en/3.4/_images/softgate-nodes-created-in-equinix.png b/en/3.4/_images/softgate-nodes-created-in-equinix.png new file mode 100644 index 0000000000..6d0971205d Binary files /dev/null and b/en/3.4/_images/softgate-nodes-created-in-equinix.png differ diff --git a/en/3.4/_images/softgate-nodes-recognized-in-netris.png b/en/3.4/_images/softgate-nodes-recognized-in-netris.png new file mode 100644 index 0000000000..ddf15d2121 Binary files /dev/null and b/en/3.4/_images/softgate-nodes-recognized-in-netris.png differ diff --git a/en/3.4/_images/softgate-one-liner-provisioning.png b/en/3.4/_images/softgate-one-liner-provisioning.png new file mode 100644 index 0000000000..f1c72da232 Binary files /dev/null and b/en/3.4/_images/softgate-one-liner-provisioning.png differ diff --git a/en/3.4/_images/softgate-provisioning-cli-output.png b/en/3.4/_images/softgate-provisioning-cli-output.png new file mode 100644 index 0000000000..a2f2bdd9bd Binary files /dev/null and b/en/3.4/_images/softgate-provisioning-cli-output.png differ diff --git a/en/3.4/_images/softgate_diagram.png b/en/3.4/_images/softgate_diagram.png new file mode 100644 index 0000000000..a6ade1fd5f Binary files /dev/null and b/en/3.4/_images/softgate_diagram.png differ diff --git a/en/3.4/_images/static_route.png b/en/3.4/_images/static_route.png new file mode 100644 index 0000000000..a022e230d4 Binary files /dev/null and b/en/3.4/_images/static_route.png differ diff --git a/en/3.4/_images/subnet-tree.png b/en/3.4/_images/subnet-tree.png new file mode 100644 index 0000000000..22710f17e9 Binary files /dev/null and b/en/3.4/_images/subnet-tree.png differ diff --git a/en/3.4/_images/switch_port.png b/en/3.4/_images/switch_port.png new file mode 100644 index 0000000000..8f0e537af5 Binary files /dev/null and b/en/3.4/_images/switch_port.png differ diff --git a/en/3.4/_images/telescope.png b/en/3.4/_images/telescope.png new file mode 100644 index 0000000000..7cfc53fcc7 Binary files /dev/null and b/en/3.4/_images/telescope.png differ diff --git a/en/3.4/_images/tenants.png b/en/3.4/_images/tenants.png new file mode 100644 index 0000000000..eb4172733d Binary files /dev/null and b/en/3.4/_images/tenants.png differ diff --git a/en/3.4/_images/topology_manager.png b/en/3.4/_images/topology_manager.png new file mode 100644 index 0000000000..ce7bb1effe Binary files /dev/null and b/en/3.4/_images/topology_manager.png differ diff --git a/en/3.4/_images/uninstallOS.png b/en/3.4/_images/uninstallOS.png new file mode 100644 index 0000000000..eb2e1965c0 Binary files /dev/null and b/en/3.4/_images/uninstallOS.png differ diff --git a/en/3.4/_images/unmanaged-vlan-equinix.png b/en/3.4/_images/unmanaged-vlan-equinix.png new file mode 100644 index 0000000000..ab09bc2d11 Binary files /dev/null and b/en/3.4/_images/unmanaged-vlan-equinix.png differ diff --git a/en/3.4/_images/unmanaged-vnet.png b/en/3.4/_images/unmanaged-vnet.png new file mode 100644 index 0000000000..1d9afc47f2 Binary files /dev/null and b/en/3.4/_images/unmanaged-vnet.png differ diff --git a/en/3.4/_images/upgrading_sonic_folder_listing.png b/en/3.4/_images/upgrading_sonic_folder_listing.png new file mode 100644 index 0000000000..10825b7e34 Binary files /dev/null and b/en/3.4/_images/upgrading_sonic_folder_listing.png differ diff --git a/en/3.4/_images/upstream-dmz-logical.png b/en/3.4/_images/upstream-dmz-logical.png new file mode 100644 index 0000000000..6a0be1d504 Binary files /dev/null and b/en/3.4/_images/upstream-dmz-logical.png differ diff --git a/en/3.4/_images/upstream-dmz-physical.png b/en/3.4/_images/upstream-dmz-physical.png new file mode 100644 index 0000000000..a3d2637c5c Binary files /dev/null and b/en/3.4/_images/upstream-dmz-physical.png differ diff --git a/en/3.4/_images/user_role.png b/en/3.4/_images/user_role.png new file mode 100644 index 0000000000..d285d9b688 Binary files /dev/null and b/en/3.4/_images/user_role.png differ diff --git a/en/3.4/_images/users.png b/en/3.4/_images/users.png new file mode 100644 index 0000000000..2f90077728 Binary files /dev/null and b/en/3.4/_images/users.png differ diff --git a/en/3.4/_images/vpc-anywhere-check-site-default.png b/en/3.4/_images/vpc-anywhere-check-site-default.png new file mode 100644 index 0000000000..5f133ebd39 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-check-site-default.png differ diff --git a/en/3.4/_images/vpc-anywhere-edit-vlan-range-default-site.png b/en/3.4/_images/vpc-anywhere-edit-vlan-range-default-site.png new file mode 100644 index 0000000000..50531fb072 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-edit-vlan-range-default-site.png differ diff --git a/en/3.4/_images/vpc-anywhere-ipam-allocation.png b/en/3.4/_images/vpc-anywhere-ipam-allocation.png new file mode 100644 index 0000000000..649c00b33a Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-ipam-allocation.png differ diff --git a/en/3.4/_images/vpc-anywhere-ipam-l4lb-subnet.png b/en/3.4/_images/vpc-anywhere-ipam-l4lb-subnet.png new file mode 100644 index 0000000000..337e63732f Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-ipam-l4lb-subnet.png differ diff --git a/en/3.4/_images/vpc-anywhere-ipam-nat-subnet.png b/en/3.4/_images/vpc-anywhere-ipam-nat-subnet.png new file mode 100644 index 0000000000..0234367484 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-ipam-nat-subnet.png differ diff --git a/en/3.4/_images/vpc-anywhere-l4lb.png b/en/3.4/_images/vpc-anywhere-l4lb.png new file mode 100644 index 0000000000..f5d39737d2 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-l4lb.png differ diff --git a/en/3.4/_images/vpc-anywhere-nat-dnat.png b/en/3.4/_images/vpc-anywhere-nat-dnat.png new file mode 100644 index 0000000000..e2c1487a6b Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-nat-dnat.png differ diff --git a/en/3.4/_images/vpc-anywhere-nat-masquerade.png b/en/3.4/_images/vpc-anywhere-nat-masquerade.png new file mode 100644 index 0000000000..d5118ed47a Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-nat-masquerade.png differ diff --git a/en/3.4/_images/vpc-anywhere-nat-snat.png b/en/3.4/_images/vpc-anywhere-nat-snat.png new file mode 100644 index 0000000000..4f2658c7db Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-nat-snat.png differ diff --git a/en/3.4/_images/vpc-anywhere-sg-to-sg-experimental.png b/en/3.4/_images/vpc-anywhere-sg-to-sg-experimental.png new file mode 100644 index 0000000000..9e7275f132 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-sg-to-sg-experimental.png differ diff --git a/en/3.4/_images/vpc-anywhere-softgates-green.png b/en/3.4/_images/vpc-anywhere-softgates-green.png new file mode 100644 index 0000000000..bb919d3185 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-softgates-green.png differ diff --git a/en/3.4/_images/vpc-anywhere-solution-traffic-flows.png b/en/3.4/_images/vpc-anywhere-solution-traffic-flows.png new file mode 100644 index 0000000000..bbc1ad06ce Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-solution-traffic-flows.png differ diff --git a/en/3.4/_images/vpc-anywhere-upstream-bgp-router-logical.png b/en/3.4/_images/vpc-anywhere-upstream-bgp-router-logical.png new file mode 100644 index 0000000000..2096aeacfa Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-upstream-bgp-router-logical.png differ diff --git a/en/3.4/_images/vpc-anywhere-upstream-bgp-router-physical.png b/en/3.4/_images/vpc-anywhere-upstream-bgp-router-physical.png new file mode 100644 index 0000000000..423d159a9e Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-upstream-bgp-router-physical.png differ diff --git a/en/3.4/_images/vpc-anywhere-vnet-experimental.png b/en/3.4/_images/vpc-anywhere-vnet-experimental.png new file mode 100644 index 0000000000..50408639d5 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-vnet-experimental.png differ diff --git a/en/3.4/_images/vpc-anywhere-vnet.png b/en/3.4/_images/vpc-anywhere-vnet.png new file mode 100644 index 0000000000..3249bf18b4 Binary files /dev/null and b/en/3.4/_images/vpc-anywhere-vnet.png differ diff --git a/en/3.4/_images/waiting_approval.png b/en/3.4/_images/waiting_approval.png new file mode 100644 index 0000000000..ac769acb93 Binary files /dev/null and b/en/3.4/_images/waiting_approval.png differ diff --git a/en/3.4/_sources/EdgeCore-SONiC-Switch-initial-setup.rst.txt b/en/3.4/_sources/EdgeCore-SONiC-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..443378ffec --- /dev/null +++ b/en/3.4/_sources/EdgeCore-SONiC-Switch-initial-setup.rst.txt @@ -0,0 +1,92 @@ +============================ +EdgeCore SONiC Switch Initial Setup +============================ +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first. + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of EC SONiC. + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/Edgecore-SONiC_20211125_074752_ec202012_227.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password: + +``admin/YourPaSsWoRd`` + +3. Set up the Out-of-Band (OOB) Management. + +Disable ztp: + +.. code-block:: shell-session + + ztp disable -y + +Configure the IP address, default gateway, and DNS to establish Internet connectivity via the management port. + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +4. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner.png + :align: center + +.. image:: images/Switch-agent-installation-cli.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/3.4/_sources/Nvidia-Cumulus-v3.7-Switch-initial-setup.rst.txt b/en/3.4/_sources/Nvidia-Cumulus-v3.7-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..baaad99023 --- /dev/null +++ b/en/3.4/_sources/Nvidia-Cumulus-v3.7-Switch-initial-setup.rst.txt @@ -0,0 +1,108 @@ +============================ +Nvidia Cumulus v3.7 Switch Initial Setup +============================ +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first. + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux. + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/cumulus-linux-3.7.15-mlx-amd64.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password for Cumulus v3.7: + +``cumulus/CumulusLinux!`` + +3. Set up the Out-of-Band (OOB) Management. + +Open the network interfaces file and add the IP address and other required details. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + echo "nameserver " | sudo tee /etc/resolv.conf + +.. code-block:: shell-session + + sudo ifreload -a + +4. Cumulus Linux license installation. + +.. code-block:: shell-session + + sudo cl-license -i + +Copy/paste the Cumulus Linux license string, then press ctrl-d. + +5. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner.png + :align: center + +.. image:: images/Switch-agent-installation-cli.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/3.4/_sources/Nvidia-Cumulus-v5-Switch-initial-setup.rst.txt b/en/3.4/_sources/Nvidia-Cumulus-v5-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..ce0a885795 --- /dev/null +++ b/en/3.4/_sources/Nvidia-Cumulus-v5-Switch-initial-setup.rst.txt @@ -0,0 +1,130 @@ +.. _switch-agent-installation: +.. meta:: + :description: Network Switch Initial Setup + +============================ +Nvidia Cumulus v5 Switch Initial Setup +============================ + + +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first + + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux. + + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/cumulus-linux-5.4.0-mlx-amd64.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password for Cumulus v5: + +``cumulus/cumulus`` + + +3. Set up the Out-of-Band (OOB) Management. + +Upon installation of Cumulus Linux v5, the default Virtual Routing and Forwarding (VRF) is set to 'mgmt.' To switch to the default VRF, please refer to the following instructions: + +Disable ztp: + +.. code-block:: shell-session + + sudo ztp -d + +.. code-block:: shell-session + + sudo ip vrf exec default bash + +Open the network interfaces file, add the IP address and other required details, and ensure that you remove the 'mgmt' VRF configuration: + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + echo "nameserver " | sudo tee /etc/resolv.conf + +.. code-block:: shell-session + + sudo ifreload -a + +.. note:: + + You might see a one-time warning in the output of ifreload, which you can ignore: + +.. code-block:: shell-session + + warning: mgmt: cmd '/usr/lib/vrf/vrf-helper delete mgmt 1001' failed: returned 1 (Failed to delete cgroup for vrf mgmt) + + +4. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner.png + :align: center + +.. image:: images/Switch-agent-installation-cli.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/3.4/_sources/SoftGate-PRO-installation.rst.txt b/en/3.4/_sources/SoftGate-PRO-installation.rst.txt new file mode 100644 index 0000000000..26417b6c7a --- /dev/null +++ b/en/3.4/_sources/SoftGate-PRO-installation.rst.txt @@ -0,0 +1,72 @@ +.. meta:: + :description: Netris SoftGate PRO Installation + +*************************** +SoftGate PRO Installation +*************************** + +Minimum Hardware Requirements +============================= +* 2 x Intel® Xeon® Silver Processor with 10 physical cores per socket (20 cores total) +* 128 GB (64 GB RAM per socket) in multichannel configuration +* 300 GB HDD +* Nvidia Mellanox Connect-X 5/6 SmartNIC card + +BIOS Configuration +================== +The following are some recommendations for BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference: + +* Before starting consider resetting all BIOS settings to their defaults +* Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report +* Select Performance as the CPU Power and Performance policy +* Enable Turbo Boost +* Set memory frequency to the highest available number, NOT auto +* Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d +* Disable Hyper-Threading + +Install the Netris Agent +======================== +Requires freshly installed Ubuntu Linux 18.04 LTS and internet connectivity configured from netplan via management port. + +1. Add the SoftGate in the controller **Inventory** or **Topology** section. Detailed configuration documentation is available here: :ref:`"Adding SoftGates"`. +2. Once the SoftGate is created, navigate to the **Inventory** section, click the **three vertical dots (⋮)** on the right side of the newly created SoftGate and select the **Install Agent** option. +3. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user. +4. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration). + +.. note:: + + If the Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured. + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The management network interface + auto ensZ + iface ensZ inet static + address + # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node. + up ip route add via + gateway + + source /etc/network/interfaces.d/* + +5. If everything seems ok, please remove/comment the **Gateway** line and save the file. + +.. note:: + + Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent. + +6. Reboot the SoftGate + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. \ No newline at end of file diff --git a/en/3.4/_sources/SoftGate-installation.rst.txt b/en/3.4/_sources/SoftGate-installation.rst.txt new file mode 100644 index 0000000000..fc29f41685 --- /dev/null +++ b/en/3.4/_sources/SoftGate-installation.rst.txt @@ -0,0 +1,91 @@ +.. meta:: + :description: Netris SoftGate Installation + +*************************** +SoftGate Installation +*************************** + +Minimum Hardware Requirements +============================= +* 8 CPU cores +* 16 GB RAM +* 300 GB HDD + +Provision Netris SoftGate software +================================== +Requires freshly installed Ubuntu Linux 22.04 LTS and internet connectivity. + +1. Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: :ref:`"Adding SoftGates"`. + +2. Navigate to the **Net-->Inventory** section and click the **three vertical dots (⋮)** on the right side of the SoftGate node you are provisioning. Then click **Install Agent** and copy the one-line installer command to your clipboard. + +.. image:: /images/softgate-install-agent.png + :align: center + + +3. Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node) + +.. image:: /images/softgate-provisioning-cli-output.png + :align: center + +.. note:: + Please note that Netris replaces Netplan with regular ifupdown and attempts to migrate any prior configuration to /etc/network/interfaces. + +4. Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as you describe in /etc/network/interfaces. + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # Physical port on SoftGate node connected to a TRUNK port of your network + auto ens + iface ens inet static + address 0.0.0.0/0 + + # Optionally you can add more physical interfaces under your bond0 + auto ens + iface ens inet static + address 0.0.0.0/0 + + # Bond interface + auto bond0 + iface bond0 inet static + address 0.0.0.0/0 + # Please replace the ensX/Y with actual interface name(s) below to one(s) present in the OS. + bond-slaves ens ens + # Optional, please adjust the bonding mode below according to the desired functionality. + bond-mode active-backup + + source /etc/network/interfaces.d/* + + +5. Ensure that SoftGate node will maintain IP connectivity with Netris Controller after reboot. + + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The management network interface + auto ensZ + iface ensZ inet static + address + # Please delete or comment the line below if Netris Controller is located in the same network with the SoftGate node. + up ip route add via + + +6. Reboot the SoftGate + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up, you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. diff --git a/en/3.4/_sources/Ubuntu-SwitchDev-Switch-initial-setup.rst.txt b/en/3.4/_sources/Ubuntu-SwitchDev-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..76d909dbfa --- /dev/null +++ b/en/3.4/_sources/Ubuntu-SwitchDev-Switch-initial-setup.rst.txt @@ -0,0 +1,94 @@ +============================ +Ubuntu SwitchDev Switch Initial Setup +============================ +.. note:: + + Further installation requires a Console and Internet connectivity via management port! + + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first. + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of the Ubuntu SwitchDev. + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +Install Ubuntu SwitchDev using the Netris customized image: + +.. code-block:: shell-session + + onie-nos-install http://downloads.netris.ai/netris-ubuntu-18.04.1.bin + +Default username/password + +``netris/newNet0ps`` + +3. Set up the Out-of-Band (OOB) Management. + +Open the network interfaces file and add the IP address and other required details. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +5. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory-ubuntu.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner-ubuntu.png + :align: center + +.. image:: images/Switch-agent-installation-cli-ubuntu.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/3.4/_sources/accounts.rst.txt b/en/3.4/_sources/accounts.rst.txt new file mode 100644 index 0000000000..92f4ab3caa --- /dev/null +++ b/en/3.4/_sources/accounts.rst.txt @@ -0,0 +1,74 @@ +.. meta:: + :description: Netris Controller User Account Management + +######## +Accounts +######## + +The accounts section is for the management of user accounts, access permissions, and tenants. + +Users +===== +Description of User account fields: + +* **Username** - Unique username. +* **Full Name** - Full Name of the user. +* **E-mail** - The email address of the user. Also used for system notifications and for password retrieval. +* **E-mail CC** - Send copies of email notifications to this address. +* **Phone Number** - User’s phone number. +* **Company** - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers. +* **Position** - Position within the company. +* **User Role** - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate. +* **Permission Group** - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used) +* **+Tenant** - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used) + +Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin. + +.. image:: images/users.png + :align: center + :class: with-shadow + :alt: User Management + +**Password**: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side. + +Example: Listing of user accounts. + + +.. image:: images/password.png + :align: center + :class: with-shadow + :alt: List User Accounts + +Tenants +======= +IP addresses and Switch Ports are network resources that can be assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs) as part of DevOps/NetOps pipeline. + +A Tenant has just two fields, the unique name and custom description. + +Example: Adding a tenant. + +.. image:: images/tenants.png + :align: center + :class: with-shadow + :alt: Adding Tenants + +Permission Groups +================= +Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections. + +Example: Permission Group. + +.. image:: images/permission_group.png + :align: center + :class: with-shadow + :alt: Managing Permissions + +User Roles +========== +Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username. + +.. image:: images/user_role.png + :align: center + :class: with-shadow + :alt: User Roles + diff --git a/en/3.4/_sources/acls.rst.txt b/en/3.4/_sources/acls.rst.txt new file mode 100644 index 0000000000..4ff5a7ab12 --- /dev/null +++ b/en/3.4/_sources/acls.rst.txt @@ -0,0 +1,115 @@ +.. meta:: + :description: Access Control Lists (ACLs) + +.. _acl_def: + +########################## +Access Control Lists (ACL) +########################## +Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access. + +Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches. + +Screenshot: TCAM utilization can be seen under Net→Inventory + +.. image:: images/TCAM.png + :align: center + :class: with-shadow + +Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements. + +ACL Default Policy +------------------ +The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules. + +Example: Changing “ACL Default Policy” for the site “siteDefault”. + +.. image:: images/siteDefault.png + :align: center + :class: with-shadow + + +ACL Rules +--------- +ACL rules can be created, listed, edited, approved under Services→ACL. + +Description of ACL fields. +General + +* **Name** - Unique name for the ACL entry. +* **Protocol** - IP protocol to match. + + * All - Any IP protocols. + * IP - Specific IP protocol number. + * TCP - TCP. + * UDP - UDP. + * ICMP ALL - Any IPv4 ICMP protocol. + * ICMP Custom - Custom IPv4 ICMP code. + * ICMPv6 ALL - Any IPv6 ICMP protocol. + * ICMPv6 Custom - Custom IPv6 ICMP code. + +* **Active Until** - Disable this rule at the defined date/time. +* **Action** - Permit or Deny forwarding of matched packets. +* **Established/Reverse** - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination. + +Source/Destination - Source and destination addresses and ports to match. + +* **Source** IPv4/IPv6 - IPv4/IPv6 address. +* **Ports Type** + + * Port Range - Match on the port or a port range defined in this window. + * Port Group - Match on a group of ports defined under Services→ ACL Port Group. + +* **From Port** - Port range starting from. +* **To Port** - Port range ending with. + +* **Comment** - Descriptive comment, commonly used for approval workflows. + +* **Check button** - Check if Another ACL on the system already permits the described network access. + +Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established). + +.. image:: images/action_permit.png + :align: center + :class: with-shadow + +Example: “Check” shows that requested access is already provided by a broader ACL rule. + +.. image:: images/ACL_rule.png + :align: center + :class: with-shadow + +ACL Approval Workflow +--------------------- +When one tenant (one team) needs to get network access to resources under the responsibility of another tenant (another team), an ACL can be created but will activate only after approval of the tenant responsible for the destination address resources. See the below example. + +Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant. + +.. image:: images/ACL_approval.png + :align: center + :class: with-shadow + +Screenshot: ACL stays in “waiting for approval” state until approved. + +.. image:: images/waiting_approval.png + :align: center + :class: with-shadow + +Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it. + +.. image:: images/approve_reject.png + :align: center + :class: with-shadow + +Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric. + +.. image:: images/ACL_active.png + :align: center + :class: with-shadow + +ACL Processing Order +-------------------- +#. User-defined Deny Rules +#. User-defined Permit Rules +#. Deny the rest + diff --git a/en/3.4/_sources/controller-k3s-installation.rst.txt b/en/3.4/_sources/controller-k3s-installation.rst.txt new file mode 100644 index 0000000000..2aec4f8276 --- /dev/null +++ b/en/3.4/_sources/controller-k3s-installation.rst.txt @@ -0,0 +1,220 @@ +.. meta:: + :description: Controller Generic Linux Host + +.. _ctl-k3s-def: + +###################################################### +Netris Controller installation on a generic Linux host +###################################################### + +Linux Host requirements +======================= + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +.. note:: + + K3s is expected to work on most modern Linux systems. + + Some OSs have specific requirements: + + * If you are using Raspbian Buster, follow `these steps `__ to switch to legacy iptables. + * If you are using Alpine Linux, follow `these steps `__ for additional setup. + * If you are using (Red Hat/CentOS) Enterprise Linux, follow `these steps `__ for additional setup. + + +Installation +============ + +The following command will install the Netris Controller on your Linux server: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +Once installed, you will be able to log in to Netris Controller using your host's IP address. + + +.. note:: + The installation script does the following: + + * Installs `k3s `_ + * Installs the `Cert-Manager Helm chart `_ + * Installs the `Netris Controller Helm chart `_ + + + +Installation with the specific host name +---------------------------------------- + +In order to set the specific ingress host name to the Netris Controller, use the ``--ctl-hostname`` installation argument: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com + +A self-signed SSL certificate will be generated from that host name. + +Installation with the Let's Encrypt SSL +--------------------------------------- + +The installation script supports Let's Encrypt SSL generation out-of-box. To instruct the installation script to do that use ``--ctl-ssl-issuer`` argument. + +.. note:: + | The argument ``--ctl-ssl-issuer`` is passing ``cert-manager.io/cluster-issuer`` value to the ingress resource of the Netris Controller. The installation script can create two types of ClusterIssuer resource: ``selfsigned`` or ``letsencrypt``, where ``selfsigned`` is just `Cert-Manager self-signed `_ SSL and the ``letsencrypt`` is the ACME issuer with `HTTP01 challenge validation `_. + | If the ``--ctl-ssl-issuer`` argument is not set, the installation script will proceed with ``selfsigned`` ClusterIssuer type. + + +Run the following command to install Netris Controller and use ``letsencrypt`` ClusterIssuer for SSL generation: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + + To successfully validate and complete Let's Encrypt SSL generation, a valid A/CNAME record for the domain/subdomain name should exist prior, and that name must be accessible from the Internet. + + +Installation with the Custom SSL Issuer +--------------------------------------- + +The HTTP01 challenge validation is the simplest way of issuing the Let's Encrypt SSL, but it does not work when the host behind the FQDN is not accessible from the public internet. +The common approach of validating and completing Let's Encrypt SSL generation for private deployments is `DNS01 challenge validation `_. +If the ``DNS01`` does not work for you either, Cert-Manager supports a number of certificate issuers, get familiar with all types of issuers `here `_. + +In order to install Netris Controller with the custom SSL issuer, you need to run installation script with the specified host name: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com + +Once the installation is complete, create a yaml file with the ``ClusterIssuer`` resource, suitable for your requirements, and apply it: + +.. code-block:: shell-session + + kubectl apply -f my-cluster-issuer.yaml + +Then rerun the installation script with the ``--ctl-ssl-issuer`` argument: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-ssl-issuer + + +Upgrading +========= + +To upgrade the Netris Controller to the latest version simply run the script: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +If a newer version of Netris Controller is available, it will be updated in a few minutes. + + +Uninstalling +============ + +To uninstall Netris Controller and K3s from a server node, run: + +.. code-block:: shell-session + + /usr/local/bin/k3s-uninstall.sh + + +.. _ctl-backup-restore: + +Backup and Restore +================== + +Netris Controller stores all critical data in MariaDB. It's highly recommended to create a cronjob with ``mysqldump``. + + +Backup +------ + +To take database snapshot run the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot-$(date +%Y-%m-%d-%H-%M-%S).sql + +After command execution, you can find ``db-snapshot-YYYY-MM-DD-HH-MM-SS.sql`` file in the current working directory. + +.. _ctl-secret-key-backup: + + + +Backup the Secret Key +~~~~~~~~~~~~~~~~~~~~~ + +Netris Controller generates a unique secret key at the first installation. If you're moving or reinstalling your controller, it makes sense to take note of the secret key for restoring purpose in the future. Overwise, you have to reinitiate all devices connected to the controller. + +.. code-block:: shell-session + + kubectl -n netris-controller get secret netris-controller-grpc-secret -o jsonpath='{.data.secret-key}{"\n"}' + + +Restore +------- + +In order to restore DB from a database snapshot, follow these steps: + +1. Drop the current database by running the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"' + +2. Create a new database: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"' + +3. Copy snapshot file to the MariaDB container: + +.. code-block:: shell-session + + kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql + +4. Run the restore command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql' + +.. note:: + + In this example the snapshot file name is db-snapshot.sql and it's located in the current working directory + + +Restore the Secret Key +~~~~~~~~~~~~~~~~~~~~~~ + +If you want to restore the controller secret key too (you might want to do that if you're reinstalling or moving the controller to the other place), follow these steps: + +1. Set ``OLD_SECRET`` environment variable (the secret key taken from :ref:`the old controller`): + +.. code-block:: shell-session + + export OLD_SECRET= + +example: ``export OLD_SECRET=VUdodFFSakJCU2lFVVA4T1c0cnpuUmdiMkQxem85Y2dnS3pkajlNSg==`` + +2. Update the secret key of the new controller: + +.. code-block:: shell-session + + kubectl -n netris-controller patch secret netris-controller-grpc-secret --type='json' -p='[{"op" : "replace" ,"path" : "/data/secret-key" ,"value" : "'$OLD_SECRET'"}]' + +3. Restart Netris Controller's all microservices + +.. code-block:: shell-session + + kubectl -n netris-controller rollout restart deployments diff --git a/en/3.4/_sources/controller-k8s-installation.rst.txt b/en/3.4/_sources/controller-k8s-installation.rst.txt new file mode 100644 index 0000000000..b88d540a63 --- /dev/null +++ b/en/3.4/_sources/controller-k8s-installation.rst.txt @@ -0,0 +1,57 @@ + +.. meta:: + :description: Controller Helm Chart Installation + +####################### +Helm Chart Installation +####################### + +Requirements +------------ + +* Kubernetes 1.12+ +* Helm 3.1+ +* PV provisioner support in the underlying infrastructure + +Get Repo Info +------------- + +Add the Netris Helm repository: + +.. code-block:: shell-session + + helm repo add netrisai https://netrisai.github.io/charts + helm repo update + +Installing the Chart +-------------------- + +In order to install the Helm chart, you must follow these steps: + +1. Create the namespace for netris-controller: + +.. code-block:: shell-session + + kubectl create namespace netris-controller + +1. Install helm chart with netris-controller: + +.. code-block:: + + helm install netris-controller netrisai/netris-controller \ + --namespace netris-controller \ + --set ingress.hosts={my.domain.com} + +Uninstalling the Chart +--------------------------- + +To uninstall/delete the ``netris-controller`` helm release: + +.. code-block:: + + helm uninstall netris-controller + +Chart Configuration +------------------- + +See the `netris-controller README `_ for details about configurable parameters and their default values. diff --git a/en/3.4/_sources/controller-k8s-quickstart.rst.txt b/en/3.4/_sources/controller-k8s-quickstart.rst.txt new file mode 100644 index 0000000000..09c3eeea16 --- /dev/null +++ b/en/3.4/_sources/controller-k8s-quickstart.rst.txt @@ -0,0 +1,41 @@ +.. meta:: + :description: Controller Quickstart + +*********************** +Quickstart Installation +*********************** + +Netris offers a simplified deployment model for users who want to quickly install the Netris Controller in the shortest amount of time. + +**This installation process is streamlined for Linux servers that do not already have Kubernetes running.** The install does the following: + +* Installs `k3s `_ +* Installs the `Netris Controller Helm chart `_ + +If you wish to install the controller on an existing Kubernetes cluster, follow `these instructions `_ instead of this Quickstart. + +Quickstart Process +------------------ + +1. Install the Netris Controller w/ k3s by running the following command on your Linux server: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +2. When the installation completes, you will be provided with the login for the web UI. Login with the provided credentials. + +3. Navigate in the UI to Net→IPAM and add a new subnet that contains the desired management IP addresses you wish to use for your SoftGates and switches. + + For example, if you are planning on using 192.168.1.100 as the IP address of your Ubuntu server, then create a subnet in Netris UI for 192.168.1.0/24. + + Detailed configuration documentation is available here: `Netris IPAM `_. + +4. Navigate in the UI to **Topology** +5. Click the **Add** in the upper right +6. Fill out the fields for the SoftGate you wish to add +7. Select the proper **Management IP address** from the subnet selector +8. Once the SoftGate is created in the Topology, **right-click** on the SoftGate and select the **Install Agent** option +9. Copy the agent install command to your clipboard and run the command on the Ubuntu server you are using as your SoftGate +10. Congratulations. The SoftGate should now be connected to your controller. + diff --git a/en/3.4/_sources/controller-vm-installation.rst.txt b/en/3.4/_sources/controller-vm-installation.rst.txt new file mode 100644 index 0000000000..73b989dde7 --- /dev/null +++ b/en/3.4/_sources/controller-vm-installation.rst.txt @@ -0,0 +1,191 @@ +.. meta:: + :description: Controller Virtual Machine Installation + +**************************** +Virtual Machine Installation +**************************** + +Requirements +============ + +Minimal system requirements for the VM: + +* CPU - 4 Core +* RAM - 4 Gb +* Disk - 100Gb +* Network - 1 virtual NIC + +Recommended system requirements for the VM: + +* CPU - 8 Core +* RAM - 16 Gb +* Disk - 100Gb +* Network - 1 virtual NIC + +KVM Hypervisor Installation +=========================== +If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04) + +.. code-block:: shell-session + + sudo apt-get install virt-manager + +VM Controller Installation +========================== + +1. Download the Netris Controller image. (contact Netris support for repository access permissions). + +.. code-block:: shell-session + + cd /var/lib/libvirt/images + + sudo wget http://img.netris.io/netris-controller3.qcow2 + +2. Download VM definition file. + +.. code-block:: shell-session + + cd /etc/libvirt/qemu + + sudo wget http://img.netris.io/netris-controller3.xml + +3. Define the KVM virtual machine + +.. code-block:: shell-session + + sudo virsh define netris-controller3.xml + +.. note:: + + Netris Controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below for the network interface configuration example. + +Example: Network configuration on host (hypervisor) machine. + +.. note:: + + replace , and + with the correct NIC and IP for your host machine. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + #Physical NIC connected to the management network + auto + iface inet static + address 0.0.0.0/0 + + #bridge interface + auto br-mgmt + iface br-mgmt inet static + address + gateway + bridge-ports + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +4. Set the virtual machine to autostart and start it. + +.. code-block:: shell-session + + sudo virsh autostart netris-controller + +.. code-block:: shell-session + + sudo virsh start netris-controller + +Accessing the Netris Controller +=============================== +By default, Netris Controller will obtain an IP address from a **DHCP** server. + +Below steps describe how to configure a **Static IP** address for the Netris Controller. + +1. Connecting to the VM console. + +default credentials. **login**: ``netris`` **password**: ``newNet0ps`` + +.. code-block:: shell-session + + sudo virsh console netris-controller + +.. note:: + + Do not forget to change the default password (using passwd command). + +2. Setting a static IP address. + +Edit network configuration file. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +Example: IP configuration file. + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +Reload the network config. + +.. code-block:: shell-session + + sudo ifreload -a + +.. note:: + + Make sure Netris Controller has Internet access. + +3. Reboot the controller + +.. code-block:: shell-session + + sudo reboot + +After reboot, the Netris Controller GUI should be accessible using a browser. Use ``netris/newNet0ps`` credentials. + +.. image:: images/credentials.png + :align: center + :class: with-shadow + :alt: Netris Credentials + +.. note::Don’t forget to change the default password by clicking your login name in the top right corner and then clicking “Change Password”. + +Replacing the SSL certificate +============================= + +1. Replace the below file with your SSL certificate file. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.cert.pem; + +2. Replace the below file with your SSL private key. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.key.pem; + +3. Restart Nginx service. + +.. code-block:: shell-session + + systemctl restart nginx.service diff --git a/en/3.4/_sources/definitions.rst.txt b/en/3.4/_sources/definitions.rst.txt new file mode 100644 index 0000000000..c49a4277cd --- /dev/null +++ b/en/3.4/_sources/definitions.rst.txt @@ -0,0 +1,26 @@ +.. meta:: + :description: Definitions + +=========== +Definitions +=========== + +When configuring and operating a Netris system, the following nomenclature is important to understand: + +* **User** - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is ``netris``, with password ``newNet0ps``. + +* **Tenant** - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request and manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs) via a DevOps/NetOps pipeline. + +* **Permission Group** - List of permissions on a per section basis can be attached individually to a User or a User Role + +* **User Role** - Group of user permissions and tenants for role-based access control (RBAC) + +* **Site** - Each separate deployment (each data center) should be defined as a *Site*. All network units and resources are attached to a site. Netris Controller comes with a "default" site preconfigured. Site entry defines global attributes such as; AS numbers, default ACL policy, and Site Mesh (site to site VPN) type. + +* **Subnet** - IPv4/IPv6 address resources linked to *Sites* and *Tenants* + +* **Switch Port** - Physical ports of all switches attached to the system + +* **Inventory** - Inventory of all network units that are operated using Netris Agent + +* **E-BGP** - Defines all External BGP peers (iBGP and eBGP) diff --git a/en/3.4/_sources/index.rst.txt b/en/3.4/_sources/index.rst.txt new file mode 100644 index 0000000000..d7f6b5b37f --- /dev/null +++ b/en/3.4/_sources/index.rst.txt @@ -0,0 +1,91 @@ +.. Read the Docs Template documentation master file, created by + sphinx-quickstart on Tue Aug 26 14:19:49 2014. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to Netris Documentation +================================================== + +Learn how to get started with Netris VPC Networking for your network environment. + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + +.. toctree:: + :maxdepth: 2 + :caption: Tutorials + + tutorials/index + tutorials/vpc-anywhere + tutorials/netris-vpc-for-equinix-metal + tutorials/netris-vpc-for-phoenixnap-bmc + tutorials/netris-vpc-for-aws + tutorials/netris-vpc-for-gcp + +.. toctree:: + :maxdepth: 4 + :caption: Cloud Native Tools + + kubernetes-integration + terraform-integration + +.. toctree:: + :maxdepth: 2 + :caption: Netris Fundamentals + + introduction + netris-architecture + supported-networks + +.. toctree:: + :maxdepth: 2 + :caption: Detailed Installation + + installation + switch-agent-installation + SoftGate-installation + SoftGate-PRO-installation + +.. toctree:: + :maxdepth: 2 + :caption: Switch-fabric Configuration + + definitions + ipam + topology-management + switch-ports + +.. toctree:: + :maxdepth: 4 + :caption: Network Policies + + network-policies + +.. toctree:: + :maxdepth: 4 + :caption: Network Services + + vnet + l3-load-balancer + l4-load-balancer + acls + roh + +.. toctree:: + :maxdepth: 4 + :caption: Operations + + visibility + accounts + +.. toctree:: + :maxdepth: 4 + :caption: Updates + + release-notes + softgate-performance + +.. toctree:: + :maxdepth: 4 + :caption: Lab Scenarios + + sandbox/Sandbox1/onprem-k8s diff --git a/en/3.4/_sources/installation.rst.txt b/en/3.4/_sources/installation.rst.txt new file mode 100644 index 0000000000..421ea55b9e --- /dev/null +++ b/en/3.4/_sources/installation.rst.txt @@ -0,0 +1,20 @@ +.. meta:: + :description: Controller Installation + +======================= +Controller Installation +======================= + +Netris Controller can be installed locally as a VM, deployed as a Kubernetes application, or hosted in the Netris Cloud. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime. + +.. note:: + + Select ONE installation option below. + + +.. toctree:: + :maxdepth: 2 + :caption: Controller Installation + + controller-k3s-installation + controller-k8s-installation diff --git a/en/3.4/_sources/installing-netris-controller.rst.txt b/en/3.4/_sources/installing-netris-controller.rst.txt new file mode 100644 index 0000000000..cc7db17230 --- /dev/null +++ b/en/3.4/_sources/installing-netris-controller.rst.txt @@ -0,0 +1,37 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Installing a Netris Controller +============================== + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment. + +It doesn't matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. So you can access the console, and nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface. + +Linux Host requirements + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +In this example I am running my Netris controller on an AWS hosted virtual machine (EC2) which has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +I'm using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my EC2 : 54.219.211.71. + +.. image:: images/cloudflare-dns-record.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-hostname ” will instruct the installer to generate a Let’s Encrypt SSL certificate for the provided domain name. That’s why it is important to create the DNS record before this step. + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com + +Once installation process is finished you will be able to access your newly installed Netris Controller web consonle using netris/newNet0ps credentials. + +Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller. diff --git a/en/3.4/_sources/introduction.rst.txt b/en/3.4/_sources/introduction.rst.txt new file mode 100644 index 0000000000..c3769c815a --- /dev/null +++ b/en/3.4/_sources/introduction.rst.txt @@ -0,0 +1,11 @@ +.. meta:: + :description: Introduction to Netris + +Introduction to Netris +====================== + +Netris is an automatic netops software for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network's health and either applies software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation. + +.. image:: images/netris-architecture.png + :align: center + \ No newline at end of file diff --git a/en/3.4/_sources/inventory-profiles.rst.txt b/en/3.4/_sources/inventory-profiles.rst.txt new file mode 100644 index 0000000000..98737f45a5 --- /dev/null +++ b/en/3.4/_sources/inventory-profiles.rst.txt @@ -0,0 +1,26 @@ +.. meta:: + :description: Inventory Profiles + +================== +Inventory Profiles +================== + +Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/softgate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except netris-defined and user-defined custom flows. Generated rules include: + +* SSH from user defined subnets +* NTP from user defined ntp services +* DNS from user defined DNS servers +* Custom user defined rules + +.. csv-table:: Inventory Profile Fields + :file: tables/inventory-profile-fields.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup). + +.. image:: images/inventory-profile.png + :align: center + :class: with-shadow + :alt: Adding an inventory profile diff --git a/en/3.4/_sources/ipam.rst.txt b/en/3.4/_sources/ipam.rst.txt new file mode 100644 index 0000000000..c32d106271 --- /dev/null +++ b/en/3.4/_sources/ipam.rst.txt @@ -0,0 +1,96 @@ +.. meta:: + :description: IP Address Management + +.. _ipam_def: + +===================== +IP Address Management +===================== + +Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting. + +Purpose: +Users define specific roles(purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-net, NAT, etc… + + +Allocations and Subnets +----------------------- + +There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets. + + +.. image:: images/subnet-tree.png + :align: center + :alt: IPAM Tree View + :class: with-shadow + +IPAM Tree View + +-------------------------- + +Add an Allocation +----------------- + +#. Navigate to Net→IPAM +#. Click the **Add** button +#. Select **Allocation** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Allocation Fields + :widths: 25 50 + :header-rows: 0 + + * - Name + - Unique name for current allocation. + * - Prefix + - Unique prefix for allocation, must not overlap with other allocations. + * - Tenant + - Owner of the allocation. + +.. image:: images/add-allocation.png + :align: center + :class: with-shadow + :alt: Add a New IP Allocation + +Add Allocation Window + +-------------------------- + +Add a Subnet +------------ + +#. Navigate to Net→IPAM +#. Click the **Add** button +#. Select **Subnet** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Subnet fields + :widths: 25 50 + :header-rows: 0 + + * - **Name** + - Unique name for current subnet. + * - **Prefix** + - Unique prefix for subnet, ust be included in one of allocations. + * - **Tenant** + - Owner of the subnet. + * - **Purpose** + - This field describes for what kind of services the current subnet can be used. It can have the following values: + + - *common* - ordinary subnet, can be used in v-nets and ROH. + - *loopback* - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates). + - *management* - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates). + - *load-balancer* - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience. + - *nat* - hosts of this subnet or subnet itself can be used to define NAT services. + - *inactive* - can't be used in any services, useful for reserving/documenting prefixes for future use. + +.. image:: images/add-subnet.png + :align: center + :alt: Add a New Subnet + :class: with-shadow + +Add Subnet Window diff --git a/en/3.4/_sources/kubernetes-integration.rst.txt b/en/3.4/_sources/kubernetes-integration.rst.txt new file mode 100644 index 0000000000..94f6d2ea76 --- /dev/null +++ b/en/3.4/_sources/kubernetes-integration.rst.txt @@ -0,0 +1,477 @@ +.. meta:: + :description: Kubernetes Integration + +######################## +Kubernetes Integration +######################## +Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris-Kubernetes integration is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters. + + +Install Netris Operator +======================= + +Integration between the Netris Controller and the Kubernetes API is completed by installing the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart: + +Helm Chart Method +----------------- +Instructions are available on Github: https://github.com/netrisai/netris-operator/tree/master/deploy/charts/netris-operator#installing-the-chart + +Regular Manifest Method +----------------------- + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='http://**your-netris-controller-ip-or-host**' \ + --from-literal=login='**your-netris-admin-username**' --from-literal=password='**your-netris-admin-password**' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Using Type 'LoadBalancer' +========================= + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :class: with-shadow + :alt: Sandbox pod provisioning + + +After provisioning has finished, inspect the service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.202 9898:32584/TCP,9999:30365/TCP 9m17s + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9898 active 50.117.59.202 9898/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9898 active 50.117.59.202 9898/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.203 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.203 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :class: with-shadow + :alt: View L4 LB instances + +V-Net Custom Resource +--------------------- + +You can also create Netris V-Nets (L2 segments) via Kubernetes with a simple manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +BGP Custom Resource +------------------- + +You can create BGP peers via Kubernetes manifests: + +1. Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1092 + localIP: 50.117.59.118/30 + remoteIP: 50.117.59.117/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.192/28 le 32 + EOF + +2. Apply the manifest file: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +3. Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.118/30 50.117.59.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.118/30 50.117.59.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Calico CNI Integration +====================== + +Netris Operator can integrate with Calico CNI. This annotation will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.118/30 50.117.59.117/30 7m59s + sandbox9-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox9-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox9-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.118/30 50.117.59.117/30 8m41s + sandbox9-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox9-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox9-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.202 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. diff --git a/en/3.4/_sources/l3-load-balancer.rst.txt b/en/3.4/_sources/l3-load-balancer.rst.txt new file mode 100644 index 0000000000..99e9ef59a8 --- /dev/null +++ b/en/3.4/_sources/l3-load-balancer.rst.txt @@ -0,0 +1,44 @@ +.. meta:: + :description: Layer-3 Load Balancer (Anycast) + +.. _l3lb_def: + +############################# +L3 Load Balancer (Anycast LB) +############################# +L3 (Anycast) load balancer leverages ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks. + +ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server. + +End-user traffic should be destined to the anycast IP address. The switch fabric uses ECMP to load balance the traffic towards every server, and will hash sessions based on IP/Protocol/Port such that TCP sessions will exist between the given end-user and server pair for the lifetime of the session. Optional health checks are used to identify application failures and reroute traffic in the case of a server outage. + +Creating an L3 Load Balancer +============================ + +To configure L3 (Anycast) load balancing: + +#. Navigate to **Services→Instances (ROH)** and locate an existing ROH instance +#. Click the ellipses and then the **Edit** button +#. Select an extra IPv4 address from the select box at the bottom, and check the **Anycast** option. +#. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances. + +.. image:: images/add-l3-lb.png + :align: center + :class: with-shadow + :alt: Add a L3 LB + +Example: Adding an Anycast IPv4 address + +.. image:: images/list-l3-lb.png + :align: center + :class: with-shadow + :alt: List L3 LBs + +Example: Under Services→Load Balancer, you can find the listing of L3 (Anycast) Load Balancers, service statuses, and you can add/remove more ROH instances and/or health checks. + +.. image:: images/list-l3-lb-detail.png + :align: center + :class: with-shadow + :alt: List L3 LB Details + +Screenshot: L3 (Anycast) Load Balancer Detail diff --git a/en/3.4/_sources/l4-load-balancer.rst.txt b/en/3.4/_sources/l4-load-balancer.rst.txt new file mode 100644 index 0000000000..63de30d5fc --- /dev/null +++ b/en/3.4/_sources/l4-load-balancer.rst.txt @@ -0,0 +1,84 @@ +.. meta:: + :description: Netris Services and Configuration Examples + +.. _l4lb_def: + +####################### +L4 Load Balancer (L4LB) +####################### +Netris L4 Load Balancer (L4LB) leverages SoftGate(Linux router) nodes to provide Layer-4 load balancing services, including on-demand cloud load balancing with native integration with Kubernetes. + +Enabling L4LB service +--------------------- +L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer). + +The IP address pool for L4LB can be defined in the Net→IPAM section by adding an Allocation and setting the purpose field to ‘load-balancer’. You can define multiple IP pools for L4LB at any given site. See the below example. + +Example: Adding a load-balancer IP pool assignment. + +.. image:: images/add-allocation.png + :align: center + :class: with-shadow + :alt: Add an IP Allocation + + +Screenshot: Listing of Net→IPAM after adding a load-balancer assignment + +.. image:: images/list-subnets.png + :align: center + :class: with-shadow + :alt: List IP Subnets + + +Consuming L4LB service +---------------------- +This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section. + +Click +add under Services→L4 Load Balancer to request an L4LB service. + +Add new L4 Load Balancer fields are described below: + +**General fields** + +* **Name** - Unique name. +* **Protocol** - TCP or UDP. +* **Tenant** - Requestor Tenant should have access to the backend IP space. +* **Site** - Site where L4LB service is being requested for. Backends should belong on this site. +* **State** - Administrative state. + +**Frontend** + +* **Address** - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses. +* **Port** - TCP or UDP port to be exposed. + +**Health-check** + +* **Type** - Probe backends on service availability. + + * **None** - load balance unconditionally. + * **TCP** - probe backend service availability through TCP connect checks. + * **HTTP** - probe backend service availability through HTTP GET checks. + +* **Timeout(ms)** - Probe timeout in milliseconds. +* **Request path** - HTTP request path. + +**Backend** + +* **+Add** - add a backend host. +* **Address** - IP address of the backend host. +* **Port** - Service port on the backend host. +* **Enabled** - Administrative state of particular backend. + +.. image:: images/request-L4.png + :align: center + :class: with-shadow + :alt: Request an L4 Load Balancer + +Example: Requesting an L4 Load Balancer service. + +.. image:: images/list-l4-load-balancers.png + :align: center + :class: with-shadow + :alt: List L4 Load Balancers + +Example: Listing of L4 Load Balancer services \ No newline at end of file diff --git a/en/3.4/_sources/netris-architecture.rst.txt b/en/3.4/_sources/netris-architecture.rst.txt new file mode 100644 index 0000000000..fbec46980b --- /dev/null +++ b/en/3.4/_sources/netris-architecture.rst.txt @@ -0,0 +1,53 @@ +.. meta:: + :description: Netris Architecture + +.. _netris_architecture: + +################### +Netris Architecture +################### + +A Netris system is composed of 4 elements: + +* Netris Controller +* Netris Switch Agent +* Netris SoftGate +* Customer Application Servers + +.. _netris_controller_def: + +Netris Controller +================= +Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. The Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud. + +Diagram: High level Netris architecture + +.. image:: images/netris_controller_diagram.png + :align: center + +* **Controller HA** We highly recommend running more than one copy of the controller for database replication. +* **Multiple sites** Netris is designed to operate multiple sites with just a single controller with HA. +* **What if the controller is unreachable.** Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers core operations will not be affected. + +.. _netris_sw_agent: + +Netris Switch Agent +=================== +Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet. + +.. _netris_sg_agent: + +Netris SoftGate +=============== +Netris SoftGate is automatic configuration software and reference architecture for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), and site-to-site VPN function on a regular x86 server with a SmartNIC card. + +Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2. + +The server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities. + +Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels. + +Diagram: Netris SoftGate high level architecture + +.. image:: images/softgate_diagram.png + :align: center diff --git a/en/3.4/_sources/network-policies.rst.txt b/en/3.4/_sources/network-policies.rst.txt new file mode 100644 index 0000000000..91acf900af --- /dev/null +++ b/en/3.4/_sources/network-policies.rst.txt @@ -0,0 +1,348 @@ +.. meta:: + :description: Netris Network Policies & Protocol Configuration + +.. _bgp_def: + +######### +Basic BGP +######### + +BGP neighbors can be declared in the Net→E-BGP section. Netris will automatically generate and program the network configuration to meet the requirements. + +Adding BGP Peers +---------------- +#. Navigate to Net→E-BGP in the web UI +#. Click the **Add** button +#. Fill in the fields as described in the table below +#. Click the **Add** button + +.. csv-table:: BGP Peer Fields + :file: tables/bgp-basic.csv + :widths: 25, 75 + :header-rows: 0 + +Example: Declare a basic BGP neighbor + +.. image:: images/add-bgp-basic.png + :align: center + :class: with-shadow + +Example2: Declare BGP neighbor terminated on V-Net. Netris will automatically configure BGP session on the switch closest to the remote IP. + +.. image:: images/add-bgp-basic-2.png + :align: center + :class: with-shadow + + +############ +Advanced BGP +############ +BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies. + +Click Advanced to expand the BGP neighbor add/edit window. + +.. csv-table:: BGP Peer Fields - Advanced + :file: tables/bgp-advanced.csv + :widths: 25, 75 + :header-rows: 0 + +-------------------------- + +BGP Objects +----------- +| Under Net→E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy. +| Supported objects include: + +* IPv4 Prefix +* IPv6 Prefix +* AS-PATH +* Community +* Extended Community +* Large Community + +IPv4 Prefix +^^^^^^^^^^^ +| Rules defined one per line. +| Each line in IPv4 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv4 prefix (mandatory). +* Length - Possible values are: le , ge or ge le . + +Example: Creating an IPv4 Prefix list. + +.. image:: images/IPv4-Prefix.png + :align: center + :class: with-shadow + +IPv6 Prefix +^^^^^^^^^^^ +| Rules defined one per line. +| Each line in IPv6 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv6 prefix (mandatory). +* Keyword - Possible values are: le , ge or ge le . + +Example: Creating an IPv6 Prefix list. + +.. image:: images/IPv6-Prefix.png + :align: center + :class: with-shadow + +Community +^^^^^^^^^ +| Community field has two parts: + +* Action - Possible values: permit or deny (mandatory). +* Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive). + +Example: Creating community. + +.. image:: images/community.png + :align: center + :class: with-shadow + +-------------------------- + +BGP route-maps +-------------- +| Under the Net→E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound. +| Description of route-map fields: + +* **Sequence Number** - Automatically assigned a sequence number. Drag and move sequences to organize the order. +* **Description** - Free description. +* **Policy** - Permit or deny the routes which match below all match clauses within the current sequence. +* **Match** - Rules for route matching. + + * **Type** - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag. + * **Object** - Select an object from the list. + +* **Action** - Action when all match clauses are met. + + * **Action type** - Define whether to manipulate a particular BGP attribute or go to another sequence. + * **Attribute** - The attribute to be manipulated. + * **Value** - New attribute value. + +Example: route-map + +.. image:: images/route-map.png + :align: center + :class: with-shadow + +-------------------------- + +############## +Static Routing +############## +Located under Net→Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. +We recommend using the Routes only if BGP is not supported by the remote end. + +| Typical use cases for static routing: + +* To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported. +* Temporary interconnection with the old network for a migration. +* Routing a subnet behind a VM hypervisor machine for an internal VM network. +* Specifically routing traffic destined to a particular prefix through an out-of-band management network. + +| Add new static route fields description: + +* **Prefix** - Route destination to match. +* **Next-Hop** - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network. +* **Description** - Free description. +* **Site** - Site where Route belongs. +* **State** - Administrative (enable/disable) state of the Route. +* **Apply to** - Limit the scope to particular units. It's typically used for Null routes. + + +Example: Default route pointing to a Next-Hop that belongs to one of V-NETs. + +.. image:: images/defaultroute.png + :align: center + :class: with-shadow + +Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network. + +.. image:: images/static_route.png + :align: center + :class: with-shadow + +Screenshot: This Shows that my back route is actually applied on leaf1 and spine1. + +.. image:: images/leaf1_spine1.png + :align: center + :class: with-shadow + +-------------------------- + +.. _nat_def: + +### +NAT +### + +Netris SoftGate nodes are required to support NAT (Network Address Translation). + +Enabling NAT +------------ +To enable NAT for a given site, you first need to create a subnet with NAT purpose in the IPAM section. NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need. + +1. Allocate a public IP subnet for NAT under Net→IPAM. + +Example: Adding an IP allocation under Net→Subnets. + +.. image:: images/IP-allocation.png + :align: center + :class: with-shadow + +1. Attach NAT IP addresses and/or NAT IP Pools to just one SoftGate node. Other SoftGate Nodes on the same site will automatically add the same NAT IP/Pool resources for proper consistency and high availability. + +Example: Adding NAT IP addresses and NAT IP Address Pools to a SoftGate node. + +.. image:: images/NATIP-address.png + :align: center + :class: with-shadow + + +Defining NAT rules +------------------ +NAT rules are defined under Net→NAT. + +.. list-table:: NAT Rule Fields + :widths: 25 75 + :header-rows: 1 + + * - Name + - Unique name + * - **State** + - State of rule (enabled or disabled) + * - **Site** + - Site to apply the rule + * - **Action** + - *SNAT* - Replace the source IP address with specified NAT IP along with port overloading + *DNAT* - Replace the destination IP address and/or destination port with specified NAT IP + *ACCEPT* - Silently forward, typically used to add an exclusion to broader SNAT or DNAT rule + *MASQUERADE* - Replace the source IP address with the IP address of the exit interface + * - **Protocol** + - *All* - Match any IP protocol + *TCP* - Match TCP traffic and ports + *UDP* - Match UDP traffic and ports + *ICMP* - Match ICMP traffic + * - **Source** + - *Address* - Source IP address to match + *Port* - Source ports range to match with this value (TCP/UDP) + * - **Destination** + - *Address* - Destination IP address to match. In the case of DNAT it should be one of the predefined NAT IP addresses + *Port* - For DNAT only, to match a single destination port + *Ports* - For SNAT/ACCEPT only. Destination ports range to match with this value (TCP/UDP) + * - **DNAT to IP** + - The global IP address for SNAT to be visible on the Public Internet. The internal IP address for DNAT to replace the original destination address with + * - **DNAT to Port** + - The Port to which destination Port of the packet should be NAT'd + * - **Status** + - Administrative state (enable/disable) + * - **Comment** + - Free optional comment + + +Example: SNAT all hosts on 10.0.0.0/8 to the Internet using 198.51.100.65 as a global IP. + +.. image:: images/globalIP.png + :align: center + :class: with-shadow + +Example: Port forwarding. DNAT the traffic destined to 198.51.100.66:80 to be forwarded to the host 10.0.4.10 on port tcp/1080. + +.. image:: images/Port-Forwarding.png + :align: center + :class: with-shadow + +-------------------------- + +######## +SiteMesh +######## +SiteMesh is a Netris service for site-to-site interconnects over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site. + +Edit Net->Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below. + +* **Disabled** - Do not participate in SiteMesh. +* **Hub** - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites) +* **Spoke** - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites) +* **Dynamic Spoke** - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites) + +Screenshot: Site Mesh parameter editing a Site under Net→Sites. + +.. image:: images/Site_Mesh.png + :align: center + +You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge. + +.. image:: images/SiteMesh_modes.png + :align: center + +Check the Net→Site Mesh section for the listing of tunnel statuses. + +Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh) + +.. image:: images/SiteMesh_listing.png + :align: center + +-------------------------- + +############# +Looking Glass +############# +The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Net→Looking Glass then selecting the device from the top-left dropdown menu. + +Looking Glass controls described for IPv4/IPv6 protocol families. + +* **BGP Summary** - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes. +* **BGP Route** - Lookup the BGP table (RIB) for the given address. +* **Route** - Lookup switch routing table for the given address. +* **Traceroute** - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address. +* **Ping** - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address. + +Example: Spine1: listing BGP neighbors and number of received prefixes. + +.. image:: images/Spine1.png + :align: center + :class: with-shadow + +Example: BGP Route - looking up my leaf1 switch’s loopback address from spine1’s perspective. Spine1 is load balancing between two available paths. + +.. image:: images/BGP_route.png + :align: center + :class: with-shadow + +Example: Ping. + +.. image:: images/ping.png + :align: center + :class: with-shadow + +| Looking Glass controls described for the EVPN family. + +* **BGP Summary** - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received. +* **VNI** - List VNIs learned. +* **BGP EVPN** - List detailed EVPN routing information optionally for the given route distinguisher. +* **MAC table** - List MAC address table for the given VNI. + + +Example: Listing of adjacent BGP neighbors and number of EVPN prefixes received. + +.. image:: images/BGP_neighbors_listing.png + :align: center + :class: with-shadow + +Example: Listing MAC addresses on VNI 2. + +.. image:: images/MAC_listing.png + :align: center + :class: with-shadow + +Example: EVPN routing information listing for a specified route distinguisher. + +.. image:: images/EVPN_routing.png + :align: center + :class: with-shadow diff --git a/en/3.4/_sources/reference-designs.rst.txt b/en/3.4/_sources/reference-designs.rst.txt new file mode 100644 index 0000000000..8c1b63a61d --- /dev/null +++ b/en/3.4/_sources/reference-designs.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Network Reference Designs + +######################### +Network Reference Designs +######################### + +Netris can support any type of standard network design. + +The majority of designs fall into one of four patterns: + +#. Unmanaged Switch +#. Single L2 Domain +#. Collapsed Core +#. Data Center Leaf/Spine Designs (Modern High Throughput Redundant Design) +#. Data Center Core/Distribution/Access (Legacy High Throughput Redundant Design) \ No newline at end of file diff --git a/en/3.4/_sources/release-notes.rst.txt b/en/3.4/_sources/release-notes.rst.txt new file mode 100644 index 0000000000..ea6c869c70 --- /dev/null +++ b/en/3.4/_sources/release-notes.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Netris Release Notes + +############# +Release notes +############# +* DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGate performance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on. +* L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, we now support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates with Kubernetes providing cloud-like load balancer service (type: load-balancer). +* Kubenet​ - a network service purpose-built for Kubernetes cluster nodes. Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking with modern physical networking. +* API logs​ - Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type. +* Site Mesh​ - a Netris service for automatically configuring site-to-site interconnect over the public Internet. Site Mesh supports configuration for WireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a few clicks, services in one site get connectivity to services in other sites over a mesh of WireGuard tunnels. +* Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loop cable. Removed the need for IP address reservation for V-NET, switching entirely to the anycast default gateway. +* Controller distributions​ - Netris controller, is now available in three deployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application. 3) Managed/Hosted in the cloud. +* Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes. +* Switch/SoftGate agents​ - New installer with easy initial config tool. Support for IP and FQDN as a controller address. Authentication key. +* GUI​ - Improved Net→Topology section, becoming the main and required place for defining the network topology. All sections got a column organizer, so every user can order and hide/show columns to their comfort. diff --git a/en/3.4/_sources/roh.rst.txt b/en/3.4/_sources/roh.rst.txt new file mode 100644 index 0000000000..f93046ad12 --- /dev/null +++ b/en/3.4/_sources/roh.rst.txt @@ -0,0 +1,58 @@ +.. meta:: + :description: Routing on the Host + +.. _roh_def: + +######################### +ROH (Routing on the Host) +######################### +To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly on their servers. This is commonly known as ROH (Routing on the Host). + +In ROH architectures, servers use a routing daemon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon/suite is FRR. + +Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead an IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. This is a modern and optimal design, leveraging Layer-3 networking from the fabric to the servers. + +By using only Layer-3 interfaces, Layer-2 protocols such as Spanning Tree (STP) can be minized and the reliability of the network increases. + +The ROH architecture that is configured by Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). For each instance of ROH, you’ll need to create a ROH entry in Netris Controller. + +Adding ROH Hosts +---------------- + +#. Navigate in the Netris UI to **Services→Instances (ROH)** +#. Click the **Add** button +#. Fill out the form based on the fields in the table below. +#. Click the **Add** button + +Description of ROH instance fields: + +- **Name** - Unique name for the ROH instance +- **Site** - Site where the current ROH instance belongs +- **Type** - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor. +- **ROH Routing Profile** - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances + + - **Default route only (most common design)** - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch + - **Default + Aggregate** - Will add prefixes of defined assignments + "Default" profile + - **Full table** - Will advertise all prefixes available in the routing table of the connected switch + - **Inherit** - Will inherit policy from site objects defined under Net→Sites + +- **Legacy Mode** - Switch from default zero-config mode to using /30 IP addresses. Used for MSFT Windows Servers or other OS that doesn’t support FRR. +- **+Port** - Physical Switch Ports anywhere on the network. +- **+IPv4** - IPv4 addresses for the loopback interface. +- **+Inbound Prefix List** - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks. + +.. tip:: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +.. image:: images/ROH-instance.png + :align: center + :class: with-shadow + :alt: ROH Instances + +Example: Adding an ROH instance. (Yes, you can use A.B.C.0/32 and A.B.C.255/32) + +.. image:: images/ROH-listing.png + :align: center + :class: with-shadow + :alt: ROH Listings + +Expanded view of ROH listing. BGP sessions are up, and the expected IP is in fact received from the actual ROH server. Traffic stats are available per port. diff --git a/en/3.4/_sources/sandbox/Sandbox1/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox1/configurations.rst.txt new file mode 100644 index 0000000000..deb5a17756 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox1/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s1-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc1::1** (from the "**2607:f358:11:ffc1::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.24 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox1/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox1/creating-services.rst.txt new file mode 100644 index 0000000000..164555443f --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox1/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s1-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s1-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s1-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1012``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.0/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.1)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s1-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.4/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.4/32**" IP address. + + * This public IP is part of **45.38.161.4/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s1-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.8/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s1-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.24 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox1/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox1/index.rst.txt new file mode 100644 index 0000000000..7d11050256 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox1/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox1 # Sandbox name Uppercase(case sensitive) + sandbox1 # Sandbox name Lowercase + 166.88.17.24 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1011 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1012 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.0/28 # PUBLIC IPv4 Allocation + 45.38.161.0/30 # PUBLIC LOOPBACK subnet + 45.38.161.1 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.4/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.4/32 # CUSTOMER V-NET SNAT IP + 45.38.161.8/30 # L3LB Subnet + 45.38.161.8/32 # L3LB IP + 45.38.161.12/30 # L4LB Subnet + 45.38.161.13 # Second usable ip address in load-balancer subnet + 45.38.161.14 # Third usable ip address in load-balancer subnet + 45.38.161.18/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.17/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.22/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.21/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc1::/64 # public IPv6 subnet + 2607:f358:11:ffc1::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::3/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::2/127 # isp1-ipv6-example BGP peer remote IPv6 + s1-pre-configured # LINK + s1-learn-by-doing # LINK + s1-e-bgp # LINK + s1-v-net # LINK + s1-nat # LINK + s1-acl # LINK + s1-l3lb # LINK + s1-k8s # LINK + s1-topology # LINK + +Sandbox1 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt new file mode 100644 index 0000000000..9f4bfbe702 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s1-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox1.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox1.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.13 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.13:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.13 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.13 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.13 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.13 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.13 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.14 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.14 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.14 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.14 + + SRV05-NYC + curl 45.38.161.14 + + SRV05-NYC + curl 45.38.161.14 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1012 + localIP: 45.38.161.22/30 + remoteIP: 45.38.161.21/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.0/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.22/30 45.38.161.21/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.22/30 45.38.161.21/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.22/30 45.38.161.21/30 7m59s + sandbox1-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox1-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox1-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.22/30 45.38.161.21/30 8m41s + sandbox1-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox1-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox1-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.13 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox1/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox1/sandbox-info.rst.txt new file mode 100644 index 0000000000..12007f5c51 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox1/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s1-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox1.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.24 -p 30061 + srv02-nyc: ssh demo@166.88.17.24 -p 30062 + srv03-nyc: ssh demo@166.88.17.24 -p 30063 + srv04-nyc: ssh demo@166.88.17.24 -p 30064 + srv05-nyc: ssh demo@166.88.17.24 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1011 + Local Address: 45.38.161.18/30 + Remote Address: 45.38.161.17/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.0/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1011 + Local Address: 2607:f358:11:ffc0::3/127 + Remote Address: 2607:f358:11:ffc0::2/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc1::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1012 + Local Address: 45.38.161.22/30 + Remote Address: 45.38.161.21/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.0/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.0/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.0/30 + |___ NAT Subnet: 45.38.161.4/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.8/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.12/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc1::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc1::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox10/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox10/configurations.rst.txt new file mode 100644 index 0000000000..b36201e479 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox10/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s10-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffca::1** (from the "**2607:f358:11:ffca::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.19 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox10/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox10/creating-services.rst.txt new file mode 100644 index 0000000000..ab10b379dc --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox10/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s10-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s10-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s10-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1102``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.208/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.209)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s10-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.212/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.212/32**" IP address. + + * This public IP is part of **50.117.59.212/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s10-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.216/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s10-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.19 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox10/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox10/index.rst.txt new file mode 100644 index 0000000000..97891e80b2 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox10/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox10 # Sandbox name Uppercase(case sensitive) + sandbox10 # Sandbox name Lowercase + 166.88.17.19 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1101 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1102 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.208/28 # PUBLIC IPv4 Allocation + 50.117.59.208/30 # PUBLIC LOOPBACK subnet + 50.117.59.209 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.212/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.212/32 # CUSTOMER V-NET SNAT IP + 50.117.59.216/30 # L3LB Subnet + 50.117.59.216/32 # L3LB IP + 50.117.59.220/30 # L4LB Subnet + 50.117.59.221 # Second usable ip address in load-balancer subnet + 50.117.59.222 # Third usable ip address in load-balancer subnet + 50.117.59.122/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.121/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.126/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.125/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffca::/64 # public IPv6 subnet + 2607:f358:11:ffca::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::15/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::14/127 # isp1-ipv6-example BGP peer remote IPv6 + s10-pre-configured # LINK + s10-learn-by-doing # LINK + s10-e-bgp # LINK + s10-v-net # LINK + s10-nat # LINK + s10-acl # LINK + s10-l3lb # LINK + s10-k8s # LINK + s10-topology # LINK + +Sandbox10 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt new file mode 100644 index 0000000000..dc02e94869 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s10-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox10.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox10.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.221 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.221:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.221 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.221 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.221 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.221 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.221 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.222 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.222 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.222 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.222 + + SRV05-NYC + curl 50.117.59.222 + + SRV05-NYC + curl 50.117.59.222 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1102 + localIP: 50.117.59.126/30 + remoteIP: 50.117.59.125/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.208/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.126/30 50.117.59.125/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.126/30 50.117.59.125/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.126/30 50.117.59.125/30 7m59s + sandbox10-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox10-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox10-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.126/30 50.117.59.125/30 8m41s + sandbox10-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox10-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox10-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.221 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox10/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox10/sandbox-info.rst.txt new file mode 100644 index 0000000000..d9f285a121 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox10/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s10-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox10.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.19 -p 30061 + srv02-nyc: ssh demo@166.88.17.19 -p 30062 + srv03-nyc: ssh demo@166.88.17.19 -p 30063 + srv04-nyc: ssh demo@166.88.17.19 -p 30064 + srv05-nyc: ssh demo@166.88.17.19 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1101 + Local Address: 50.117.59.122/30 + Remote Address: 50.117.59.121/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.208/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1101 + Local Address: 2607:f358:11:ffc0::15/127 + Remote Address: 2607:f358:11:ffc0::14/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffca::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1102 + Local Address: 50.117.59.126/30 + Remote Address: 50.117.59.125/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.208/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.208/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.208/30 + |___ NAT Subnet: 50.117.59.212/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.216/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.220/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffca::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffca::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox11/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox11/configurations.rst.txt new file mode 100644 index 0000000000..a4080614ee --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox11/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s11-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcb::1** (from the "**2607:f358:11:ffcb::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.82 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox11/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox11/creating-services.rst.txt new file mode 100644 index 0000000000..4e2386cce4 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox11/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s11-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s11-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.82 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s11-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1112``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.118``. + 10. In the **Remote IP** field, type in ``45.38.161.117``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.96/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.97)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s11-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.82 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.100/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.100/32**" IP address. + + * This public IP is part of **45.38.161.100/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s11-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.104/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.104**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.104**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.104**) into the browser's address bar or simply visit `http://45.38.161.104/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.104 (name_45.38.161.104)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.104** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s11-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.82 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox11/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox11/index.rst.txt new file mode 100644 index 0000000000..2f648ba500 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox11/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox11 # Sandbox name Uppercase(case sensitive) + sandbox11 # Sandbox name Lowercase + 50.117.27.82 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1111 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1112 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.96/28 # PUBLIC IPv4 Allocation + 45.38.161.96/30 # PUBLIC LOOPBACK subnet + 45.38.161.97 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.100/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.100/32 # CUSTOMER V-NET SNAT IP + 45.38.161.104/30 # L3LB Subnet + 45.38.161.104/32 # L3LB IP + 45.38.161.108/30 # L4LB Subnet + 45.38.161.109 # Second usable ip address in load-balancer subnet + 45.38.161.110 # Third usable ip address in load-balancer subnet + 45.38.161.114/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.113/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.118/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.117/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcb::/64 # public IPv6 subnet + 2607:f358:11:ffcb::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::17/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::16/127 # isp1-ipv6-example BGP peer remote IPv6 + s11-pre-configured # LINK + s11-learn-by-doing # LINK + s11-e-bgp # LINK + s11-v-net # LINK + s11-nat # LINK + s11-acl # LINK + s11-l3lb # LINK + s11-k8s # LINK + s11-topology # LINK + +Sandbox11 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt new file mode 100644 index 0000000000..3403e69911 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s11-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox11.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox11.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.109 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.109:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.109 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.109 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.109 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.109 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.109 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.110 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.110 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.110 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.110 + + SRV05-NYC + curl 45.38.161.110 + + SRV05-NYC + curl 45.38.161.110 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1112 + localIP: 45.38.161.118/30 + remoteIP: 45.38.161.117/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.96/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.118/30 45.38.161.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.118/30 45.38.161.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.118/30 45.38.161.117/30 7m59s + sandbox11-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox11-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox11-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.118/30 45.38.161.117/30 8m41s + sandbox11-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox11-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox11-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.109 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox11/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox11/sandbox-info.rst.txt new file mode 100644 index 0000000000..8f5a79f0f9 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox11/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s11-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox11.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.82 -p 30061 + srv02-nyc: ssh demo@50.117.27.82 -p 30062 + srv03-nyc: ssh demo@50.117.27.82 -p 30063 + srv04-nyc: ssh demo@50.117.27.82 -p 30064 + srv05-nyc: ssh demo@50.117.27.82 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1111 + Local Address: 45.38.161.114/30 + Remote Address: 45.38.161.113/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.96/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1111 + Local Address: 2607:f358:11:ffc0::17/127 + Remote Address: 2607:f358:11:ffc0::16/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcb::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1112 + Local Address: 45.38.161.118/30 + Remote Address: 45.38.161.117/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.96/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.96/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.96/30 + |___ NAT Subnet: 45.38.161.100/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.104/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.108/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcb::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcb::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox12/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox12/configurations.rst.txt new file mode 100644 index 0000000000..24784ae823 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox12/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s12-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox12.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcc::1** (from the "**2607:f358:11:ffcc::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.83 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox12/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox12/creating-services.rst.txt new file mode 100644 index 0000000000..105b9d4454 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox12/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s12-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s12-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.83 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s12-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1122``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.126``. + 10. In the **Remote IP** field, type in ``45.38.161.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.128/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.129)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s12-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.83 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.132/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.132/32**" IP address. + + * This public IP is part of **45.38.161.132/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s12-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.136/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.136**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.136**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.136**) into the browser's address bar or simply visit `http://45.38.161.136/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.136 (name_45.38.161.136)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.136** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s12-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.83 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox12.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox12/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox12/index.rst.txt new file mode 100644 index 0000000000..004beb139b --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox12/index.rst.txt @@ -0,0 +1,61 @@ + +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox12 # Sandbox name Uppercase(case sensitive) + sandbox12 # Sandbox name Lowercase + 50.117.27.83 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1121 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1122 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.128/28 # PUBLIC IPv4 Allocation + 45.38.161.128/30 # PUBLIC LOOPBACK subnet + 45.38.161.129 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.132/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.132/32 # CUSTOMER V-NET SNAT IP + 45.38.161.136/30 # L3LB Subnet + 45.38.161.136/32 # L3lB IP + 45.38.161.140/30 # L4LB Subnet + 45.38.161.141 # Second usable ip address in load-balancer subnet + 45.38.161.142 # Third usable ip address in load-balancer subnet + 45.38.161.122/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.121/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.126/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.125/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcc::/64 # public IPv6 subnet + 2607:f358:11:ffcc::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::19/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::18/127 # isp1-ipv6-example BGP peer remote IPv6 + s12-pre-configured # LINK + s12-learn-by-doing # LINK + s12-e-bgp # LINK + s12-v-net # LINK + s12-nat # LINK + s12-acl # LINK + s12-l3lb # LINK + s12-k8s # LINK + s12-topology # LINK + +Sandbox12 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt new file mode 100644 index 0000000000..1029a22203 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s12-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox12.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox12.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.141 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.141:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.141 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.141 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.141 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.141 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.141 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.142 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.142 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.142 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.142 + + SRV05-NYC + curl 45.38.161.142 + + SRV05-NYC + curl 45.38.161.142 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1122 + localIP: 45.38.161.126/30 + remoteIP: 45.38.161.125/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.128/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.126/30 45.38.161.125/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.126/30 45.38.161.125/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.126/30 45.38.161.125/30 7m59s + sandbox12-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox12-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox12-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.126/30 45.38.161.125/30 8m41s + sandbox12-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox12-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox12-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.141 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox12/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox12/sandbox-info.rst.txt new file mode 100644 index 0000000000..b68063b030 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox12/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s12-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox12.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.83 -p 30061 + srv02-nyc: ssh demo@50.117.27.83 -p 30062 + srv03-nyc: ssh demo@50.117.27.83 -p 30063 + srv04-nyc: ssh demo@50.117.27.83 -p 30064 + srv05-nyc: ssh demo@50.117.27.83 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1121 + Local Address: 45.38.161.122/30 + Remote Address: 45.38.161.121/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.128/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1121 + Local Address: 2607:f358:11:ffc0::19/127 + Remote Address: 2607:f358:11:ffc0::18/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcc::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1122 + Local Address: 45.38.161.126/30 + Remote Address: 45.38.161.125/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.128/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.128/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.128/30 + |___ NAT Subnet: 45.38.161.132/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.136/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.140/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcc::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcc::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox13/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox13/configurations.rst.txt new file mode 100644 index 0000000000..f8aa5b37cc --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox13/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s13-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcd::1** (from the "**2607:f358:11:ffcd::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.84 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox13/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox13/creating-services.rst.txt new file mode 100644 index 0000000000..001fd9aca0 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox13/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s13-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s13-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.84 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s13-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1132``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.166``. + 10. In the **Remote IP** field, type in ``45.38.161.165``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.144/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.145)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s13-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.84 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.148/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.148/32**" IP address. + + * This public IP is part of **45.38.161.148/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s13-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.152/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.152**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.152**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.152**) into the browser's address bar or simply visit `http://45.38.161.152/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.152 (name_45.38.161.152)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.152** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s13-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.84 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox13/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox13/index.rst.txt new file mode 100644 index 0000000000..4c48c2cf2b --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox13/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox13 # Sandbox name Uppercase(case sensitive) + sandbox13 # Sandbox name Lowercase + 50.117.27.84 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1131 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1132 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.144/28 # PUBLIC IPv4 Allocation + 45.38.161.144/30 # PUBLIC LOOPBACK subnet + 45.38.161.145 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.148/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.148/32 # CUSTOMER V-NET SNAT IP + 45.38.161.152/30 # L3LB Subnet + 45.38.161.152/32 # L3LB IP + 45.38.161.156/30 # L4LB Subnet + 45.38.161.157 # Second usable ip address in load-balancer subnet + 45.38.161.158 # Third usable ip address in load-balancer subnet + 45.38.161.162/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.161/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.166/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.165/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcd::/64 # public IPv6 subnet + 2607:f358:11:ffcd::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1b/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1a/127 # isp1-ipv6-example BGP peer remote IPv6 + s13-pre-configured # LINK + s13-learn-by-doing # LINK + s13-e-bgp # LINK + s13-v-net # LINK + s13-nat # LINK + s13-acl # LINK + s13-l3lb # LINK + s13-k8s # LINK + s13-topology # LINK + +Sandbox13 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt new file mode 100644 index 0000000000..0982f397dd --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s13-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox13.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox13.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.157 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.157:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.157 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.157 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.157 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.157 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.157 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.158 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.158 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.158 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.158 + + SRV05-NYC + curl 45.38.161.158 + + SRV05-NYC + curl 45.38.161.158 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1132 + localIP: 45.38.161.166/30 + remoteIP: 45.38.161.165/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.144/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.166/30 45.38.161.165/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.166/30 45.38.161.165/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.166/30 45.38.161.165/30 7m59s + sandbox13-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox13-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox13-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.166/30 45.38.161.165/30 8m41s + sandbox13-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox13-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox13-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.157 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox13/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox13/sandbox-info.rst.txt new file mode 100644 index 0000000000..36d8ead019 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox13/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s13-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox13.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.84 -p 30061 + srv02-nyc: ssh demo@50.117.27.84 -p 30062 + srv03-nyc: ssh demo@50.117.27.84 -p 30063 + srv04-nyc: ssh demo@50.117.27.84 -p 30064 + srv05-nyc: ssh demo@50.117.27.84 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1131 + Local Address: 45.38.161.162/30 + Remote Address: 45.38.161.161/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.144/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1131 + Local Address: 2607:f358:11:ffc0::1b/127 + Remote Address: 2607:f358:11:ffc0::1a/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcd::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1132 + Local Address: 45.38.161.166/30 + Remote Address: 45.38.161.165/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.144/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.144/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.144/30 + |___ NAT Subnet: 45.38.161.148/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.152/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.156/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcd::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcd::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox14/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox14/configurations.rst.txt new file mode 100644 index 0000000000..66aad691d7 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox14/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s14-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox14.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffce::1** (from the "**2607:f358:11:ffce::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.85 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox14/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox14/creating-services.rst.txt new file mode 100644 index 0000000000..55e9e0fdd0 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox14/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s14-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s14-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.85 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s14-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1142``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.174``. + 10. In the **Remote IP** field, type in ``45.38.161.173``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.176/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.177)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s14-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.85 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.180/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.180/32**" IP address. + + * This public IP is part of **45.38.161.180/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s14-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.184/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.184**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.184**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.184**) into the browser's address bar or simply visit `http://45.38.161.184/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.184 (name_45.38.161.184)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.184** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s14-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.85 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox14.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox14/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox14/index.rst.txt new file mode 100644 index 0000000000..f48f80f4f6 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox14/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox15 # Sandbox name Uppercase(case sensitive) + sandbox15 # Sandbox name Lowercase + 50.117.27.85 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1141 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1142 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.176/28 # PUBLIC IPv4 Allocation + 45.38.161.176/30 # PUBLIC LOOPBACK subnet + 45.38.161.177 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.180/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.180/32 # CUSTOMER V-NET SNAT IP + 45.38.161.184/30 # L3LB Subnet + 45.38.161.184/32 # L3LB IP + 45.38.161.188/30 # L4LB Subnet + 45.38.161.189 # Second usable ip address in load-balancer subnet + 45.38.161.190 # Third usable ip address in load-balancer subnet + 45.38.161.170/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.169/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.174/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.173/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffce::/64 # public IPv6 subnet + 2607:f358:11:ffce::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1d/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1c/127 # isp1-ipv6-example BGP peer remote IPv6 + s14-pre-configured # LINK + s14-learn-by-doing # LINK + s14-e-bgp # LINK + s14-v-net # LINK + s14-nat # LINK + s14-acl # LINK + s14-l3lb # LINK + s14-k8s # LINK + s14-topology # LINK + +Sandbox14 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt new file mode 100644 index 0000000000..c7e87caaf6 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s14-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox15.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox15.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.189 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.189:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.189 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.189 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.189 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.189 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.189 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.190 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.190 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.190 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.190 + + SRV05-NYC + curl 45.38.161.190 + + SRV05-NYC + curl 45.38.161.190 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1142 + localIP: 45.38.161.174/30 + remoteIP: 45.38.161.173/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.176/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.174/30 45.38.161.173/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.174/30 45.38.161.173/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.174/30 45.38.161.173/30 7m59s + sandbox15-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox15-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox15-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.174/30 45.38.161.173/30 8m41s + sandbox15-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox15-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox15-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.189 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox14/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox14/sandbox-info.rst.txt new file mode 100644 index 0000000000..c28d67e462 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox14/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s14-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox14.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.85 -p 30061 + srv02-nyc: ssh demo@50.117.27.85 -p 30062 + srv03-nyc: ssh demo@50.117.27.85 -p 30063 + srv04-nyc: ssh demo@50.117.27.85 -p 30064 + srv05-nyc: ssh demo@50.117.27.85 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1141 + Local Address: 45.38.161.170/30 + Remote Address: 45.38.161.169/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.176/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1141 + Local Address: 2607:f358:11:ffc0::1d/127 + Remote Address: 2607:f358:11:ffc0::1c/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffce::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1142 + Local Address: 45.38.161.174/30 + Remote Address: 45.38.161.173/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.176/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.176/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.176/30 + |___ NAT Subnet: 45.38.161.180/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.184/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.188/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffce::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffce::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox15/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox15/configurations.rst.txt new file mode 100644 index 0000000000..a0e84b4a0c --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox15/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s15-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffcf::1** (from the "**2607:f358:11:ffcf::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@50.117.27.86 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox15/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox15/creating-services.rst.txt new file mode 100644 index 0000000000..0d5bcff2d0 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox15/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s15-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s15-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.86 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s15-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1152``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.214``. + 10. In the **Remote IP** field, type in ``45.38.161.213``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.192/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.193)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s15-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.86 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.196/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.196/32**" IP address. + + * This public IP is part of **45.38.161.196/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s15-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.200/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.200**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.200**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.200**) into the browser's address bar or simply visit `http://45.38.161.200/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**45.38.161.200 (name_45.38.161.200)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.200** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s15-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@50.117.27.86 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox15/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox15/index.rst.txt new file mode 100644 index 0000000000..1fbb4baf56 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox15/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox15 # Sandbox name Uppercase(case sensitive) + sandbox15 # Sandbox name Lowercase + 50.117.27.86 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1151 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1152 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.192/28 # PUBLIC IPv4 Allocation + 45.38.161.192/30 # PUBLIC LOOPBACK subnet + 45.38.161.193 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.196/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.196/32 # CUSTOMER V-NET SNAT IP + 45.38.161.200/30 # L3LB Subnet + 45.38.161.200/32 # L3LB IP + 45.38.161.204/30 # L4LB Subnet + 45.38.161.205 # Second usable ip address in load-balancer subnet + 45.38.161.206 # Third usable ip address in load-balancer subnet + 45.38.161.210/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.209/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.214/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.213/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcf::/64 # public IPv6 subnet + 2607:f358:11:ffcf::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1f/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1e/127 # isp1-ipv6-example BGP peer remote IPv6 + s15-pre-configured # LINK + s15-learn-by-doing # LINK + s15-e-bgp # LINK + s15-v-net # LINK + s15-nat # LINK + s15-acl # LINK + s15-l3lb # LINK + s15-k8s # LINK + s15-topology # LINK + +Sandbox15 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt new file mode 100644 index 0000000000..b871a06b17 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s15-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox15.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox15.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.205 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.205:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.205 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.205 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.205 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.205 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.205 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.206 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.206 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.206 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.206 + + SRV05-NYC + curl 45.38.161.206 + + SRV05-NYC + curl 45.38.161.206 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1152 + localIP: 45.38.161.214/30 + remoteIP: 45.38.161.213/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.192/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.214/30 45.38.161.213/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.214/30 45.38.161.213/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.214/30 45.38.161.213/30 7m59s + sandbox15-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox15-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox15-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.214/30 45.38.161.213/30 8m41s + sandbox15-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox15-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox15-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.205 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox15/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox15/sandbox-info.rst.txt new file mode 100644 index 0000000000..b79a32fc2c --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox15/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s15-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox15.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@50.117.27.86 -p 30061 + srv02-nyc: ssh demo@50.117.27.86 -p 30062 + srv03-nyc: ssh demo@50.117.27.86 -p 30063 + srv04-nyc: ssh demo@50.117.27.86 -p 30064 + srv05-nyc: ssh demo@50.117.27.86 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1151 + Local Address: 45.38.161.210/30 + Remote Address: 45.38.161.209/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.192/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1151 + Local Address: 2607:f358:11:ffc0::1f/127 + Remote Address: 2607:f358:11:ffc0::1e/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcf::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1152 + Local Address: 45.38.161.214/30 + Remote Address: 45.38.161.213/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.192/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.192/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.192/30 + |___ NAT Subnet: 45.38.161.196/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.200/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.204/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcf::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcf::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox2/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox2/configurations.rst.txt new file mode 100644 index 0000000000..0060b88be6 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox2/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s2-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc2::1** (from the "**2607:f358:11:ffc2::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.190 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox2/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox2/creating-services.rst.txt new file mode 100644 index 0000000000..4a22bdbcd3 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox2/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s2-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s2-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s2-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1022``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.32/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.33)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s2-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.36/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.36/32**" IP address. + + * This public IP is part of **45.38.161.36/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s2-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.40/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s2-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.190 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox2/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox2/index.rst.txt new file mode 100644 index 0000000000..b5f4205fd0 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox2/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox2 # Sandbox name Uppercase(case sensitive) + sandbox2 # Sandbox name Lowercase + 166.88.17.190 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1021 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1022 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.32/28 # PUBLIC IPv4 Allocation + 45.38.161.32/30 # PUBLIC LOOPBACK subnet + 45.38.161.33 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.36/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.36/32 # CUSTOMER V-NET SNAT IP + 45.38.161.40/30 # L3LB Subnet + 45.38.161.40/32 # L3LB IP + 45.38.161.44/30 # L4LB Subnet + 45.38.161.45 # Second usable ip address in load-balancer subnet + 45.38.161.46 # Third usable ip address in load-balancer subnet + 45.38.161.26/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.25/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.30/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.29/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc2::/64 # public IPv6 subnet + 2607:f358:11:ffc2::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::5/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::4/127 # isp1-ipv6-example BGP peer remote IPv6 + s2-pre-configured # LINK + s2-learn-by-doing # LINK + s2-e-bgp # LINK + s2-v-net # LINK + s2-nat # LINK + s2-acl # LINK + s2-l3lb # LINK + s2-k8s # LINK + s2-topology # LINK + +Sandbox2 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt new file mode 100644 index 0000000000..28a2d294ff --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s2-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox2.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.45 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.45:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.45 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.45 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.45 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.45 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.45 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.46 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.46 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.46 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.46 + + SRV05-NYC + curl 45.38.161.46 + + SRV05-NYC + curl 45.38.161.46 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1022 + localIP: 45.38.161.30/30 + remoteIP: 45.38.161.29/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.32/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.30/30 45.38.161.29/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.30/30 45.38.161.29/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.30/30 45.38.161.29/30 7m59s + sandbox2-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox2-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox2-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.30/30 45.38.161.29/30 8m41s + sandbox2-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox2-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox2-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.45 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox2/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox2/sandbox-info.rst.txt new file mode 100644 index 0000000000..557b923633 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox2/sandbox-info.rst.txt @@ -0,0 +1,135 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s2-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= + +https://sandbox2.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.190 -p 30061 + srv02-nyc: ssh demo@166.88.17.190 -p 30062 + srv03-nyc: ssh demo@166.88.17.190 -p 30063 + srv04-nyc: ssh demo@166.88.17.190 -p 30064 + srv05-nyc: ssh demo@166.88.17.190 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1021 + Local Address: 45.38.161.26/30 + Remote Address: 45.38.161.25/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.32/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1021 + Local Address: 2607:f358:11:ffc0::5/127 + Remote Address: 2607:f358:11:ffc0::4/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc2::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1022 + Local Address: 45.38.161.30/30 + Remote Address: 45.38.161.29/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.32/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.32/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.32/30 + |___ NAT Subnet: 45.38.161.36/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.40/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.44/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc2::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc2::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox3/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox3/configurations.rst.txt new file mode 100644 index 0000000000..d654c6afb9 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox3/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s3-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc3::1** (from the "**2607:f358:11:ffc3::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.189 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox3/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox3/creating-services.rst.txt new file mode 100644 index 0000000000..dd7db25126 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox3/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s3-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s3-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s3-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1032``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.48/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.49)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s3-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.52/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.52/32**" IP address. + + * This public IP is part of **45.38.161.52/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s3-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.56/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s3-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.189 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox3/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox3/index.rst.txt new file mode 100644 index 0000000000..45e3a10c1d --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox3/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox3 # Sandbox name Uppercase(case sensitive) + sandbox3 # Sandbox name Lowercase + 166.88.17.189 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1031 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1032 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.48/28 # PUBLIC IPv4 Allocation + 45.38.161.48/30 # PUBLIC LOOPBACK subnet + 45.38.161.49 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.52/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.52/32 # CUSTOMER V-NET SNAT IP + 45.38.161.56/30 # L3LB Subnet + 45.38.161.56/32 # L3LB IP + 45.38.161.60/30 # L4LB Subnet + 45.38.161.61 # Second usable ip address in load-balancer subnet + 45.38.161.62 # Third usable ip address in load-balancer subnet + 45.38.161.66/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.65/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.70/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.69/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc3::/64 # public IPv6 subnet + 2607:f358:11:ffc3::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::7/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::6/127 # isp1-ipv6-example BGP peer remote IPv6 + s3-pre-configured # LINK + s3-learn-by-doing # LINK + s3-e-bgp # LINK + s3-v-net # LINK + s3-nat # LINK + s3-acl # LINK + s3-l3lb # LINK + s3-k8s # LINK + s3-topology # LINK + +Sandbox3 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt new file mode 100644 index 0000000000..b283829cfb --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s3-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox3.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox3.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.61 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.61:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.61 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.61 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.61 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.61 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.61 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.62 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.62 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.62 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.62 + + SRV05-NYC + curl 45.38.161.62 + + SRV05-NYC + curl 45.38.161.62 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1032 + localIP: 45.38.161.70/30 + remoteIP: 45.38.161.69/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.48/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.70/30 45.38.161.69/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.70/30 45.38.161.69/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.70/30 45.38.161.69/30 7m59s + sandbox3-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox3-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox3-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.70/30 45.38.161.69/30 8m41s + sandbox3-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox3-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox3-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.61 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox3/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox3/sandbox-info.rst.txt new file mode 100644 index 0000000000..c7974c50b9 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox3/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s3-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox3.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.189 -p 30061 + srv02-nyc: ssh demo@166.88.17.189 -p 30062 + srv03-nyc: ssh demo@166.88.17.189 -p 30063 + srv04-nyc: ssh demo@166.88.17.189 -p 30064 + srv05-nyc: ssh demo@166.88.17.189 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1031 + Local Address: 45.38.161.66/30 + Remote Address: 45.38.161.65/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.48/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1031 + Local Address: 2607:f358:11:ffc0::7/127 + Remote Address: 2607:f358:11:ffc0::6/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc3::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1032 + Local Address: 45.38.161.70/30 + Remote Address: 45.38.161.69/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.48/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.48/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.48/30 + |___ NAT Subnet: 45.38.161.52/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.56/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.60/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc3::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc3::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox4/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox4/configurations.rst.txt new file mode 100644 index 0000000000..d0b2bff05a --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox4/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s4-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc4::1** (from the "**2607:f358:11:ffc4::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.188 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox4/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox4/creating-services.rst.txt new file mode 100644 index 0000000000..b5b92ba8a9 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox4/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s4-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s4-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s4-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1042``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 45.38.161.80/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(45.38.161.81)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s4-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.84/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.84/32**" IP address. + + * This public IP is part of **45.38.161.84/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s4-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.188 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. + +.. _s4-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.88/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox4/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox4/index.rst.txt new file mode 100644 index 0000000000..066815ccea --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox4/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox4 # Sandbox name Uppercase(case sensitive) + sandbox4 # Sandbox name Lowercase + 166.88.17.188 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1041 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1042 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.80/28 # PUBLIC IPv4 Allocation + 45.38.161.80/30 # PUBLIC LOOPBACK subnet + 45.38.161.81 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.84/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.84/32 # CUSTOMER V-NET SNAT IP + 45.38.161.88/30 # L3LB Subnet + 45.38.161.88/32 # L3LB IP + 45.38.161.92/30 # L4LB Subnet + 45.38.161.93 # Second usable ip address in load-balancer subnet + 45.38.161.94 # Third usable ip address in load-balancer subnet + 45.38.161.74/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.73/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.78/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.77/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc4::/64 # public IPv6 subnet + 2607:f358:11:ffc4::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::9/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::8/127 # isp1-ipv6-example BGP peer remote IPv6 + s4-pre-configured # LINK + s4-learn-by-doing # LINK + s4-e-bgp # LINK + s4-v-net # LINK + s4-nat # LINK + s4-acl # LINK + s4-l3lb # LINK + s4-k8s # LINK + s4-topology # LINK + +Sandbox4 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt new file mode 100644 index 0000000000..4d3db5eb0d --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s4-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox4.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox4.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 45.38.161.93 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.93:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.93 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 45.38.161.93 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 45.38.161.93 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 45.38.161.93 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 45.38.161.93 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 45.38.161.94 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.94 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.94 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.94 + + SRV05-NYC + curl 45.38.161.94 + + SRV05-NYC + curl 45.38.161.94 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1042 + localIP: 45.38.161.78/30 + remoteIP: 45.38.161.77/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 45.38.161.80/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.78/30 45.38.161.77/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 45.38.161.78/30 45.38.161.77/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 45.38.161.78/30 45.38.161.77/30 7m59s + sandbox4-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox4-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox4-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 45.38.161.78/30 45.38.161.77/30 8m41s + sandbox4-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox4-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox4-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.93 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox4/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox4/sandbox-info.rst.txt new file mode 100644 index 0000000000..8764b4816d --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox4/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s4-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox4.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.188 -p 30061 + srv02-nyc: ssh demo@166.88.17.188 -p 30062 + srv03-nyc: ssh demo@166.88.17.188 -p 30063 + srv04-nyc: ssh demo@166.88.17.188 -p 30064 + srv05-nyc: ssh demo@166.88.17.188 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1041 + Local Address: 45.38.161.74/30 + Remote Address: 45.38.161.73/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.80/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1041 + Local Address: 2607:f358:11:ffc0::9/127 + Remote Address: 2607:f358:11:ffc0::8/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc4::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1042 + Local Address: 45.38.161.78/30 + Remote Address: 45.38.161.77/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.80/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.80/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.80/30 + |___ NAT Subnet: 45.38.161.84/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.88/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.92/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc4::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc4::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox5/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox5/configurations.rst.txt new file mode 100644 index 0000000000..61782c29f7 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox5/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s5-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc5::1** (from the "**2607:f358:11:ffc5::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.187 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox5/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox5/creating-services.rst.txt new file mode 100644 index 0000000000..9d35e266d3 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox5/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s5-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s5-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s5-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1052``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.128/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.129)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s5-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.132/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.132/32**" IP address. + + * This public IP is part of **50.117.59.132/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s5-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.136/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s5-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.187 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox5/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox5/index.rst.txt new file mode 100644 index 0000000000..fabe3e986e --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox5/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox5 # Sandbox name Uppercase(case sensitive) + sandbox5 # Sandbox name Lowercase + 166.88.17.187 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1051 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1052 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.128/28 # PUBLIC IPv4 Allocation + 50.117.59.128/30 # PUBLIC LOOPBACK subnet + 50.117.59.129 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.132/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.132/32 # CUSTOMER V-NET SNAT IP + 50.117.59.136/30 # L3LB Subnet + 50.117.59.136/32 # L3LB IP + 50.117.59.140/30 # L4LB Subnet + 50.117.59.141 # Second usable ip address in load-balancer subnet + 50.117.59.142 # Third usable ip address in load-balancer subnet + 50.117.59.82/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.81/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.86/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.85/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc5::/64 # public IPv6 subnet + 2607:f358:11:ffc5::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::b/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::a/127 # isp1-ipv6-example BGP peer remote IPv6 + s5-pre-configured # LINK + s5-learn-by-doing # LINK + s5-e-bgp # LINK + s5-v-net # LINK + s5-nat # LINK + s5-acl # LINK + s5-l3lb # LINK + s5-k8s # LINK + s5-topology # LINK + +Sandbox5 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt new file mode 100644 index 0000000000..93061cfef0 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s5-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox5.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox5.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.141 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.141:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.141 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.141 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.141 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.141 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.141 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.142 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.142 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.142 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.142 + + SRV05-NYC + curl 50.117.59.142 + + SRV05-NYC + curl 50.117.59.142 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1052 + localIP: 50.117.59.86/30 + remoteIP: 50.117.59.85/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.128/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.86/30 50.117.59.85/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.86/30 50.117.59.85/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.86/30 50.117.59.85/30 7m59s + sandbox5-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox5-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox5-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.86/30 50.117.59.85/30 8m41s + sandbox5-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox5-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox5-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.141 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox5/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox5/sandbox-info.rst.txt new file mode 100644 index 0000000000..849e14f5a1 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox5/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s5-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox5.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.187 -p 30061 + srv02-nyc: ssh demo@166.88.17.187 -p 30062 + srv03-nyc: ssh demo@166.88.17.187 -p 30063 + srv04-nyc: ssh demo@166.88.17.187 -p 30064 + srv05-nyc: ssh demo@166.88.17.187 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1051 + Local Address: 50.117.59.82/30 + Remote Address: 50.117.59.81/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.128/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1051 + Local Address: 2607:f358:11:ffc0::b/127 + Remote Address: 2607:f358:11:ffc0::a/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc5::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1052 + Local Address: 50.117.59.86/30 + Remote Address: 50.117.59.85/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.128/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.128/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.128/30 + |___ NAT Subnet: 50.117.59.132/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.136/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.140/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc5::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc5::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox6/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox6/configurations.rst.txt new file mode 100644 index 0000000000..f7823f5139 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox6/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s6-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@166.88.17.186 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/3.4/_sources/sandbox/Sandbox6/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox6/creating-services.rst.txt new file mode 100644 index 0000000000..fd8cc5864f --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox6/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s6-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s6-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s6-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1062``. + 10. In the **Local IP** field, type in ``50.117.59.94`` + 11. In the **Remote IP** field, type in ``50.117.59.93``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.144/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s6-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.150/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s6-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@166.88.17.186 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/3.4/_sources/sandbox/Sandbox6/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox6/index.rst.txt new file mode 100644 index 0000000000..1af6cfa727 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox6/index.rst.txt @@ -0,0 +1,46 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox6 # sandbox name + 166.88.17.186 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1061 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1062 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.144/28 # customer public subnet + 50.117.59.154 # second usable ip address in load-balancer subnet + 50.117.59.155 # third usable ip address in load-balancer subnet + 50.117.59.90/30 # isp1-customer bgp peer local ip + 50.117.59.89/30 # isp1-customer bgp peer remote ip + 50.117.59.94/30 # isp2-customer bgp peer local ip + 50.117.59.93/30 # isp2-customer bgp peer remote ip + 50.117.59.150/32 # customer v-net nat ip + s6-pre-configured # LINKS + s6-learn-by-doing # LINKS + s6-e-bgp # LINKS + s6-v-net # LINKS + s6-nat # LINKS + s6-acl # LINKS + s6-k8s # LINKS + +Sandbox6 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt new file mode 100644 index 0000000000..8a268c4945 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt @@ -0,0 +1,618 @@ +.. _s6-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox6.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox6.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.154 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.154:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.154 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.155 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.155 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.155 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1062 + localIP: 50.117.59.94/30 + remoteIP: 50.117.59.93/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.144/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.94/30 50.117.59.93/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.94/30 50.117.59.93/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.94/30 50.117.59.93/30 7m59s + sandbox6-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox6-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox6-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.94/30 50.117.59.93/30 8m41s + sandbox6-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox6-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox6-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.154 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox6/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox6/sandbox-info.rst.txt new file mode 100644 index 0000000000..bbc9a9d779 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox6/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + +* **Netris Controller**: A cloud-hosted Netris Controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all operated by Netris. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + +Netris GUI +========== +https://sandbox6.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01: ssh demo@166.88.17.186 -p 30061 + srv02: ssh demo@166.88.17.186 -p 30062 + srv03: ssh demo@166.88.17.186 -p 30063 + srv04: ssh demo@166.88.17.186 -p 30064 + srv05: ssh demo@166.88.17.186 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1061 + IP customer: 50.117.59.90/30 + IP Iris: 50.117.59.89/30 + + Neighbor AS: 65007 + Vlan: 1062 + IP customer: 50.117.59.94/30 + IP Iris: 50.117.59.93/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 50.117.59.144/28 + diff --git a/en/3.4/_sources/sandbox/Sandbox7/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox7/configurations.rst.txt new file mode 100644 index 0000000000..f29a4f15e1 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox7/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s7-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc7::1** (from the "**2607:f358:11:ffc7::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.185 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox7/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox7/creating-services.rst.txt new file mode 100644 index 0000000000..4293c56812 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox7/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s7-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s7-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s7-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1072``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.160/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.161)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s7-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.164/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.164/32**" IP address. + + * This public IP is part of **50.117.59.164/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s7-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.168/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s7-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.185 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox7/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox7/index.rst.txt new file mode 100644 index 0000000000..04275ae820 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox7/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox7 # Sandbox name Uppercase(case sensitive) + sandbox7 # Sandbox name Lowercase + 166.88.17.185 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1071 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1072 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.160/28 # PUBLIC IPv4 Allocation + 50.117.59.160/30 # PUBLIC LOOPBACK subnet + 50.117.59.161 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.164/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.164/32 # CUSTOMER V-NET SNAT IP + 50.117.59.168/30 # L3LB Subnet + 50.117.59.168/32 # L3LB IP + 50.117.59.172/30 # L4LB Subnet + 50.117.59.173 # Second usable ip address in load-balancer subnet + 50.117.59.174 # Third usable ip address in load-balancer subnet + 50.117.59.98/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.97/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.102/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.101/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc7::/64 # public IPv6 subnet + 2607:f358:11:ffc7::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::f/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::e/127 # isp1-ipv6-example BGP peer remote IPv6 + s7-pre-configured # LINK + s7-learn-by-doing # LINK + s7-e-bgp # LINK + s7-v-net # LINK + s7-nat # LINK + s7-acl # LINK + s7-l3lb # LINK + s7-k8s # LINK + s7-topology # LINK + +Sandbox7 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt new file mode 100644 index 0000000000..851bdeef6a --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s7-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox7.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox7.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.173 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.173:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.173 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.173 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.173 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.173 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.173 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.174 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.174 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.174 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.174 + + SRV05-NYC + curl 50.117.59.174 + + SRV05-NYC + curl 50.117.59.174 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1072 + localIP: 50.117.59.102/30 + remoteIP: 50.117.59.101/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.160/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.102/30 50.117.59.101/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.102/30 50.117.59.101/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.102/30 50.117.59.101/30 7m59s + sandbox7-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox7-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox7-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.102/30 50.117.59.101/30 8m41s + sandbox7-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox7-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox7-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.173 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox7/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox7/sandbox-info.rst.txt new file mode 100644 index 0000000000..83a2632ea6 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox7/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s7-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox7.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.185 -p 30061 + srv02-nyc: ssh demo@166.88.17.185 -p 30062 + srv03-nyc: ssh demo@166.88.17.185 -p 30063 + srv04-nyc: ssh demo@166.88.17.185 -p 30064 + srv05-nyc: ssh demo@166.88.17.185 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1071 + Local Address: 50.117.59.98/30 + Remote Address: 50.117.59.97/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.160/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1071 + Local Address: 2607:f358:11:ffc0::f/127 + Remote Address: 2607:f358:11:ffc0::e/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc7::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1072 + Local Address: 50.117.59.102/30 + Remote Address: 50.117.59.101/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.160/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.160/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.160/30 + |___ NAT Subnet: 50.117.59.164/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.168/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.172/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc7::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc7::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox8/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox8/configurations.rst.txt new file mode 100644 index 0000000000..17a8c7bb74 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox8/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s8-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc8::1** (from the "**2607:f358:11:ffc8::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.29 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox8/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox8/creating-services.rst.txt new file mode 100644 index 0000000000..246d60f011 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox8/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s8-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s8-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.29 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s8-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1082``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.110``. + 10. In the **Remote IP** field, type in ``50.117.59.109``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.176/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.177)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s8-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.29 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.180/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.180/32**" IP address. + + * This public IP is part of **50.117.59.180/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s8-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.184/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s8-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.29 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox8/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox8/index.rst.txt new file mode 100644 index 0000000000..53b95ef0d1 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox8/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox8 # Sandbox name Uppercase(case sensitive) + sandbox8 # Sandbox name Lowercase + 166.88.17.29 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1081 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1082 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.176/28 # PUBLIC IPv4 Allocation + 50.117.59.176/30 # PUBLIC LOOPBACK subnet + 50.117.59.177 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.180/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.180/32 # CUSTOMER V-NET SNAT IP + 50.117.59.184/30 # L3LB Subnet + 50.117.59.184/32 # L3LB IP + 50.117.59.188/30 # L4LB Subnet + 50.117.59.189 # Second usable ip address in load-balancer subnet + 50.117.59.190 # Third usable ip address in load-balancer subnet + 50.117.59.106/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.105/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.110/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.109/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc8::/64 # public IPv6 subnet + 2607:f358:11:ffc8::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::11/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::10/127 # isp1-ipv6-example BGP peer remote IPv6 + s8-pre-configured # LINK + s8-learn-by-doing # LINK + s8-e-bgp # LINK + s8-v-net # LINK + s8-nat # LINK + s8-acl # LINK + s8-l3lb # LINK + s8-k8s # LINK + s8-topology # LINK + +Sandbox8 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt new file mode 100644 index 0000000000..83d3b92e2f --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s8-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox8.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox8.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.189 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.189:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.189 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.189 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.189 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.189 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.189 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.190 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.190 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.190 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.190 + + SRV05-NYC + curl 50.117.59.190 + + SRV05-NYC + curl 50.117.59.190 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1082 + localIP: 50.117.59.110/30 + remoteIP: 50.117.59.109/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.176/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.110/30 50.117.59.109/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.110/30 50.117.59.109/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.110/30 50.117.59.109/30 7m59s + sandbox8-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox8-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox8-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.110/30 50.117.59.109/30 8m41s + sandbox8-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox8-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox8-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.189 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox8/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox8/sandbox-info.rst.txt new file mode 100644 index 0000000000..2e728a56d4 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox8/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s8-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox8.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.29 -p 30061 + srv02-nyc: ssh demo@166.88.17.29 -p 30062 + srv03-nyc: ssh demo@166.88.17.29 -p 30063 + srv04-nyc: ssh demo@166.88.17.29 -p 30064 + srv05-nyc: ssh demo@166.88.17.29 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1081 + Local Address: 50.117.59.106/30 + Remote Address: 50.117.59.105/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.176/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1081 + Local Address: 2607:f358:11:ffc0::11/127 + Remote Address: 2607:f358:11:ffc0::10/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc8::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1082 + Local Address: 50.117.59.110/30 + Remote Address: 50.117.59.109/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.176/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.176/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.176/30 + |___ NAT Subnet: 50.117.59.180/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.184/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.188/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc8::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc8::/64 + diff --git a/en/3.4/_sources/sandbox/Sandbox9/configurations.rst.txt b/en/3.4/_sources/sandbox/Sandbox9/configurations.rst.txt new file mode 100644 index 0000000000..b5f67e54dd --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox9/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s9-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffc9::1** (from the "**2607:f358:11:ffc9::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@166.88.17.22 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/3.4/_sources/sandbox/Sandbox9/creating-services.rst.txt b/en/3.4/_sources/sandbox/Sandbox9/creating-services.rst.txt new file mode 100644 index 0000000000..78df2f26d5 --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox9/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s9-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s9-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.22 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s9-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1092``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.192/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.193)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s9-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.22 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.196/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.196/32**" IP address. + + * This public IP is part of **50.117.59.196/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s9-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.200/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s9-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@166.88.17.22 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/3.4/_sources/sandbox/Sandbox9/index.rst.txt b/en/3.4/_sources/sandbox/Sandbox9/index.rst.txt new file mode 100644 index 0000000000..351ff318dd --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox9/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox9 # Sandbox name Uppercase(case sensitive) + sandbox9 # Sandbox name Lowercase + 166.88.17.22 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1091 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1092 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.192/28 # PUBLIC IPv4 Allocation + 50.117.59.192/30 # PUBLIC LOOPBACK subnet + 50.117.59.193 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.196/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.196/32 # CUSTOMER V-NET SNAT IP + 50.117.59.200/30 # L3LB Subnet + 50.117.59.200/32 # L3LB IP + 50.117.59.204/30 # L4LB Subnet + 50.117.59.205 # Second usable ip address in load-balancer subnet + 50.117.59.206 # Third usable ip address in load-balancer subnet + 50.117.59.114/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.113/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.118/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.117/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc9::/64 # public IPv6 subnet + 2607:f358:11:ffc9::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::13/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::12/127 # isp1-ipv6-example BGP peer remote IPv6 + s9-pre-configured # LINK + s9-learn-by-doing # LINK + s9-e-bgp # LINK + s9-v-net # LINK + s9-nat # LINK + s9-acl # LINK + s9-l3lb # LINK + s9-k8s # LINK + s9-topology # LINK + +Sandbox9 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/3.4/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt b/en/3.4/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt new file mode 100644 index 0000000000..973ea4aa0e --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s9-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox9.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox9.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.205 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.205:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.205 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.205 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.205 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.205 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.205 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.206 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.206 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.206 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.206 + + SRV05-NYC + curl 50.117.59.206 + + SRV05-NYC + curl 50.117.59.206 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1092 + localIP: 50.117.59.118/30 + remoteIP: 50.117.59.117/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.192/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.118/30 50.117.59.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.118/30 50.117.59.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.118/30 50.117.59.117/30 7m59s + sandbox9-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox9-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox9-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.118/30 50.117.59.117/30 8m41s + sandbox9-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox9-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox9-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.205 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/3.4/_sources/sandbox/Sandbox9/sandbox-info.rst.txt b/en/3.4/_sources/sandbox/Sandbox9/sandbox-info.rst.txt new file mode 100644 index 0000000000..b9998b2d5b --- /dev/null +++ b/en/3.4/_sources/sandbox/Sandbox9/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s9-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox9.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@166.88.17.22 -p 30061 + srv02-nyc: ssh demo@166.88.17.22 -p 30062 + srv03-nyc: ssh demo@166.88.17.22 -p 30063 + srv04-nyc: ssh demo@166.88.17.22 -p 30064 + srv05-nyc: ssh demo@166.88.17.22 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1091 + Local Address: 50.117.59.114/30 + Remote Address: 50.117.59.113/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.192/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1091 + Local Address: 2607:f358:11:ffc0::13/127 + Remote Address: 2607:f358:11:ffc0::12/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc9::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1092 + Local Address: 50.117.59.118/30 + Remote Address: 50.117.59.117/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.192/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.192/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.192/30 + |___ NAT Subnet: 50.117.59.196/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.200/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.204/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc9::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc9::/64 + diff --git a/en/3.4/_sources/softgate-performance.rst.txt b/en/3.4/_sources/softgate-performance.rst.txt new file mode 100644 index 0000000000..1b1de3a508 --- /dev/null +++ b/en/3.4/_sources/softgate-performance.rst.txt @@ -0,0 +1,15 @@ +.. meta:: + :description: Netris Network Policies & Protocol Configuration + +#################### +SoftGate Performance +#################### + +The following tested results are offered to help properly size the hardware needed for a SoftGate with various types of services: + + +.. csv-table:: SoftGate Performance + :file: tables/softgate-performance.csv + :widths: 30, 10, 15, 15, 10, 10, 10 + :header-rows: 0 + diff --git a/en/3.4/_sources/supported-networks.rst.txt b/en/3.4/_sources/supported-networks.rst.txt new file mode 100644 index 0000000000..e68c713389 --- /dev/null +++ b/en/3.4/_sources/supported-networks.rst.txt @@ -0,0 +1,43 @@ +.. meta:: + :description: Reference Network Architectures + +############################### +Reference Network Architectures +############################### + + +Unmanaged Switch & Netris SoftGate +---------------------------------- + +.. image:: images/slide-1.png + :align: center + :alt: Unmanaged Switch & SoftGate + +-------------------------- + +Unmanaged Switch & SoftGate (HA) +--------------------------------------- + +.. image:: images/slide-2.png + :align: center + :alt: Unmanaged Switch & SoftGate (HA) + +-------------------------- + +Netris Managed Switch & SoftGate (HA) +-------------------------------------------- + +.. image:: images/slide-3.png + :align: center + :alt: Netris Managed Switch & SoftGate small data center (HA) + +-------------------------- + +Netris Managed Switch & SoftGate scalable data center (HA) +---------------------------------------------------------- + +.. image:: images/slide-4.png + :align: center + :alt: Netris Managed Switch & SoftGate scalable data center (HA) + + diff --git a/en/3.4/_sources/switch-agent-installation.rst.txt b/en/3.4/_sources/switch-agent-installation.rst.txt new file mode 100644 index 0000000000..9b49efe005 --- /dev/null +++ b/en/3.4/_sources/switch-agent-installation.rst.txt @@ -0,0 +1,15 @@ +.. _Network-Switch-initial-setup: +.. meta:: + :description: Network Switch initial setup + +############################ +Network Switch Initial Setup +############################ + +.. toctree:: + :maxdepth: 2 + + Nvidia-Cumulus-v5-Switch-initial-setup + Ubuntu-SwitchDev-Switch-initial-setup + EdgeCore-SONiC-Switch-initial-setup + Nvidia-Cumulus-v3.7-Switch-initial-setup diff --git a/en/3.4/_sources/switch-ports.rst.txt b/en/3.4/_sources/switch-ports.rst.txt new file mode 100644 index 0000000000..1a5946e3f1 --- /dev/null +++ b/en/3.4/_sources/switch-ports.rst.txt @@ -0,0 +1,46 @@ +.. meta:: + :description: Switch Ports + +============ +Switch Ports +============ + +Switch ports can be directly managed in the **Switch Port** UI section. Both physical and virtual ports (extended, aggregate, etc…) will appear in this section once they have been added to inventory. The Netris Controller will automatically sync the list of available ports that appear on each device. + +The following options are available for editing on each port: + +* Description - Description of the port. +* Tenant - Tenant to whom the port is assigned, by default it is the owner tenant of the device to whom the port belongs to. +* Breakout - Available only for physical switch ports, used to split physical ports into multiple physical ports. When there is a need to use other supported option supported by switch not shown in the dropdown list user must set breakout to "Manual" and configure breakout manually on the switch. For certain platforms some ports need to be disabled to support breakout into other ports, for that option use "Disable" mode of breakout. For Cumulus, after configuration, user must manually restart switchd daemon on the switch via command "systemctl restart switchd". +* MTU - Maximum transmission unit of the port. +* Autoneg - Toggle autonegotiation. Available only for physical ports. +* Speed - Toggle speed. Available only for physical ports. +* Duplex - Toggle duplex. Available only for physical ports. +* Extension - Create extension ports. Available for physical and aggregate ports. +* Extension Name - Name for new extension. +* VLAN Range - VLAN id range for new extension port. + +.. image:: images/edit-port.png + :align: center + :class: with-shadow + +Example: Edit physical port + + +Quick action menu provides following actions for ports (note that Bulk Action also available for multiple ports: + +Edit - Edit the port. +Admin UP/Down - Toggle admin status of the port. +Add to V-net - Add selected port(s) to a V-net. + +.. image:: images/quick-action-ports.png + :align: center + :class: with-shadow + +Add to LAG - Add selected ports into a LAG. + +.. image:: images/add-to-lag-port.png + :align: center + :class: with-shadow + +Free Up Port - Detach port from all resources. diff --git a/en/3.4/_sources/terraform-integration.rst.txt b/en/3.4/_sources/terraform-integration.rst.txt new file mode 100644 index 0000000000..7fd6bedad2 --- /dev/null +++ b/en/3.4/_sources/terraform-integration.rst.txt @@ -0,0 +1,317 @@ +.. meta:: + :description: Terraform: Netris provider + +########################## +Terraform: Netris provider +########################## + +Use Netris provider to interact with the many resources supported by Netris. You must configure the provider with the proper credentials before you can use it. +To learn the basics of Terraform using this provider, follow the hands-on `get started tutorials `_ on HashiCorp’s Learn platform. + +.. image:: images/diagrams_terraform.png + :align: center + +When you make changes in the Terraform files and apply them, Terraform automatically decides which part of your configuration is already deployed into Netris controller and what should be added or removed. + + +.. contents:: To create your first Terraform configuration: + :local: + + +Install Terraform +================= + +Download and install the `Terraform `_ + + + +Create a directory for Terraform files +====================================== + +#. Create a directory with any name, for example, ``netris-terraform``. It stores the configuration files and saved states for Terraform and your infrastructure. +#. Create a configuration file with the ``.tf`` extension in this directory, such as ``main.tf``. + + +Configure a provider +==================== + +1. At the beginning of the configuration file, specify the provider settings. + +.. code-block:: + + terraform { + required_providers { + netris = { + source = "netrisai/netris" + version = ">= 2.0.0" + } + } + } + + provider "netris" { + address = "" + login = "" + password = "" + } + +Specify the provider required arguments: + +* ``address`` - This is your Netris-Controller address (http://example.com). This can also be specified with the ``NETRIS_ADDRESS`` environment variable. +* ``login`` - This is your Netris-Controller login. This can also be specified with the ``NETRIS_LOGIN`` environment variable. +* ``password`` - This is your Netris-Controller password. This can also be specified with the ``NETRIS_PASSWORD`` environment variable. + +2. Execute the command ``terraform init`` in the folder with the configuration file. This command initializes the providers specified in the configuration files and lets you work with the provider resources and data sources. + + +Prepare an infrastructure plan +============================== + +By using Netris Terraform Provider, you can create all kinds of resources, such as Sites, IPAMs, Topology, Inventory, etc. +To create a resource, specify a set of required and optional parameters that define the resource properties. Such resource descriptions make up an infrastructure plan. + +Infrastructure provisioning in Netris starts with Site resources. The Netris-Controller comes with the initial site ``Default``. You can use it in your Terraform configuration files by getting its ID with the Terraform `Data source element `_. + +Let's create a separate file for site resource, and get its ID via Terraform Data source element. + +.. code-block:: + + cat << EOF > site.tf + data "netris_site" "default" { + name = "Default" + } + EOF + +Or, you can create a new ``Site`` resource, `here `_ is the detailed documentation with examples. + +Now, when we're clear on ``Site`` resource usage, let's define our IPAM. There are two types of IPAM resources in the Netris-Controller it’s ``Allocation`` and ``Subnet``. +IPAM resources only require ``tenantid`` field, let's get our default Admin tenant ID with the Data source element. + +.. code-block:: + + cat << EOF > tenant.tf + data "netris_tenant" "admin"{ + name = "Admin" + } + EOF + +Then, when we have the ``tenantid``, we can create IPAM resources. + +.. code-block:: + + cat << EOF > ipam.tf + resource "netris_allocation" "my-allocation-mgmt" { + name = "my-allocation-mgmt" + prefix = "192.0.2.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_allocation" "my-allocation-loopback" { + name = "my-allocation-loopback" + prefix = "198.51.100.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_allocation" "my-allocation-common" { + name = "my-allocation-common" + prefix = "203.0.113.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_subnet" "my-subnet-mgmt" { + name = "my-subnet-mgmt" + prefix = "192.0.2.0/24" + tenantid = data.netris_tenant.admin.id + purpose = "management" + defaultgateway = "192.0.2.254" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-mgmt, + ] + } + + resource "netris_subnet" "my-subnet-loopback" { + name = "my-subnet-loopback" + prefix = "198.51.100.0/24" + tenantid = data.netris_tenant.admin.id + purpose = "loopback" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-loopback, + ] + } + + resource "netris_subnet" "my-subnet-common" { + name = "my-subnet-common" + prefix = "203.0.113.0/25" + tenantid = data.netris_tenant.admin.id + purpose = "common" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-common, + ] + } + EOF + +With the command above, we've defined 6 resources, 3 of the type of Allocation, 3 of the type of Subnet, each Subnet resource has a different purpose. +For more details, get familiar with the IPAM `docs `_. + +Now, when we have all the required resources let's define our Inventory. +We're going to create 1 SoftGate, 1 switch and connect them with a link. + +.. code-block:: + + cat << EOF > inventory.tf + resource "netris_softgate" "my-softgate" { + name = "my-softgate" + tenantid = data.netris_tenant.admin.id + siteid = data.netris_site.default.id + description = "Softgate 1" + mainip = "auto" + mgmtip = "auto" + depends_on = [ + netris_subnet.my-subnet-mgmt, + netris_subnet.my-subnet-loopback, + ] + } + + resource "netris_switch" "my-switch" { + name = "my-switch" + tenantid = data.netris_tenant.admin.id + siteid = data.netris_site.default.id + description = "Switch 01" + nos = "cumulus_linux" + asnumber = "auto" + mainip = "auto" + mgmtip = "auto" + portcount = 16 + depends_on = [ + netris_subnet.my-subnet-mgmt, + netris_subnet.my-subnet-loopback, + ] + } + + resource "netris_link" "sg-to-sw" { + ports = [ + "swp1@my-softgate", + "swp16@my-switch" + ] + depends_on = [ + netris_softgate.my-softgate, + netris_switch.my-switch, + ] + } + EOF + +Next, let's define a local L3 network for our servers, suppose we want to connect 3 servers to our switch first 3 ports + +.. code-block:: + + cat << EOF > vnet.tf + resource "netris_vnet" "my-vnet" { + name = "my-vnet" + tenantid = data.netris_tenant.admin.id + state = "active" + sites{ + id = data.netris_site.default.id + gateways { + prefix = "203.0.113.1/25" + } + ports { + name = "swp1@my-switch" + vlanid = 1050 + } + ports { + name = "swp2@my-switch" + vlanid = 1050 + } + ports { + name = "swp3@my-switch" + } + } + depends_on = [ + netris_switch.my-switch, + netris_subnet.my-subnet-common, + ] + } + EOF + +And finally, we have to provide internet connectivity to our fabric, for that we'll define BGP resource. Suppose we're going to connect our ISP cable to the 10th port of our switch, and want to establish the BGP session on our Softgate. + +.. code-block:: + + cat << EOF > bgp.tf + data "netris_port" "swp10_my_switch"{ + name = "swp10@my-switch" + depends_on = [netris_switch.my-switch] + } + + resource "netris_bgp" "my-bgp" { + name = "my-bgp" + siteid = data.netris_site.default.id + hardware = "my-softgate" + neighboras = 23456 + portid = data.netris_port.swp10_my_switch.id + vlanid = 3000 + localip = "172.16.0.2/30" + remoteip = "172.16.0.1/30" + description = "My First BGP" + prefixlistinbound = ["deny 127.0.0.0/8 le 32", "permit 0.0.0.0/0 le 24"] + prefixlistoutbound = ["permit 192.0.2.0/24", "permit 198.51.100.0/24 le 25", "permit 203.0.113.0/24 le 26"] + depends_on = [netris_link.sg-to-sw] + } + EOF + +.. note:: + + For more information about all resources, how to create and manage them in Terraform, see the `provider's documentation `_. + +Now, when we've done with the configuration files, let's check whether they are valid + +.. code-block:: shell-session + + terraform validate + +If the configuration is valid, the following message is returned: + +.. code-block:: shell-session + + Success! The configuration is valid. + + + +Create resources +================ + +1. After preparing and checking the configuration, run the command: + +.. code-block:: shell-session + + terraform plan + +The terminal will display a list of resources with parameters. This is a test step. No resources are created. If there are errors in the configuration, Terraform points them out. + +2. To create resources, run the command: + +.. code-block:: shell-session + + terraform apply + +3. Confirm the resource creation: type ``yes`` in the terminal and press **Enter**. + +Terraform will create all the required resources and the terminal will display the progress. After creation, you can check resource availability and their settings in the Netris-Controller UI. + + +Delete resources +================ + +1. To delete resources created using Terraform: + +Run the command: + +.. code-block:: shell-session + + terraform destroy + +After the command is executed, the terminal will display a list of resources to be deleted. + +2. Type ``yes`` to confirm their deletion and press **Enter**. diff --git a/en/3.4/_sources/topology-management.rst.txt b/en/3.4/_sources/topology-management.rst.txt new file mode 100644 index 0000000000..0de0381821 --- /dev/null +++ b/en/3.4/_sources/topology-management.rst.txt @@ -0,0 +1,154 @@ +.. _topology-management: +.. meta:: + :description: Topology Management + +========= +Inventory +========= +The Inventory section allows you to add/edit/delete network switches and SoftGates. Initial setup of a Netris managed network is a three part process: + +#. Create Inventory Profiles +#. Adding Switches +#. Adding Softgates + +Inventory Profiles +================== +Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/SoftGate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except Netris-defined and user-defined custom flows. Generated rules include: + +* SSH from user defined subnets +* NTP from user defined ntp services +* DNS from user defined DNS servers +* Custom user defined rules + +.. csv-table:: Inventory Profile Fields + :file: tables/inventory-profile-fields.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup). + +.. image:: images/inventory-profile.png + :align: center + :class: with-shadow + +.. _topology-management-adding-switches: + +Adding Switches +=============== +Every switch needs to be added to the Netris Controller inventory. You can add new devices with the following process: + +#. Navigate to **Net→Inventory** +#. Click the **Add** button +#. Fill in the fields as described below +#. Click the **Add** button + + .. csv-table:: Add Inventory Fields - Switch + :file: tables/inventory-add-switch.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** Add a new Switch. + + .. image:: images/add-new-hardware.png + :align: center + :class: with-shadow + +.. note:: Repeat this process to define all your switches. + +.. _topology-management-adding-softgates: + +Adding SoftGates +================ +Every SoftGate node needs to be added to the Netris Controller inventory. To add a SoftGate node: + +#. Navigate to **Net→Topology** +#. Click **Add** +#. Fill in the fields as described below +#. Click the **Add** button + +.. csv-table:: Add Inventory Fields - SoftGate + :file: tables/inventory-add-softgate.csv + :widths: 25, 75 + :header-rows: 0 + +Example: Adding a SoftGate Node to Topology. + +.. image:: images/add-softgate.png + :align: center + :class: with-shadow + +Viewing Inventory +================= + +Inventory Listing shows also Heartbeat and monitoring statuses of each device. + +Heartbeat - Shows the status of device reachability. +Health - Shows number of successful and failed checks on the device. + + .. image:: images/inventory-listing.png + :align: center + :class: with-shadow + +.. note:: You can also add new devices in the Topology view. + +================ +Topology Manager +================ + +The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents will configure the underlying network devices according to this topology dynamically and start watching against potential failures. + +Adding Links +============ + +To define the links in the network: + +#. Right-click on the spine switch +#. Click **Create Link** +#. Select the **From Port** and the **To Port** + +See the example below: + +.. image:: images/create_link.png + :align: center + :class: with-shadow + +.. image:: images/topology_manager.png + :align: center + :class: with-shadow + +Once the links have been defined, the network is automatically configured as long as physical connectivity is in place and Netris Agents can communicate with Netris Controller. + +.. tip:: You can drag/move the units to your desired positions and click “Save positions”. + +Hairpin Links (Nvidia Cumulus only) +=================================== +With Nvidia Cumulus Linux only, we need to loop two ports on spine switches (hairpin cable) in the current release, usually two upstream (higher capacity) ports. We are planning to lift this requirement in the next Netris release (v2.10). + +To define what ports will be used as a hairpin, navigate to Net→Switch Ports, or right-click on the spine switch, click Ports in Net-->Topology. + +Example: Accessing Switch Ports from Net→Topology + +.. image:: images/switch_port.png + :align: center + :class: with-shadow + +For each spine switch, find the two ports that you are going to connect (loop/hairpin) and configure one port as a “hairpin **l2**” and another port as “hairpin **l3**”. The order doesn’t matter. The system needs to know which ports you have dedicated for the hairpin/loop on each spine switch. (do not do this for non-Cumulus switches) +| +Example: Editing Switch Port from Net→Switch Ports. + +.. image:: images/edit_switch_port.png + :align: center + :class: with-shadow + +Example: Setting port types to “hairpin l2” and “hairpin l3”. + +.. image:: images/hairpin.png + :align: center + :class: with-shadow + +Screenshot: Hairpin visualized in Net→Topology + +.. image:: images/hairpin_topology.png + :align: center + :class: with-shadow diff --git a/en/3.4/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt b/en/3.4/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..4ff8166f44 --- /dev/null +++ b/en/3.4/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt @@ -0,0 +1,27 @@ +####################################### +Activating BGP on Equinix Metal Project +####################################### + +Why use BGP with Equinix Metal? +SoftGate nodes are like border routers to your VPC, they are routing traffic between hosts inside your project and the Internet. We are going to establish 2 BGP sessions between SoftGate nodes and Equinix Metal. So there will be 4 BGP sessions total. + +We need these BGP sessions for moving further. In the next chapters we are going to request pools of public IP addresses, that Netris will automatically advertise to Equinix Metal, so inbound traffic “knows” how to reach Load Balancer, NAT, and other services that you will use within your VPC. + +.. image:: /tutorials/images/equinix-metal-bgp-diagram.png + :align: center + +You only need to activate BGP on the Equinix Metal Project. Netris will handle the rest. +In the Equinix Metal web console go to IPs & Networks → BGP then click Activate BGP on This Project. (see below screenshots) + +.. image:: /tutorials/images/equinix-metal-activate-bgp.png + :align: center + +Netris will handle the rest behind the scenes automatically. Netris will enable BGP peering on the Equinix Metal side, Netris will pull the metadata with the BGP info, and will automatically configure FRR (Free Range Routing BGP daemon) on both SoftGate nodes to bring up the BGP sessions up. + +After a few minutes you should see 4 new BGP sessions in your Netris web console under Net → E-BGP. (example screenshot below). + +.. image:: /tutorials/images/equinix-metal-netris-bgp-up.png + :align: center + + +Now your Netris VPC has established BGP sessions with Equinix Metal Project, and you can proceed to the next step. diff --git a/en/3.4/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt b/en/3.4/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt new file mode 100644 index 0000000000..3e865a8da0 --- /dev/null +++ b/en/3.4/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt @@ -0,0 +1,36 @@ +########################################################### +Provisioning Netris SoftGate nodes in Equinix Metal Project +########################################################### + +For SoftGate nodes you can start with two c3.small.x86 or larger servers. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one. + +Request two servers(c3.small.x86) from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned. + +1) At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory + +.. image:: /tutorials/images/softgate-nodes-created-in-equinix.png + :align: center + +2) When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”. + + Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location). + +Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + +.. image:: /tutorials/images/softgate-nodes-recognized-in-netris.png + :align: center + +3) Provision SoftGate nodes. + +Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there. + +.. image:: /tutorials/images/softgate-one-liner-provisioning.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/softgate-green.png + :align: center diff --git a/en/3.4/_sources/tutorials/aws-concept.rst.txt b/en/3.4/_sources/tutorials/aws-concept.rst.txt new file mode 100644 index 0000000000..97af34ccc8 --- /dev/null +++ b/en/3.4/_sources/tutorials/aws-concept.rst.txt @@ -0,0 +1,19 @@ +########################### +Site Mesh with AWS Overview +########################### + +Introduction +------------- + +This guide provides a step-by-step process to set up and configure Netris Softgate in AWS for establishing a site mesh network between the user's on-premises, AWS, and other cloud environments. + + +Concept +-------- + +Netris Softgate in AWS is an EC2 instance that runs the Netris software. Therefore, you'll first need to create an EC2 instance for Netris Softgate and install the Netris software on it. Once that's done, you'll need to configure the routes in your AWS VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your AWS VPC to access those destinations through the Netris Softgate EC2 instance. + +.. image:: images/aws-concept-traffic-flows.png + :align: center + +Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way. diff --git a/en/3.4/_sources/tutorials/aws-deploy-softgate.rst.txt b/en/3.4/_sources/tutorials/aws-deploy-softgate.rst.txt new file mode 100644 index 0000000000..081679bea6 --- /dev/null +++ b/en/3.4/_sources/tutorials/aws-deploy-softgate.rst.txt @@ -0,0 +1,183 @@ +.. meta:: + :description: Deploy a Softgate in AWS + +######################## +Deploy a Softgate in AWS +######################## + +As stated in the previous section, the following sequence of actions must be taken in order to proceed: create an EC2 instance, add Softgate into the Netris Controller, install Netris Softgate software on the EC2 instance, and configure routes in AWS VPC. Let us commence with these steps in the specified order. + +Create an EC2 instance +====================== + +Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated EC2. To achieve this, create a security group with the "All traffic" type and "Anywhere" source for both inbound and outbound rules. Afterward, an EC2 instance can be created using the security group above. + +.. image:: images/aws-security-group.png + :align: center + +To enable connectivity with other Netris sites, it is essential to create the EC2 instance in the desired VPC. Therefore, deploy a new EC2 instance with the Ubuntu 22.04 operating system installed, utilizing an instance type that meets the minimum hardware requirements of 2 virtual CPUs and 4 GB of RAM, such as t2.medium/t3.medium or any other type that satisfies these specifications. It is also recommended to allocate at least 30 GB of drive space. + + +.. image:: images/aws-softgate-deployed.png + :align: center + +After successfully deploying the EC2 instance, it is crucial to take note of its Public IPv4 address. This address will be required in the upcoming step. + + +Configure Netris Controller +=========================== + +Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed. + +Pre-Requisite Steps +------------------- + +In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below: + +1. Open the Netris Web Console. +2. Navigate to "Net" and select "Sites". +3. Click on the "+ Add" button. +4. Select "Dot1q Trunk" as the "Switch Fabric". +5. Input a descriptive name for the site. +6. Specify ``65500`` in the "Public ASN" field. +7. Click "Add" to create the new site. + +.. image:: images/aws-netris-site-create.png + :align: center + +Subsequently, it is necessary to create a private subnet for Softgate management. To achieve this, follow the steps below: + +1. Go to the "IPAM" section under the "Net" tab. +2. Click on the "+ Add" button located at the top-right corner. +3. Enter a unique "Prefix" for the new subnet, such as ``10.255.255.0/24``. +4. Type a descriptive "Name" for the subnet. +5. Select the desired tenant name from the "Tenant" dropdown menu. +6. From the "Type" dropdown menu, select "Subnet". +7. Select "management" from the "Purpose" dropdown menu. +8. Choose the appropriate site from the "Sites" dropdown menu. +9. Click on the "Add" button to create the subnet. + + +.. image:: images/aws-netris-ipam-mgmt.png + :align: center + +Following the creation of the private subnet for Softgate management, it is necessary to create another subnet with the "Purpose" of loopback. The "Prefix" for this subnet will be the Public IPv4 address of the AWS EC2 instance with a netmask length of /32. For instance, if the EC2 instance's IP address is 54.176.11.144, then the "Prefix" for the loopback subnet will be 54.176.11.144/32. However, unlike the subnet for Softgate management, an allocation for that "Prefix" must first be created before creating the subnet for loopback. + +.. image:: images/aws-netris-ipam-lo.png + :align: center + + +Add AWS VPC Subnet into the Netris Controller +--------------------------------------------- + +To register your AWS VPC's entire CIDR block into Netris IPAM, follow these steps: + +1. From the AWS Console, navigate to your VPC and take note of your CIDR blocks. +2. In Netris Controller, go to the "IPAM" section under the "Net" tab. +3. Click the "+ Add" button located at the top-right corner. +4. Enter the VPC CIDR block into the "Prefix" field for the new subnet. For example, if your CIDR block is "172.31.0.0/16", enter that value. +5. Type a descriptive name for the subnet. +6. From the "Tenant" dropdown menu, select the desired tenant name. +7. From the "Type" dropdown menu, select "Subnet". +8. Select "inactive" from the "Purpose" dropdown menu. +9. Choose the appropriate site from the "Sites" dropdown menu. +10. Click the "Add" button. + +.. image:: images/aws-vpc-cidr-to-netris.png + :align: center + + +Static route for AWS VPC Subnet in Netris Controller +---------------------------------------------------- + +We need to create route entry in Netris. The prefix for the route will be the AWS VPC CIDR block, and the next-hop will be the default gateway of Netris Softgate EC2. + +Here are the steps to create the static route: + +1. Securely log in to the Netris Softgate's EC2 instance using SSH. +2. Retrieve the default gateway address by typing the command ``ip route show default``. +3. In Netris Controller, go to the "Routes" section under the "Net" tab. +4. Click on the "+ Add" button located at the top-right corner of the screen. +5. Enter your VPC CIDR block in the "Prefix" field. +6. Enter the default gateway of the Netris Softgate EC2 instance in the "Next-Hop" field. +7. Select the appropriate site from the "Site" dropdown menu. +8. Finally, click on the "Add" button to create the static route. + + +.. image:: images/aws-netris-static-route.png + :align: center + + +Create the Softgate in the Netris Controller +-------------------------------------------- + +After completing all the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide: + +1. Ensure that you have completed all the pre-required steps. +2. Navigate to the "Net" tab in the Netris Controller and select the "Inventory" section. +3. Click on the "+Add" button to create a new Softgate. +4. Provide a descriptive name for the Softgate in the "Name" field. +5. From the "Tenant" dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets. +6. From the "Type" dropdown menu, select "SoftGate". +7. Choose the appropriate site from the "Site" dropdown menu. +8. For the "Main IP address" and "Management IP Address" fields, select "Assign automatically". +9. In the "Description" field, add ``int=eth0`` to specify that Netris should use the softgate's eth0 interface instead of the default bond0 interface that Netris Softgate Agent looks for. +10. Finally, click on the "Add" button to create the Softgate. + +.. image:: images/aws-netris-create-sg.png + :align: center + +Netris SoftGate node provisioning +--------------------------------- + +After creating a softgate resource in Netris Controller, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting "Install Agent". Copy the one-line installer command to your clipboard and connect to your EC2 instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it's done, reboot the server. + +.. image:: images/aws-netris-provision-sg.png + :align: center + + +Configure AWS VPC +================= + +Adding routes +------------- + +To enable specific traffic to be routed to the Netris Softgate EC2 instance in your AWS VPC, you need to modify your VPC's Route Table. To get started, go to the AWS Console and navigate to your VPC's Route Table. From there, click the "Edit routes" button to access the routing table, and then click "Add route" to create a new routing entry. + +In the "Destination" field, enter the subnet CIDR block for the Netris other Sites' subnets you want to access from this VPC. Next, in the "Target" field, select the "Instance" option and then select the Netris Softgate EC2 instance you previously created. This will ensure that traffic for those subnets is directed to the Softgate instance. + + +.. image:: images/aws-vpc-routes-created.png + :align: center + +EC2 Source / destination check +------------------------------ + +To allow Netris SoftGate to work properly, it is necessary to disable the 'Source / destination check' for the SoftGate's EC2 Instance. To do so, follow the steps below: + +1. Go to the AWS Console and navigate to the EC2 service page. +2. Select the SoftGate EC2 instance. +3. Click on the "Actions" button in the "Networking" section. +4. Select "Change Source / destination check". +5. Mark the "Stop" checkbox to disable the feature. +6. Click "Save" to apply the changes. + +.. image:: images/aws-ec2-stop-fwd-check.png + :align: center + + +By disabling the "Source / destination check", the SoftGate EC2 instance can receive and forward traffic between the Netris other Sites and AWS VPC subnets. + + +Enabling Site Mesh +================== + +To enable Site Mesh, in Netris Controller, navigate to the "Net" tab and select "Sites". Next, click on the three vertical dots (⋮) on the right side of the AWS site and select "Edit", and then from the "Site Mesh" dropdown menu, select "Hub". Save the changes. Repeat these steps for all sites that need to have meshed. + +.. image:: images/aws-netris-enable-site-mesh.png + :align: center + +The Site Mesh status can be viewed by navigating to the "Site Mesh" section under the "Net" tab. This will display the current status of Site Mesh for all Sites. + +.. image:: images/aws-netris-site-mesh-status.png + :align: center diff --git a/en/3.4/_sources/tutorials/create-interconnection-to-fabric.rst.txt b/en/3.4/_sources/tutorials/create-interconnection-to-fabric.rst.txt new file mode 100644 index 0000000000..4e6c530308 --- /dev/null +++ b/en/3.4/_sources/tutorials/create-interconnection-to-fabric.rst.txt @@ -0,0 +1,122 @@ +.. meta:: + :description: Creating an Interconnection to Equinix Fabric + +============================================= +Creating an Interconnection to Equinix Fabric +============================================= + +Once you have successfully set up Netris on Equinix Metal, you may want to use it to connect your private networks from Equinix Fabric back to Metal. + +Netris can easily manage the routing and other required VPC services for this. + + +**Create VLANs in Metal** + +To be able to create an interconnect into Equinix Fabric, first you must create a pair of VLANs for the Primary and Secondary connections. + +To do this, go to Equinix Console -> IP&Networks -> Layer 2. +Click "Add Vlan". +Choose the location of your Netris Softgates, then use a couple of VLAN IDs that are unused. + +In the example below, VLAN 1000 is used for Primary, and VLAN 1001 is used for the Secondary. + +.. image:: images/add-new-vlan-equinix.png + :align: center + + + +**Request Interconnection** + +Now we are ready to request the Equinix Interconnection on the Equinix Metal side. + +Go to Equinix Console -> IP&Networks -> Interconnections. + +Click "Request New Interconnection". + +From here, under Fabric VC select "Fabric Billed". + +Choose the Location where your Netris Softgates are located. + +Add an Interconnection name. This can be any name. + +Choose "Redundant interconnection". + +Under "Primary Metal VLAN" select the VLAN created in the previous step, in this example it is VLAN 1000. + +Under "Secondary Metal VLAN" select the VLAN created in the previous step, in this example it is VLAN 1001. + +Then, finalize the request by clicking "Submit Interconnection Request". + +.. image:: images/add-interconnection-request.png + :align: center + + + + +Once this is accomplished, use the token generated to connect your virtual Network Edge device to Equinix Metal. Setting up a connection on the Fabric side is outside the scope of this document. Once that is complete, the Interconnection will show as Active, and you may move onto creating BGP connections with the Network Edge Virtual Device. + + + +**Primary BGP Connection to Fabric Network Edge Device** + +Up to this point, a redundant pair of layer 2 connections have been created and connected back to an Equinix Fabric Network Edge device (or some other port with a router on the other end). To be able to utilize the redundancy of both connections, a BGP peering must be established over the two layer 2 VLANs. + +On the Netris Controller, go to Net -> E-BGP, then click Add. + +Fill in the following information: + +.. code-block:: None + + + Name: Choose any name + Site: Choose the site of your Equinix Project + V-Net: None + BGP Router: Choose the primary Softgate + VLAN ID: Choose the Primary VLAN ID created when creating the Interconnection in the previous steps + Neighbor AS: Choose the ASN configured on the Network Edge router + IP Family: IPv4 + Local IP: Any /30 can be used. Using a Private IP is recommended. + Remote IP: Use the IP of the remote side. + + +.. image:: images/add-equinix-bgp-primary.png + :align: center + + + + +**Secondary BGP Connection to Fabric Network Edge Device** + +Now to create the backup/secondary connection. + + +On the Netris Controller, go to Net -> E-BGP, then click Add. + +Fill in the following information: + +.. code-block:: None + + + Name: Choose any name + Site: Choose the site of your Equinix Project + V-Net: None + BGP Router: Choose the secondary Softgate + VLAN ID: Choose the Secondary VLAN ID created when creating the Interconnection in the previous steps + Neighbor AS: Choose the ASN configured on the Network Edge router + IP Family: IPv4 + Local IP: Any /30 can be used. Using a Private IP is recommended. + Remote IP: Use the IP of the remote side. + + + +Once completed, the EBGP status shoudl look something like this: + +.. image:: images/equinix-ebgp-links-up.png + :align: center + + + + +.. note:: + At this point, you should have both BGP sessions up, but the link may show yellow if the Network Edge device is not advertising any routes back to the Netris Softgates. Once the Network Edge router begins sending routes from other connections, the status should turn green. + diff --git a/en/3.4/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt b/en/3.4/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..66a3a58cec --- /dev/null +++ b/en/3.4/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt @@ -0,0 +1,68 @@ +####################################################### +Enabling services (NAT, V-Net, Load Balancer, IP pools) +####################################################### + +Although bare metal servers in Equinix Metal Project get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities. + +Both NAT and on-demand Load Balancer services need public IP addresses. + + +1) Requesting new Public IP address block +========================================= + +Go to Equinix Metal web console and click on IPs & Networks → IPs (see the screenshot below) + +In this example, I’m requesting two IP address blocks, one /30 (4 IPs) for NAT and one /28 (16 IPs) for Load Balancer. + +It’s important to tag IP blocks as “netris”. This is a signal for Netris Controller that this block is intended for Netris VPC. + +You can always request more IP address blocks in the future. Also it is possible to request a large block and then use Netris IPAM for crushing it into smaller blocks. You can read more about Netris IPAM in Netris docs. + +.. image:: /tutorials/images/equinix-metal-request-ip-block.png + :align: center + +Once IP address blocks are provisioned on Equinix Metal Project you should be able to find them automatically replicated in Netris web console under Net → IPAM + +.. image:: /tutorials/images/equinix-metal-netris-ipam-synced.png + :align: center + +You don’t need to worry about advertising them over BGP, Netris will handle that automatically when that makes sense (associated with any service). + + +2) Enable on-demand (elastic) Load Balancer +=========================================== + +To Enable on-demand (elastic) Load Balancer you only need to change the “purpose” field of appropriate IP address block from “common” into “load-balancer” + +Click on the 3 dots menu (in this example of /28 IP address block), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field. + +.. image:: /tutorials/images/netris-enable-elb.png + :align: center + +Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform. + +Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it. + + +3) Enable V-Net +=============== + +V-Net is a service for virtual private networks. You need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources. + +.. image:: /tutorials/images/netris-create-common-subnets.png + :align: center + +4) Enable NAT +============= + +To enable NAT, you need to repurpose a block of IP addresses for NAT. In the below example I’m repurposing the newly requested /30 subnet for NAT. + +.. image:: /tutorials/images/netris-ipam-nat.png + :align: center + +Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. In this example I’m enabling SNAT for the entire 10.0.0.0/8 private network, so basically I just want to ensure that VMs that will get IPs from private networks will get outbound Internet access through NAT. You can always have more granular control either through NAT rule or using Services → ACLs. + +.. image:: /tutorials/images/netris-create-nat-rule.png + :align: center + +At this point the minimal configuration of Netris VPC networking is done, next chapters will describe how to consume the VPC, how to request resources and services. diff --git a/en/3.4/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt b/en/3.4/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt new file mode 100644 index 0000000000..04bc58dc37 --- /dev/null +++ b/en/3.4/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt @@ -0,0 +1,51 @@ +.. meta:: + :description: Getting Started for Equinix Metal + +#################################### +Enable Equinix Metal API integration +#################################### + + +For each Equinix Metal Project+location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "Equinix Metal" from the dropdown menu. + * - Name + - Type a descriptive name for your Equinix Metal Project+location. + * - Equinix Project ID + - Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID. + * - Equinix Project API key + - Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here. + * - Equinix Location + - Select your equinix location from the dropdown menu. + + +Equinix Metal Project ID + +.. image:: /tutorials/images/equinix-metal-project-id.png + :align: center + + +Equinix Metal Project API key + +.. image:: /tutorials/images/equinix-metal-project-api-keys.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/netris-create-equinix-metal-site.png + :align: center + + diff --git a/en/3.4/_sources/tutorials/gcp-concept.rst.txt b/en/3.4/_sources/tutorials/gcp-concept.rst.txt new file mode 100644 index 0000000000..c3f4a1e315 --- /dev/null +++ b/en/3.4/_sources/tutorials/gcp-concept.rst.txt @@ -0,0 +1,19 @@ +########################### +Site Mesh with GCP Overview +########################### + +Introduction +------------- + +This guide provides a step-by-step process to set up and configure Netris Softgate in GCP for establishing a site mesh network between the user's on-premises, GCP, and other cloud environments. + + +Concept +-------- + +Netris Softgate in GCP is an VM instance that runs the Netris software. Therefore, you'll first need to create an VM instance for Netris Softgate and install the Netris software on it. Once that's done, you'll need to configure the routes in your GCP VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your GCP VPC to access those destinations through the Netris Softgate VM instance. + +.. image:: images/gcp-concept-traffic-flows.png + :align: center + +Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way. diff --git a/en/3.4/_sources/tutorials/gcp-deploy-softgate.rst.txt b/en/3.4/_sources/tutorials/gcp-deploy-softgate.rst.txt new file mode 100644 index 0000000000..d0cec6a7b4 --- /dev/null +++ b/en/3.4/_sources/tutorials/gcp-deploy-softgate.rst.txt @@ -0,0 +1,196 @@ +.. meta:: + :description: Deploy a Softgate in GCP + +######################## +Deploy a Softgate in GCP +######################## + +As stated in the previous section, the following sequence of actions must be taken in order to proceed: create a VM instance, add Softgate into the Netris Controller, install Netris Softgate software on the VM instance, and configure routes in GCP VPC. Let's commence with these steps in the specified order. + +Create a VM instance +==================== + +Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated VM instance. To achieve this, create a VPC firewall rule with the following parameters: + +* Name: Any name +* Direction of traffic: "Ingress" +* Action on match: "Allow" +* Targets: "Specified target tags" +* Target tags: ``allow-all`` +* Source filter: "IPv4 ranges" +* Source IPv4 ranges: ``0.0.0.0/0`` +* Protocols and ports: "Allow all" + +Afterward, a VM instance can be created using the network tag above. + +.. image:: images/gcp-firewall-rule.png + :align: center + +To enable connectivity with other Netris sites, creating the VM instance in the desired VPC is essential. Therefore, deploy a new VM instance with the **Ubuntu 22.04** operating system installed, utilizing an instance type that meets the minimum hardware requirements of **2** virtual CPUs, **4 GB** of RAM, and **30 GB** of drive space, such as e2-medium or any other type that satisfies these specifications. From the Advanced Networking options enable ``IP forwarding`` and specify the previously created ``allow-all`` network tag. + +.. image:: images/gcp-softgate-deployed.png + :align: center + +After successfully deploying the VM instance, it is crucial to take note of its External IP. This address will be required in the upcoming step. + + +Configure Netris Controller +=========================== + +Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed. + +Pre-Requisite Steps +------------------- + +In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below: + +1. Open the Netris Web Console. +2. Navigate to "Net" and select "Sites". +3. Click on the "+ Add" button. +4. Select "Dot1q Trunk" as the "Switch Fabric". +5. Input a descriptive name for the site. +6. Specify ``65500`` in the "Public ASN" field. +7. Click "Add" to create the new site. + +.. image:: images/gcp-netris-site-create.png + :align: center + +Subsequently, it is necessary to create a private subnet for Softgate management. To achieve this, follow the steps below: + +1. Go to the "IPAM" section under the "Net" tab. +2. Click on the "+ Add" button located at the top-right corner. +3. Enter a unique "Prefix" for the new subnet, such as ``10.254.254.0/24``. +4. Type a descriptive "Name" for the subnet. +5. Select the desired tenant name from the "Tenant" dropdown menu. +6. From the "Type" dropdown menu, select "Subnet". +7. Select "management" from the "Purpose" dropdown menu. +8. Choose the appropriate site from the "Sites" dropdown menu. +9. Click on the "Add" button to create the subnet. + + +.. image:: images/gcp-netris-ipam-mgmt.png + :align: center + +Following the creation of the private subnet for Softgate management, it is necessary to create another subnet with the "Purpose" of loopback. The "Prefix" for this subnet will be the External IP of the GCP VM instance with a netmask length of /32. For instance, if the VM instance's External IP address is 34.85.167.128, then the "Prefix" for the loopback subnet will be 34.85.167.128/32. However, unlike the subnet for Softgate management, an allocation for that "Prefix" must first be created before creating the subnet for loopback. + +.. image:: images/gcp-netris-ipam-lo.png + :align: center + +Create the Softgate in the Netris Controller +-------------------------------------------- + +After completing the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide: + +1. Ensure that you have completed all the pre-required steps. +2. Navigate to the "Net" tab in the Netris Controller and select the "Inventory" section. +3. Click on the "+Add" button to create a new Softgate. +4. Provide a descriptive name for the Softgate in the "Name" field. +5. From the "Tenant" dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets. +6. From the "Type" dropdown menu, select "SoftGate". +7. Choose the appropriate site from the "Site" dropdown menu. +8. For the "Main IP address" and "Management IP Address" fields, select "Assign automatically". +9. In the "Description" field, add ``int=ens4`` to specify that Netris should use the softgate's ens4 interface instead of the default bond0 interface that Netris Softgate Agent looks for. +10. Finally, click on the "Add" button to create the Softgate. + +.. image:: images/gcp-netris-create-sg.png + :align: center + + +Add GCP VPC Subnet(s) into the Netris Controller +------------------------------------------------ + +The next step involves registering into Netris IPAM the subnet of the region where the SoftGate node has been created. Follow the steps below to accomplish this: + +1. Access the GCP Console and navigate to your VPC Subnets. +2. Take note of the Internal IP range associated with the relevant region. +3. In Netris Controller, go to the "IPAM" section under the "Net" tab. +4. Click the "+ Add" button located at the top-right corner. +5. Enter the region's Internal IP range into the "Prefix" field for the new subnet. For example, if your Internal IP range is "10.150.0.0/20", enter that value. +6. Type a descriptive name for the subnet. +7. From the "Tenant" dropdown menu, select the desired tenant name. +8. From the "Type" dropdown menu, select "Subnet". +9. Select "inactive" from the "Purpose" dropdown menu. +10. Choose the appropriate site from the "Sites" dropdown menu. +11. Click the "Add" button. + + +Once the subnet of the region where the SoftGate node was created has been successfully registered in Netris IPAM, you can proceed to create subnets for other necessary regions in a similar manner. + +.. image:: images/gcp-vpc-subnet-to-netris.png + :align: center + + +Static route for GCP VPC Subnet(s) in Netris Controller +------------------------------------------------------- + +We need to create route entries in Netris. The prefix for the route will be the GCP VPC Subnet(s), and the next-hop will be the default gateway of Netris Softgate VM Instance. + +Here are the steps to create the static route: + +1. Securely log in to the Netris Softgate's VM instance using SSH. +2. Retrieve the default gateway address by typing the command ``ip route show default``. +3. In Netris Controller, go to the "Routes" section under the "Net" tab. +4. Click on the "+ Add" button located at the top-right corner of the screen. +5. Enter the subnet of the GCP region in the "Prefix" field. +6. Enter the default gateway of the Netris Softgate VM instance in the "Next-Hop" field. +7. Choose the appropriate site from the "Site" dropdown menu. +8. From the "Apply to" dropdown menu, select the SoftGate. +9. Finally, click on the "Add" button to create the static route. + +Repeat these steps for each GCP VPC Subnet that has been registered in Netris IPAM. + +*Note: Regardless of the GCP region's subnet, the "Next-Hop" field should always contain the default gateway of the Netris SoftGate VM instance.* + +.. image:: images/gcp-netris-static-route.png + :align: center + + +Netris SoftGate node provisioning +--------------------------------- + +After creating a softgate resource in Netris Controller and defining all necessary routes, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting "Install Agent". Copy the one-line installer command to your clipboard and connect to your VM instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it's done, reboot the server. + +.. image:: images/gcp-netris-provision-sg.png + :align: center + + +Configure routing in GCP VPC +============================ + +To ensure that specific traffic is directed to the Netris Softgate VM instance within your GCP VPC, it is necessary to configure new route entries in VPC routes. Follow the step-by-step guide below to add GCP VPC routes: + +1. Access the Google Cloud Platform (GCP) Console. +2. Navigate to the VPC Network page. +3. Click on the "Routes" then switch to the "ROUTE MANAGEMENT" tab to view the existing routes. +4. Click the "+ Create Route" button to create a new route. +5. Provide the following details for the new route: + + * Name: Assign a descriptive name to the route. + * Network: Select the appropriate network for the route. + * Destination IP Range: Specify the subnets of other Netris sites that you wish to access from this VPC. + * Next hop: From the dropdown menu, select the "Specify an instance" + * Next hop instance: From the dropdown menu, select the SoftGate VM Instance + +6. Review the route configuration and ensure all the details are accurate. +7. Click the "Create" button to add the route to the GCP VPC network. +8. Verify that the new route appears in the list of routes for the selected VPC network. + +Repeat these steps for any additional routes you need to add. + + +.. image:: images/gcp-vpc-routes-created.png + :align: center + + +Enabling Site Mesh +================== + +To enable Site Mesh, in Netris Controller, navigate to the "Net" tab and select "Sites". Next, click on the three vertical dots (⋮) on the right side of the GCP site and select "Edit", and then from the "Site Mesh" dropdown menu, select "Hub". Save the changes. Repeat these steps for all sites that need to have meshed. + +.. image:: images/gcp-netris-enable-site-mesh.png + :align: center + +The Site Mesh status can be viewed by navigating to the "Site Mesh" section under the "Net" tab. This will display the current status of Site Mesh for all Sites. + +.. image:: images/gcp-netris-site-mesh-status.png + :align: center diff --git a/en/3.4/_sources/tutorials/getting-started-for-equinix-metal.rst.txt b/en/3.4/_sources/tutorials/getting-started-for-equinix-metal.rst.txt new file mode 100644 index 0000000000..14d5f88ed9 --- /dev/null +++ b/en/3.4/_sources/tutorials/getting-started-for-equinix-metal.rst.txt @@ -0,0 +1,86 @@ +.. meta:: + :description: Getting Started for Equinix Metal + +######################################## +Equinix Metal API integration enablement +######################################## + + +For each Equinix Metal Project+location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "Equinix Metal" from the dropdown menu. + * - Name + - Type a descriptive name for your Equinix Metal Project+location. + * - Equinix Project ID + - Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID. + * - Equinix Project API key + - Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here. + * - Equinix Location + - Select your equinix location from the dropdown menu. + + +Equinix Metal Project ID + +.. image:: /tutorials/images/equinix-metal-project-id.png + :align: center + + +Equinix Metal Project API key + +.. image:: /tutorials/images/equinix-metal-project-api-keys.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/netris-create-equinix-metal-site.png + :align: center + + +Adding Netris SoftGate nodes +============================ + +For SoftGate nodes you can start with two servers of the smallest flavor. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one. + +Request two servers from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned. + +1) At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory + +.. image:: /tutorials/images/softgate-nodes-created-in-equinix.png + :align: center + +2) When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”. + + Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location). + +Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + +.. image:: /tutorials/images/softgate-nodes-recognized-in-netris.png + :align: center + +3) Provision SoftGate nodes. + +Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there. + +.. image:: /tutorials/images/softgate-one-liner-provisioning.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/softgate-green.png + :align: center diff --git a/en/3.4/_sources/tutorials/index-equinix.rst.txt b/en/3.4/_sources/tutorials/index-equinix.rst.txt new file mode 100644 index 0000000000..cda62c2b18 --- /dev/null +++ b/en/3.4/_sources/tutorials/index-equinix.rst.txt @@ -0,0 +1,26 @@ +================================================== +Netris VPC for Equinix Metal Getting Started Guide +================================================== + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + +This `video `_ walks through Netris VPC setup steps described below. + + +.. toctree:: + :maxdepth: 2 + :caption: Step-by-step Netris Setup Guide for Equinix Metal + + install-netris-controller-in-equinix-metal + equinix-metal-api-integration-enablement + adding-netris-softgate-nodes-in-equinix-metal + activating-bgp-on-equinix-metal-project + enable-services-on-equinix-metal-project + adding-netris-softgate-nodes-in-equinix-metal + +.. toctree:: + :maxdepth: 1 + :caption: Examples of Using Netris VPC on Equinix Metal Project + + using-vnet-in-equinix-metal-project + using-l4-load-balancer diff --git a/en/3.4/_sources/tutorials/index-vpc.rst.txt b/en/3.4/_sources/tutorials/index-vpc.rst.txt new file mode 100644 index 0000000000..6a2498839f --- /dev/null +++ b/en/3.4/_sources/tutorials/index-vpc.rst.txt @@ -0,0 +1,13 @@ +====================================== +VPC for Anywhere Getting Started Guide +====================================== + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + + +.. toctree:: + :maxdepth: 2 + + vpc-anywhere-concept + vpc-anywhere-controller-installation + vpc-anywhere-softgate-provisioning diff --git a/en/3.4/_sources/tutorials/index.rst.txt b/en/3.4/_sources/tutorials/index.rst.txt new file mode 100644 index 0000000000..46c6f6031b --- /dev/null +++ b/en/3.4/_sources/tutorials/index.rst.txt @@ -0,0 +1,12 @@ +======================== +Netris Generic Tutorials +======================== + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + + +.. toctree:: + :maxdepth: 2 + + installing-netris-controller + upgrading-netris diff --git a/en/3.4/_sources/tutorials/install-netris-controller-in-equinix-metal.rst.txt b/en/3.4/_sources/tutorials/install-netris-controller-in-equinix-metal.rst.txt new file mode 100644 index 0000000000..b0503c08df --- /dev/null +++ b/en/3.4/_sources/tutorials/install-netris-controller-in-equinix-metal.rst.txt @@ -0,0 +1,75 @@ +.. meta:: + :description: Installing a Netris Controller in Equinix Metal + +================================================================ +Installing a Netris Controller on Equinix Metal on-demand server +================================================================ + +Installation Steps +------------------ + +You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions). + +**Request a server** + +The smallest flavor, c3.small.x86, is enough for most users. + +.. image:: images/equinix-request-c3-small-server.png + :align: center + +if you decide to use a VM, please see below the minimal VM requirements. + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**DNS record** + +In my example my host got a public IP address 139.178.89.255. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is example using Cloudflare DNS service. (same idea with any DNS software or service) + +.. image:: images/dns-cloudflare-equinix-ip.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 139.178.89.255 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + + +Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center + diff --git a/en/3.4/_sources/tutorials/installing-netris-controller.rst.txt b/en/3.4/_sources/tutorials/installing-netris-controller.rst.txt new file mode 100644 index 0000000000..6cb57398b9 --- /dev/null +++ b/en/3.4/_sources/tutorials/installing-netris-controller.rst.txt @@ -0,0 +1,85 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Installing a Netris Controller +============================== + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there's no need for an individual controller for each deployment. + +It doesn't matter where you host the Netris controller. What matters is that 1) the Netris controller needs to be accessible over the Internet. 2) You can access the web console. 3) Nodes that are going to be managed by Netris have access to the Netris controller through their management network interface. + +**Linux host requirements** + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**DNS record** + +In the example below, the host has a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, it is recommended to use a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is an example using Cloudflare DNS service (may use any DNS service). + +.. image:: images/dns-record-netrisctl.png + :align: center + +Ensure that newly created domain name resolves to the right IP address of the machine that will host the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 54.183.23.201 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + The Netris Controller installer will create a K3s cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let's Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That's why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + + +Once the installation process is finished, you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center + +**Restrict incoming TCP requests to the list below:** + ++----------+--------------------------------+ +| TCP Port | Service | ++==========+================================+ +| 22 | SSH | ++----------+--------------------------------+ +| 80 | HTTP | ++----------+--------------------------------+ +| 443 | Netris Web Console | ++----------+--------------------------------+ +| 2003 | Streaming Telemetry (Collectd) | ++----------+--------------------------------+ +| 3033 | Netris Monitoring (Telescope) | ++----------+--------------------------------+ +| 50051 | Netris Agent (gRPC) | ++----------+--------------------------------+ + diff --git a/en/3.4/_sources/tutorials/netris-vpc-for-aws.rst.txt b/en/3.4/_sources/tutorials/netris-vpc-for-aws.rst.txt new file mode 100644 index 0000000000..97b963522f --- /dev/null +++ b/en/3.4/_sources/tutorials/netris-vpc-for-aws.rst.txt @@ -0,0 +1,15 @@ +=========================== +Enabling Site Mesh with AWS +=========================== + + +.. toctree:: + :maxdepth: 2 + + aws-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + aws-deploy-softgate diff --git a/en/3.4/_sources/tutorials/netris-vpc-for-equinix-metal.rst.txt b/en/3.4/_sources/tutorials/netris-vpc-for-equinix-metal.rst.txt new file mode 100644 index 0000000000..3370abd342 --- /dev/null +++ b/en/3.4/_sources/tutorials/netris-vpc-for-equinix-metal.rst.txt @@ -0,0 +1,21 @@ +============================ +Netris VPC for Equinix Metal +============================ + + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + install-netris-controller-in-equinix-metal + equinix-metal-api-integration-enablement + adding-netris-softgate-nodes-in-equinix-metal + activating-bgp-on-equinix-metal-project + enable-services-on-equinix-metal-project + +.. toctree:: + :maxdepth: 1 + :caption: Use + + using-vnet-in-equinix-metal-project + using-l4-load-balancer diff --git a/en/3.4/_sources/tutorials/netris-vpc-for-gcp.rst.txt b/en/3.4/_sources/tutorials/netris-vpc-for-gcp.rst.txt new file mode 100644 index 0000000000..53cc4f6e06 --- /dev/null +++ b/en/3.4/_sources/tutorials/netris-vpc-for-gcp.rst.txt @@ -0,0 +1,15 @@ +=========================== +Enabling Site Mesh with GCP +=========================== + + +.. toctree:: + :maxdepth: 2 + + gcp-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + gcp-deploy-softgate diff --git a/en/3.4/_sources/tutorials/netris-vpc-for-phoenixnap-bmc.rst.txt b/en/3.4/_sources/tutorials/netris-vpc-for-phoenixnap-bmc.rst.txt new file mode 100644 index 0000000000..29e2f495d3 --- /dev/null +++ b/en/3.4/_sources/tutorials/netris-vpc-for-phoenixnap-bmc.rst.txt @@ -0,0 +1,24 @@ +============================= +Netris VPC for phoenixNAP BMC +============================= + + +.. toctree:: + :maxdepth: 2 + + phoenixnap-bmc-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + phoenixnap-bmc-link-to-installation + phoenixnap-bmc-ipam-setup + +.. toctree:: + :maxdepth: 2 + :caption: Use + + phoenixnap-bmc-using-nat + phoenixnap-bmc-using-vnet + phoenixnap-bmc-using-l4lb diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst.txt new file mode 100644 index 0000000000..e13e2abe1a --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst.txt @@ -0,0 +1,37 @@ +.. meta:: + :description: Provisioning Netris SoftGate nodes in phoenixNAP BMC + +.. _phxnap_sgs: + +#################################################### +Provisioning Netris SoftGate nodes in phoenixNAP BMC +#################################################### + +For SoftGate nodes you can start with two ``s2.c1.small`` or larger servers. In the future, if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration), you can upgrade the servers one-by-one. + +1) Request two servers (s2.c1.small) from phoenixNAP BMC with Ubuntu Jammy OS, type ``netris-softgate`` in the "Server Description" field, +choose a /31 Public IP Allocation and wait until provisioned. + +.. note:: + + It’s required to type the ``netris-softgate`` in the description. This signals Netris Controller that those are not regular bare-metal servers, and they should be synced with the type of SoftGate. + +.. image:: /tutorials/images/phoenixnap-softgate-nodes-creation.png + :align: center + +2) At this point you should see Netris Controller listing newly created servers as “Softgate Softgate1(2)” under Netris Web Console → Net → Inventory. +You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + + +3) Provision SoftGate nodes. Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server with the ``ubuntu`` user and paste the one-liner there. + +.. image:: /tutorials/images/phoenixnap-softgate-nodes-created.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/phoenixnap-softgate-nodes-green.png + :align: center diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-api-integration-enablement.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-api-integration-enablement.rst.txt new file mode 100644 index 0000000000..26c9b7ecec --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-api-integration-enablement.rst.txt @@ -0,0 +1,47 @@ +.. meta:: + :description: Enable phoenixNAP BMC API integration + +.. _phxnap_api: + +##################################### +Enable phoenixNAP BMC API integration +##################################### + + +For each phoenixNAP BMC location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "PhoenixNAP BMC" from the dropdown menu. + * - Name + - Type a descriptive name for your phoenixNAP BMC location. + * - PhoenixNAP Client ID + - Create a new API Credential with "bmc" scope in phoenixNAP BMC portal under API Credentials → + Create Credentials. Then copy/paste Client ID. + * - PhoenixNAP Client Secret + - Copy/Paste the Client Secret from the already created API Credential. + * - PhoenixNAP Location + - Select your phoenixNAP BMC location from the dropdown menu. + + +phoenixNAP BMC API Credential + +.. image:: /tutorials/images/phoenixnap-api-credential.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/phoenixnap-site-create.png + :align: center + + diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-concept.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-concept.rst.txt new file mode 100644 index 0000000000..1bdf8cc43d --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-concept.rst.txt @@ -0,0 +1,40 @@ +###################################### +Netris VPC for phoenixNAP BMC Overview +###################################### + +Introduction +------------- + +Netris VPC for phoenixNAP BMC is a solution that enables VPC network functionality in phoenixNAP's Bare Metal Cloud. Netris VPC can be used for all or part of the servers/network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures. + + +Concept +-------- + +There is **an agent** inside the Netris VPC controller for phoenixNAP BMC that communicates to phoenixNAP API once the `integration has been enabled `_. That agent is responsible for metadata synchronization between phoenixNAP BMC and Netris VPC controller. Also, it configures the BMC network services based on the services defined in the Netris VPC controller. + +**SoftGate** is a highly optimized automatic Linux gateway, which in turn communicates with the Netris VPC controller via an encrypted protocol. A SoftGate node (typically two of them for redundancy) is a regular BMC server that should be deployed on desired phoenixNAP location. Once the server has been `deployed `_, it starts consuming the BMC public and private networks with various VLANS. + +.. image:: images/phoenixnap-concept-solution-traffic-flows.png + :align: center + +Due to phoenixNAP BMC creating a new VLAN for each server's Public IP Allocation and Public Network/Private Network, we decided to use the upper VLANS Range - 3000-4094 (if you need to change the default range, that can be done in the Netris Site settings). Netris will consume VLANS only from that range. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the BMC network. + +SoftGate node becomes the default gateway for the workloads consuming the VPC network. PhoenixNAP BMC Private Network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers. Netris also creates a Public Network in the phoenixNAP BMC when the IP Addresses from Public IP Allocations (passed to Netris VPC) :ref:`have been used` in any Netris VPC's services. Besides Public/Private Networks creation Netris also attaches all necessary servers to that networks. Therefore, you don’t need to worry about assigning servers to the networks on the phoenixNAP side, Netris will handle that automatically when that makes sense. + +.. image:: images/phoenixnap-concept-public-network.png + :align: center + + + +**What's next** + +* :doc:`Installing Netris on phoenixNAP BMC ` +* :doc:`IPAM Setup for Services ` + + +Usage + +* :doc:`Using NAT services in phoenixNAP BMC ` +* :doc:`Using V-Net (isolated virtual network) services in phoenixNAP BMC ` +* :doc:`Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC ` diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-install-netris-controller.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-install-netris-controller.rst.txt new file mode 100644 index 0000000000..e6bd4faf1f --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-install-netris-controller.rst.txt @@ -0,0 +1,74 @@ +.. meta:: + :description: Installing a Netris Controller in phoenixNAP BMC + +================================================================= +Installing a Netris Controller on phoenixNAP BMC server +================================================================= + +Installation Steps +------------------ + +You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions). + +**Minimum Hardware Requirements** + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**Deploy New Server** + +Deploy a new server in phoenixNAP BMC: select the Location, choose the server flavor (the smallest flavor, s0.d1.small, is enough for most users), type some name for Controller Server, request a new /31 public allocation for this instance. + +.. image:: images/phoenixnap-request-ctl-server.png + :align: center + +**DNS record** + +In my example my host got a public IP address 131.153.154.61. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is example using Cloudflare DNS service. (same idea with any DNS software or service) + +.. image:: images/phoenixnap-dns-cloudflare.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 131.153.154.61 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netrisctl.netris.dev --ctl-ssl-issuer letsencrypt + +.. note:: + Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + + +Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-ipam-setup.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-ipam-setup.rst.txt new file mode 100644 index 0000000000..b30014d49d --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-ipam-setup.rst.txt @@ -0,0 +1,116 @@ +.. meta:: + :description: IPAM Setup for Services + +.. _phxnap_services: + +####################### +IPAM Setup for Services +####################### + +Although bare metal servers in phoenixNAP BMC may get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization, something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities. + +Both NAT and on-demand Load Balancer services need public IP addresses. + +Go to phoenixNAP BMC web console and click on Networking → Public IP Allocations (see the screenshot below) + +Option 1 - Separated allocation for each purpose +================================================ + +In this example, I’m requesting two /29 (5 assignable IPs) IP Allocations, one for NAT and one for Load Balancer. + +It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC. + +You can always request more IP address allocations in the future. + +.. image:: /tutorials/images/phoenixnap-request-ip-allocation.png + :align: center + +Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM + +.. image:: /tutorials/images/phoenixnap-netris-ipam-synced.png + :align: center + +You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service). + + +1. Enable on-demand (elastic) Load Balancer +------------------------------------------- + +To Enable an on-demand (elastic) Load Balancer, you only need to change the “purpose” field of appropriate IP subnet from “inactive” into “load-balancer” + +Click on the 3 dots menu (in this example of first /29 subnet), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-lb-purpose.png + :align: center + +Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform. + +Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it. + + +2. Enable NAT +------------- + +To enable NAT, you need to repurpose a subnet for NAT. In the below example, I’m repurposing the second of the newly requested /29 subnets for NAT. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-nat-purpose.png + :align: center + +Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. + + +Option 2 - Splitting single allocation into different subnets +============================================================= + +In this example, I’m requesting a single /28 (13 assignable IPs) Public IP Allocation, then splitting it into two /29 subnets, one for NAT and one for Load Balancer. + +It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC. + +You can always request more IP address allocations in the future. + +.. image:: /tutorials/images/phoenixnap-request-ip-allocation-slash-28.png + :align: center + +Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM + +.. image:: /tutorials/images/phoenixnap-netris-ipam-synced-slash-28.png + :align: center + +You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service). + + +1. Enable on-demand (elastic) Load Balancer +------------------------------------------- + +In this scenario, to Enable an on-demand (elastic) Load Balancer, you need to create a smaller subnet with the purpose of “load-balancer”. + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-lb-purpose-slash-28.png + :align: center + +Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform. + +Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it. + + +2. Enable NAT +------------- + +To enable NAT, you need to create a smaller subnet with the purpose of “nat”. + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-nat-purpose-slash-28.png + :align: center + +Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. + +Note* +===== + +It doesn't matter with what Option (1,2) you will go. Once the Public IP Allocation has been replicated in the Netris IPAM, Netris will automatically reserve the network, first usable, and broadcast IP addresses because they are unusable in this (phoenixNAP BMC) scenario. + +.. image:: /tutorials/images/phoenixnap-reserved-ips.png + :align: center + diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-link-to-installation.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-link-to-installation.rst.txt new file mode 100644 index 0000000000..8be20c8905 --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-link-to-installation.rst.txt @@ -0,0 +1,12 @@ +.. meta:: + :description: Installing Netris on phoenixNAP BMC + +.. _pnap_doc_link: + +################################### +Installing Netris on phoenixNAP BMC +################################### + +To deploy Netris on the phoenixNAP Bare Metal Cloud, please follow the official guide provided by phoenixNAP. This guide will walk you through the step-by-step installation process. + +`Netris for Bare Metal Cloud `_ diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-using-l4lb.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-using-l4lb.rst.txt new file mode 100644 index 0000000000..466bcee6c8 --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-using-l4lb.rst.txt @@ -0,0 +1,18 @@ +.. meta:: + :description: Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC + +.. _phxnap_l4lb: + +#################################################################### +Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC +#################################################################### + +Services --> L4 Load Balancer is an on-demand (elastic) load balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console. + +.. image:: /tutorials/images/phoenixnap-l4lb.png + :align: center + diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-using-nat.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-using-nat.rst.txt new file mode 100644 index 0000000000..735aa2c992 --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-using-nat.rst.txt @@ -0,0 +1,42 @@ +.. meta:: + :description: Using NAT services in phoenixNAP BMC + +.. _phxnap_nat: + +#################################### +Using NAT services in phoenixNAP BMC +#################################### + +NAT, under Net --> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 10.0.0.0/8 to have access to services outside their VPC. The goal is to provide bare-metal servers/VMs with the capability to connect to the Internet through NAT for outbound access. + +Option 1 - MASQUERADE +===================== + +The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn't require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT). + +.. image:: images/phoenixnap-nat-masquerade.png + :align: center + + +Option 2 - SNAT +=============== + +Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a :ref:`NAT purpose`. SNAT replaces the source IP address of the instances with the IP address of the "SNAT to IP". Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity. + +.. image:: images/phoenixnap-nat-snat.png + :align: center + + +You can always have more granular control either through NAT rule or using Services → ACLs. + + +DNAT +==== + +A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a "DNAT to IP/Port". This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database. + +I'm creating a DNAT rule for the ssh port in the example below. It forwards the public IP's 55022 port to the local IP's 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server. + +.. image:: images/phoenixnap-nat-dnat.png + :align: center + diff --git a/en/3.4/_sources/tutorials/phoenixnap-bmc-using-vnet.rst.txt b/en/3.4/_sources/tutorials/phoenixnap-bmc-using-vnet.rst.txt new file mode 100644 index 0000000000..1526b45f43 --- /dev/null +++ b/en/3.4/_sources/tutorials/phoenixnap-bmc-using-vnet.rst.txt @@ -0,0 +1,93 @@ +.. meta:: + :description: Using V-Net (isolated virtual network) services in phoenixNAP BMC + +.. _phxnap_vnet: + +################################################################## +Using V-Net (isolated virtual network) services in phoenixNAP BMC +################################################################## + +V-Net, under Services --> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from phoenixNAP BMC into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network. + +Netris V-Net (in phoenixNAP BMC scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Private Network using phoenixNAP BMC API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Private Network. SoftGate nodes are the default gateway for the V-Net services. + +Adding Subnets for V-Net +======================== + +Before starting to use the V-NETs, you need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources. + +.. image:: /tutorials/images/phoenixnap-netris-create-common-subnets.png + :align: center + +Creating a V-Net +================ + +You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, phoenixNAP BMC, and your Compute. + +.. image:: /tutorials/images/phoenixnap-netris-creating-vnet.png + :align: center + +In this example, the new V-NET has VLAN ID 3000, subnet 10.128.1.0/24, and gateway 10.128.1.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 3000, and they should use IP addresses from 10.128.1.2-254 pointing to 10.128.1.1 as the default gateway or use DHCP. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 3000 will have Internet access over the NAT. + +.. image:: /tutorials/images/phoenixnap-netris-vnet-ready.png + :align: center + + +Deploy a new server into an existing V-Net +------------------------------------------ + +Netris creates a private network in phoenixNAP BMC based on declared V-Nets. Besides creation, Netris continuously monitors that private networks. As a result of this continuous monitoring, you can't edit private networks created by Netris from the phoenixNAP BMC console. However, if any modifications are made, Netris will automatically roll everything back to its state. + +The only exception is a case with newly deployed servers. When you request a server from phoenixNAP BMC without any public IP address, use some Netris V-Net as a private network and mark the DHCP checkbox "**Use your own privately managed DHCP server (Obtain IP address automatically)**" - Netris will automatically add that server into the V-Net. + +.. image:: /tutorials/images/phoenixnap-vnet-import-a-new-server.png + :align: center + +Thanks to that functionality, you can request a bare-metal server directly into Netris V-Net. As a result, you will have bare-metal servers in a private subnet that can connect to services outside your VPC (since we have enabled NAT globally in previous chapters), but external services cannot initiate a connection with those servers. Once the servers have been provisioned, they will get a private IP from Netris DHCP, and you can find those IPs by pressing the "**IP/MAC Info**" button on the V-NET. + +.. image:: /tutorials/images/phoenixnap-vnet-imported-new-server.png + :align: center + +In order to connect via SSH to the newly deployed server, you can either create a DNAT rule and connect via Public IP, or if you don't need permanent ssh access to that server, you can simply connect using Softgate as a JumpHost. + +.. code-block:: shell-session + + ssh -o ProxyCommand="ssh -W %h:%p ubuntu@" ubuntu@ + + +.. image:: /tutorials/images/phoenixnap-vnet-ssh-to-server.png + :align: center + + +Tags +---- + +Tags are used to associate bare-metal servers with V-NET dynamically. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers from the Private Network based on that tag. Thus, you can make flexible V-NETs, and there is no need to include every new server in the V-NET. + +.. image:: /tutorials/images/phoenixnap-vnet-with-tag.png + :align: center + +This feature is even more efficient when you build your infrastructure via Terraform. For example, let's say you've created a V-NET with a tag using Netris Terraform Provider, then order several servers with the same tag using phoenixNAP Terraform Provider. And that's it, when the servers are ready, Netris will detect them and make them part of the V-NET. + +.. image:: /tutorials/images/phoenixnap-vnet-with-tag-terraform.png + :align: center + + +Unmanaged +--------- + +Another option is turning the existing private network into Netris V-Net. All private networks from the allowed VLAN IDs range and in the proper location that Netris has not created are visible as "unmanaged" in the V-Net section. + +.. image:: /tutorials/images/phoenixnap-vnet-unmanaged-vnet.png + :align: center + +The "manage" button will open a dialogue window where it's also possible to add a default gateway for the appropriate VLAN. + + +.. warning:: + Once the private network is being converted into V-Net, it will be managed by Netris and no longer manageable through phoenixNAP BMC console. + +.. image:: /tutorials/images/phoenixnap-vnet-managed-vnet.png + :align: center + +Note that you can use Services --> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites) diff --git a/en/3.4/_sources/tutorials/upgrading-netris.rst.txt b/en/3.4/_sources/tutorials/upgrading-netris.rst.txt new file mode 100644 index 0000000000..417dc3d5ba --- /dev/null +++ b/en/3.4/_sources/tutorials/upgrading-netris.rst.txt @@ -0,0 +1,199 @@ +.. meta:: + :description: Upgrading Netris + +.. raw:: html + + + + +.. role:: green + +.. role:: red + +************************************** +Netris Upgrade and Rollback Procedures +************************************** + +Upgrade Procedure +================= + +Due to potential database structural changes between Netris versions, it's highly recommended to take a backup of the database before upgrading. The backup will be used in the unlikely event of the need to perform a rollback. + +1. To create a database backup, run the following command by first SSHing the Controller: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot.sql + +Ensure that SQL file ``db-snapshot.sql`` is generated and present in the current directory. + +.. note:: + + An SQL dump is enough for this scenario, but that's not Netris-Controller's entire backup procedure. Get familiar with the Backup/Restore documentation :ref:`here`. + +2. Stop all Netris agents on devices managed by the controller (switch & SoftGate). + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sg + +Ensure that all devices in the *Net → Inventory* section are ":red:`red`" with the "**check_agent**" status being "**Agent is unavailable**". + +.. note:: + + A stopped Netris agent has no impact on production traffic through the device. + +.. _upgrade 3: + +3. Before upgrading the Netris Controller, take a note of the "*Netris Version*" by navigating to *Setting → General* in the Controller web interface. This version number will be used in case of the need to perform a rollback procedure. + +.. image:: /tutorials/images/netris_version_example.png + :align: center + :alt: Netris Version Example + +4. Start the upgrade of the Netris Controller using the one-liner after SSHing to the Controller. + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +.. note:: + + This process can take up to 5 minutes + + +Afterwards, make sure that all pods have either "*Running*" or "*Completed*" status by executing the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller get pods + + +The output is similar to this: + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + svclb-netris-controller-haproxy-6tkgj 4/4 Running 0 38d + netris-controller-haproxy-bcb944b7c-qcbf8 1/1 Running 0 13d + netris-controller-squid-7f6fdc6cf9-7fdx8 1/1 Running 0 38d + svclb-netris-controller-squid-58rnp 1/1 Running 0 38d + netris-controller-graphite-0 1/1 Running 0 38d + netris-controller-mongodb-0 1/1 Running 0 38d + netris-controller-redis-master-0 1/1 Running 0 38d + netris-controller-smtp-76778cf85f-lw5v5 1/1 Running 0 10d + netris-controller-mariadb-0 1/1 Running 0 10d + netris-controller-web-session-generator-8b9dbbcd8-8snhd 1/1 Running 0 10d + netris-controller-telescope-notifier-647975848f-fs5dn 1/1 Running 0 10d + netris-controller-app-b9b8d8f8d-4ssqb 1/1 Running 0 10d + netris-controller-grpc-987669fb9-jjskp 1/1 Running 0 10d + netris-controller-telescope-777c98c5d9-mqwl6 1/1 Running 0 10d + helm-install-netris-controller-lqmq7 0/1 Completed 0 20h + + +.. warning:: + + If, after 5 minutes, you see pods with a status other than "*Running*" or "*Completed*", please reach out to us via `Slack `__. + +Then verify that the "*Netris Version*" reflects the version change by navigating to *Setting → General* in the Controller web interface. + +5. Once you have verified that the Netris controller is up-to-date, it is time to update the switch and SoftGate agents. + +Upgrade the switch & SoftGate agents by copying the one-liner from the "*Install Agent*" option of the device's 3-dot menu found under the *Net → Inventory* section and pasting it after SSHing to the corresponding device. + +.. image:: /tutorials/images/install_agent.gif + :align: center + :alt: Install Agent + +After all the agents have finished the upgrade process, make sure all devices in the *Net → Inventory* section have a ":green:`green`" status and the *Netris version* for each device reflects the version change. + +In the event the "**check_agent**" status is "**Agent is unavailable**" after the agent upgrade has finished, perform agent restart on the affected device(s). + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sg + +Rollback Procedure +================== + +A rollback procedure can be executed in the event the upgrade introduced any adverse impact on the production traffic. + +1. Stop all Netris agents on the devices managed by the controller (switch & SoftGate). + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sg + +2. Restore the database from the previously taken snapshot. + +Drop the current database and create a new one by running the following command after SSHing to the Controller: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"' + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"' + +While still connected to the Controller, copy the backup file from the controller host system to the MariaDB container and restore the database: + +.. code-block:: shell-session + + kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql' + +3. Downgrade Netris Controller application with the following command. + +.. note:: + + For the version number, use the number collected from :ref:`step #3` during the upgrade procedure. + +Example: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-version 3.0.10-031 + +Afterwards, verify that the version of the "*Netris Version*" reflects the downgraded version by navigating to *Setting → General* in the Netris Controller. + +4. Once you have verified that the Netris controller has been downgraded to the correct version, it is time to downgrade the switch and SoftGate agents. + +Install the correct and appropriate versions of the switch & SoftGate agents simply by copying the one-liner from the "*Install Agent*" option of the device's 3-dot menu found under the *Net → Inventory* section and pasting it after SSHing to the corresponding device. + +After all the switches and SoftGates have been successfully downgraded, make sure all the devices in the *Net → Inventory* section have a ":green:`green`" status and the *Netris version* for each device reflects the version downgrade. + +In case the "**check_agent**" status is "**Agent is unavailable**" after agent downgrade, perform agent restart. + +For the switch agent, first SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sw + +For the SoftGate agent, first SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sg diff --git a/en/3.4/_sources/tutorials/upgrading-sonic-os.rst.txt b/en/3.4/_sources/tutorials/upgrading-sonic-os.rst.txt new file mode 100644 index 0000000000..d9e1c4d42d --- /dev/null +++ b/en/3.4/_sources/tutorials/upgrading-sonic-os.rst.txt @@ -0,0 +1,63 @@ +.. meta:: + :description: Upgrading SONiC OS + +.. raw:: html + + + + +.. role:: green + +.. role:: red + +************************************** +Upgrading SONiC Operating System +************************************** + +Upgrade Procedure +================= + +The SONiC OS image is being distributed by the Vendor of the switch, you need to obtain one via the support page of the vendor. +The image should be placed on a web server to which switch(es) have access. +In this example we will use a server with an IP address of *10.0.0.36*, the image is located in the */var/www/html/* folder, which is the root of the web server. + +.. image:: /tutorials/images/upgrading_sonic_folder_listing.png + :align: center + +.. hint:: To make sure that the image is available as expected, please try to download the image with browser. + +The upgrade process consists of the following steps: + +1. Login to the switch and execute + +.. code-block:: shell-session + + sudo sonic-installer install -y http://10.0.0.36/Edgecore-SONiC_20220929_052156_ec202012_420.bin + +The process will take some time, after installation complete, reboot is mandatory. + +.. warning:: + + The installation will wipe all the data on the switch, including the admin user password, the authorized keys and ssh identity. After the reboot, a new ssh identity is generated. Because of that, you will be prompted with an identity mismatch message on the first login attempt. Please use your OS-specific procedure to remove the old key. + +2. Login to the switch and verify that the OS version is updated + +.. code-block:: shell-session + + admin@switch15:~$ show version + + SONiC Software Version: SONiC.Edgecore-SONiC_20220929_052156_ec202012_420 + Distribution: Debian 10.13 + Kernel: 4.19.0-12-2-amd64 + Build commit: 895d178f6 + Build date: Thu Sep 29 05:49:16 UTC 2022 + Built by: ubuntu@ip-10-5-1-225 + + Platform: x86_64-accton_as7326_56x-r0 + HwSKU: Accton-AS7326-56X + ASIC: broadcom + ASIC Count: 1 + Serial Number: REDACTED + Uptime: 00:06:00, 1 user, load average: 1.39, 2.08, 2.29 + +3. Perform netris agent installation steps 2 to 4 from :ref:`"Install the Netris Agent"` tutorial. diff --git a/en/3.4/_sources/tutorials/using-l4-load-balancer.rst.txt b/en/3.4/_sources/tutorials/using-l4-load-balancer.rst.txt new file mode 100644 index 0000000000..40b7477c34 --- /dev/null +++ b/en/3.4/_sources/tutorials/using-l4-load-balancer.rst.txt @@ -0,0 +1,13 @@ +################################################## +Using on-demand (elastic) L4 Load Balancer service +################################################## + +Services --> L4 Load Balancer is an on-demand (elastic) server load balancer. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console. + +.. image:: /tutorials/images/netris-l4-load-balancer.png + :align: center + diff --git a/en/3.4/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt b/en/3.4/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..1a7e4fbb21 --- /dev/null +++ b/en/3.4/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt @@ -0,0 +1,51 @@ +######################################################################## +Using V-Net (isolated virtual network) services in Equinix Metal Project +######################################################################## + +V-Net, under Services --> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from Equinix Metal Project into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network. + +Netris V-Net (in Equinix Metal scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Layer-2 network using Equinix Metal API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Layer-2 network. SoftGate nodes are the default gateway for the V-Net services. + +You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, Equinix, and your Compute. + +.. image:: /tutorials/images/netris-creating-vnet-for-equinix-metal.png + :align: center + +In this example, the new V-NET has VLAN ID 2, subnet 10.0.0.0/24, and gateway 10.0.0.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 2, and they should use IP addresses from 10.0.0.2-254 pointing to 10.0.0.1 as the default gateway. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 2 will have Internet access over the NAT. + +.. image:: /tutorials/images/netris-vnet-ready-in-equinix-metal.png + :align: center + +Tags +==== + +Starting from Netris `v3.3.0 `_, it is possible to dynamically associate bare-metal servers with V-NET. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers to the Layer-2 network based on that tag. Due to this, you can make flexible V-NETs, and there is no need to include every new server in the V-NET. + +.. image:: /tutorials/images/equinix-metal-vnet-with-tag.png + :align: center + +This feature is even more efficient when you build your infrastructure via Terraform. For example, let's say you've created a V-NET with a tag using Netris Terraform Provider, then order several metal servers with the same tag using Equinix Metal Terraform Provider. And that's it, when the servers are ready, Netris will detect them and make them part of the V-NET. + +.. image:: /tutorials/images/equinix-metal-vnet-with-tag-terraform.png + :align: center + +Unmanaged +========= + +Another option is turning the existing Layer-2 network (VLAN) into Netris V-Net. All VLANs in the particular project that aren't used in other services, like an E-BGP, are visible as "unmanaged" in the V-Net section. + +.. image:: /tutorials/images/unmanaged-vlan-equinix.png + :align: center +.. image:: /tutorials/images/unmanaged-vnet.png + :align: center + +The "manage" button will open a dialogue window where it's also possible to add a default gateway for the appropriate VLAN. + + +.. warning:: + Once the VLAN is being converted into V-Net, it will be managed by Netris and no longer manageable through Equinix Metal console. + +.. image:: /tutorials/images/manage-vnet.gif + :align: center + +Note that you can use Services --> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites) diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-check-default-site.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-check-default-site.rst.txt new file mode 100644 index 0000000000..9af39953a5 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-check-default-site.rst.txt @@ -0,0 +1,16 @@ +########################### +Check Default Site Settings +########################### + + +The VLAN ID range (under Net-->Site-->Default) defines what VLAN IDs Netris can use. You may want to ensure that this range does not overlap with other VLAN IDs you may have in your network. + + +.. image:: images/vpc-anywhere-check-site-default.png + :align: center + +The default value is 700-900. Feel free to change to any range your network administrators are comfortable. + +.. image:: images/vpc-anywhere-edit-vlan-range-default-site.png + :align: center + diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-concept.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-concept.rst.txt new file mode 100644 index 0000000000..ebcf0e4598 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-concept.rst.txt @@ -0,0 +1,25 @@ +######################## +VPC Anywhere Overview +######################## + +Introduction +------------- + +VPC Anywhere is a solution designed by Netris that allows the integration of VPC network functionality into any network. Netris VPC can be used for all or part of the network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures. + + +Concept +--------- + +SoftGate is a highly optimized automatic Linux gateway that communicates with Netris VPC controller through an encrypted management link. + +A SoftGate node (typically two of them for redundancy) connects to the physical network on an 802.1q trunk port (similar to your Vmware or KVM hypervisors). + +Netris needs to know what VLAN IDs it is allowed to utilize. We recommend reserving a range of VLAN IDs dedicated to Netris VPC. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the network. + +SoftGate node becomes the default gateway for the workloads consuming the VPC network. Your existing switch network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers. + + +.. image:: images/vpc-anywhere-solution-traffic-flows.png + :align: center + diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-controller-installation.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-controller-installation.rst.txt new file mode 100644 index 0000000000..9da791d671 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-controller-installation.rst.txt @@ -0,0 +1,86 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Install a Netris Controller +============================== + +Requirements and Installation steps +----------------------------------- + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact, if there are multiple Netris managed deployments there's no need for an individual controller for each deployment. + +It doesn't matter where to host the Netris controller. What does matter is that the Netris controller needs to be accessible over the Internet. 1) So you can access the web console. 2) Nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface. + +**Linux Host requirements** + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**DNS record** + +In my example, my host got a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is an example using Cloudflare DNS service. (same idea with any DNS software or service) + +.. image:: images/dns-record-netrisctl.png + :align: center + +Ensure that the newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 54.183.23.201 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let's Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That's why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + +Once the installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center + +**Restrict incoming TCP requests to the list below:** + ++----------+--------------------------------+ +| TCP Port | Service | ++==========+================================+ +| 22 | SSH | ++----------+--------------------------------+ +| 80 | HTTP | ++----------+--------------------------------+ +| 443 | Netris Web Console | ++----------+--------------------------------+ +| 2003 | Streaming Telemetry (Collectd) | ++----------+--------------------------------+ +| 3033 | Netris Monitoring (Telescope) | ++----------+--------------------------------+ +| 50051 | Netris Agent (gRPC) | ++----------+--------------------------------+ + diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-ipam-setup.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-ipam-setup.rst.txt new file mode 100644 index 0000000000..926e261992 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-ipam-setup.rst.txt @@ -0,0 +1,46 @@ +.. meta:: + :description: IPAM Setup for Services + +####################### +IPAM Setup for Services +####################### + +To take advantage of the services provided by Netris, it is imperative to define Allocations/Subnets within the Netris IPAM. The Netris Controller comes with predefined subnets that serve the purpose of "common" and these subnets can be used to create V-Nets. However, to leverage services like NAT/L4 (elastic) Load Balancer, it is essential to create new Allocations and Subnets that align with the appropriate purpose. Alternatively, if an existing predefined subnet satisfies the requirements of your network architecture, it is possible to reconfigure it with the desired purpose. + +Create an allocation +==================== + +In addition to the predefined subnets, the Netris Controller also includes predefined allocations, consisting of private IP addresses defined in RFC 1918 - ``10.0.0.0/8``, ``172.16.0.0/12``, and ``192.168.0.0/16``. If you intend to create subnets that fall outside of these predefined allocations, you must first create allocations that encompass those subnets. + + +.. image:: /tutorials/images/vpc-anywhere-ipam-allocation.png + :align: center + + +Enable on-demand L4 (elastic) Load Balancer +=========================================== + +Once the allocation is created, you can proceed with creating a subnet with the purpose of "load-balancer". + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/vpc-anywhere-ipam-l4lb-subnet.png + :align: center + +The on-demand L4 (elastic) Load Balancer service has been successfully activated and can be accessed through either the web console or Kubernetes using a service of the type load-balancer, or with Terraform. + +It is worth noting that in the example provided, the Tenant field was left at its default setting of "Admin." Tenancy is a feature that enables role-based access control and resource delegation. To illustrate, you may want to establish a user role and tenant for your colleagues who are meant to consume Netris VPC services but are not authorized to administer them. + +.. _vpc_anywhere_ipam_nat: + +Enable Network Address Translation (NAT) +======================================== + +To enable NAT, you need to create a subnet with the purpose of "nat". + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/vpc-anywhere-ipam-nat-subnet.png + :align: center + +The NAT service has been successfully activated. To create a NAT rule, navigate to the Net → NAT section of the Netris web console. Netris supports most standard SNAT and DNAT rules. diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-softgate-installation.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-softgate-installation.rst.txt new file mode 100644 index 0000000000..7179c4965e --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-softgate-installation.rst.txt @@ -0,0 +1,90 @@ +.. meta:: + :description: Netris SoftGate Installation + +.. _softgate-installation-vpc_def: + +*************************** +SoftGate Installation +*************************** + +Minimum Hardware Requirements +============================= +* 8 CPU cores +* 16 GB RAM +* 300 GB HDD + + +SoftGate software provisioning +============================== +The SoftGate deployment needs a freshly installed Ubuntu Linux 22.04 LTS. + +.. note:: + Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: :ref:`"Adding SoftGates"`. + +1) Navigate to the **Net-->Inventory** section and click the **three vertical dots (⋮)** on the right side of the SoftGate node you are provisioning. Then click **Install Agent** and copy the one-line installer command to your clipboard. + +.. image:: /images/softgate-install-agent.png + :align: center + +2) Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node) + +.. image:: /images/softgate-provisioning-cli-output.png + :align: center + +.. note:: + Please note that the Netris installation script replaces the default Netplan networking backend with regular ifupdown and attempts to migrate the configuration set during installation to /etc/network/interfaces. + +3) Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as described in /etc/network/interfaces. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The management network interface + auto ensZ + iface ensZ inet static + address + # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node, otherwise adjust the line + # according to your setup. + up ip route add via + + # Physical port on SoftGate node connected to a TRUNK port of your network + auto ens + iface ens inet static + address 0.0.0.0/0 + + # Optionally you can add more physical interfaces under your bond0, uncomment as needed + #auto ens + #iface ens inet static + # address 0.0.0.0/0 + + # Bond interface + auto bond0 + iface bond0 inet static + address 0.0.0.0/0 + # Please replace/remove the ensX/Y with actual interface name(s) below to one(s) present in the OS. + bond-slaves ens + # Optional, please adjust the bonding mode below according to the desired functionality. + bond-mode active-backup + + source /etc/network/interfaces.d/* + +.. note:: + Ensure that the Management network interface IP address is as expected so the SoftGate node will maintain IP connectivity with Netris Controller after reboot. + +4) Reboot the SoftGate + +.. code-block:: shell-session + + sudo reboot + +Once the server boots up, you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. + +.. image:: images/vpc-anywhere-softgates-green.png + :align: center diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-upstream-peering.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-upstream-peering.rst.txt new file mode 100644 index 0000000000..0aee7f765a --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-upstream-peering.rst.txt @@ -0,0 +1,65 @@ +.. meta:: + :description: Netris VPC anywhere upstream peering options + +************************************************************** +Connecting VPC to upstream networks (use one of two options) +************************************************************** + +Using a VLAN with public IP addresses (DMZ) +=========================================== +Netris VPC can use a traditional subnet or vlan with public IP addresses (a DMZ network) for upstream network peering. + +**logical diagram** + +.. image:: images/upstream-dmz-logical.png + +**physical diagram** + +.. image:: images/upstream-dmz-physical.png + +BGP Upstream +============ +BGP Upstream: overview +---------------------- +A BGP peering between Netris VPC and upstream routers is the most recommended option, especially for production use. + +In this scenario, SoftGate nodes form BGP peering with upstream routers. Typically each BGP session will use an individual VLAN id and /30 subnet. Upstream BGP router should advertise the default route 0.0.0.0/0 or full Internet table if necessary. Netris SoftGate nodes are designed to handle over 1M routes in the routing table and can perform as border routers for the full-view table. +SoftGate nodes will automatically advertise public IP subnets in the current site as defined under the Net->IPAM section. Alternatively, you can optionally alter the default settings for full granular control over sent and received prefixes. + +**Logical diagram** + +.. image:: images/vpc-anywhere-upstream-bgp-router-logical.png + :align: center + + +**Physical diagram** + +.. image:: images/vpc-anywhere-upstream-bgp-router-physical.png + :align: center + + +BGP Upstream: configuration +--------------------------- + +Your local public AS number is a site attribute located under the Net->Sites section. You can use the default private AS number or change to a real AS number depending on local network architecture needs. + +.. image:: images/local-public-asn.png + :align: center + +BGP peers can be defined under the Net->E-BGP section. + +.. image:: images/add-new-ebgp-form.png + + +Check BGP neighbor statuses under the Net->E-BGP + +.. image:: images/bgp-listing.png + + +To check/trobleshoot BGP use Net->Looking Glass + +.. image:: images/bgp-looking-glass.png + + + + diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-using-l4lb.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-using-l4lb.rst.txt new file mode 100644 index 0000000000..00bf7c4497 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-using-l4lb.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Using on-demand (elastic) L4 Load Balancer service + +################################################## +Using on-demand (elastic) L4 Load Balancer service +################################################## + +Services --> L4 Load Balancer is an on-demand (elastic) L4 Load Balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console. + +.. image:: /tutorials/images/vpc-anywhere-l4lb.png + :align: center + diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-using-multi-interface-softgate.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-using-multi-interface-softgate.rst.txt new file mode 100644 index 0000000000..d10535a939 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-using-multi-interface-softgate.rst.txt @@ -0,0 +1,35 @@ +.. meta:: + :description: Using Multi-interface SoftGate (experimental) + +############################################# +Using Multi-interface SoftGate (experimental) +############################################# + +By default, Netris uses the bond0 interface of SoftGates exclusively. For each V-NET, Netris creates a new subinterface (e.g., bond0.700) with the next available VLAN ID from the :doc:`Site's defined VLAN ID range `. However, a proposed experimental solution aims to modify this behavior. + + + +Creating a V-Net +================ + +If you want to create a V-Net but prefer Netris to use existing interfaces instead of creating a new subinterface on the bond0 interface, you can achieve that by using the "int=" tag. + +example - ``int=eth1`` + +.. image:: /tutorials/images/vpc-anywhere-vnet-experimental.png + :align: center + + +SoftGate to SoftGate Link +========================= + +In addition to creating a subinterface on the bond0 interface for each V-NET, Netris also creates a separate subinterface for SoftGates' internal use. This subinterface enables two SoftGates to communicate and synchronize mission-critical information. By default, Netris uses the last VLAN ID from the :doc:`Site's defined VLAN ID range ` for this subinterface (e.g., bond0.900). + +To change this behavior and use an existing interface instead of creating a new subinterface, navigate to the **Net → Inventory** section of the Netris web console. Click the **three vertical dots (⋮)** on the right side of the SoftGate node and select **Edit**. In the **Description** field of the opened window, type "int=" and save the changes. + +example - ``int=eth2`` + +.. image:: /tutorials/images/vpc-anywhere-sg-to-sg-experimental.png + :align: center + +Repeat this action for the second SoftGate. diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-using-nat.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-using-nat.rst.txt new file mode 100644 index 0000000000..83b267eba4 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-using-nat.rst.txt @@ -0,0 +1,41 @@ +.. meta:: + :description: Using NAT services + +################## +Using NAT services +################## + +NAT, under Net --> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 172.24.0.0/16 to have access to services outside their VPC. The goal is to provide instances with the capability to connect to the Internet through NAT for outbound access. + +MASQUERADE +========== + +The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn't require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT). + +The Netris Controller includes a preconfigured MASQUERADE rule that facilitates instances within a private network with a subnet of 172.24.0.0/16 to have immediate access to services outside their VPC environment. + +.. image:: images/vpc-anywhere-nat-masquerade.png + :align: center + + +SNAT +==== + +Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a :ref:`NAT purpose`. SNAT replaces the source IP address of the instances with the IP address of the "SNAT to IP". Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity. + +.. image:: images/vpc-anywhere-nat-snat.png + :align: center + + +You can always have more granular control either through NAT rule or using Services → ACLs. + + +DNAT +==== + +A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a "DNAT to IP/Port". This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database. + +I'm creating a DNAT rule for the ssh port in the example below. It forwards the public IP's 55022 port to the local IP's 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server. + +.. image:: images/vpc-anywhere-nat-dnat.png + :align: center diff --git a/en/3.4/_sources/tutorials/vpc-anywhere-using-vnet.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere-using-vnet.rst.txt new file mode 100644 index 0000000000..1aeb0c9b46 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere-using-vnet.rst.txt @@ -0,0 +1,22 @@ +.. meta:: + :description: Using V-Net (isolated virtual network) services + +############################################### +Using V-Net (isolated virtual network) services +############################################### + +V-Net, under Services --> The V-Net service is a virtual network that operates in an isolated environment. In Netris, each V-Net takes the next available VLAN ID from the :doc:`Site's defined VLAN ID range ` and uses it as a global VLAN ID. The SoftGate nodes serve as the default gateway for the V-Net services. + + + +Creating a V-Net +================ + +To ensure proper connectivity, it is recommended to create a corresponding V-Net for each virtual network used in Vmware, KVM, or any other server virtualization platform. The same applies to an isolated group of physical servers. VLAN ID serves as the unique identifier between Netris and the computing platform used. + +.. image:: /tutorials/images/vpc-anywhere-vnet.png + :align: center + +In this example, the new V-NET is assigned VLAN ID 700, subnet 172.24.0.0/20, and gateway 172.24.0.1. As a result, on your Compute side, you can launch VMs or subinterfaces on physical servers with a VLAN ID of 700 and obtain all necessary IP configurations from DHCP. Alternatively, if you prefer not to use DHCP, instances should use IP addresses from the 172.24.0.0/20 subnet and set the default gateway to 172.24.0.1. Netris SoftGate will manage this traffic, and the preconfigured MASQUERADE rule for the 172.24.0.0/16 subnet included in the Netris Controller will enable hosts in VLAN 700 to have internet access through NAT. + +Note that you can use Services --> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites) diff --git a/en/3.4/_sources/tutorials/vpc-anywhere.rst.txt b/en/3.4/_sources/tutorials/vpc-anywhere.rst.txt new file mode 100644 index 0000000000..2e85e99bd4 --- /dev/null +++ b/en/3.4/_sources/tutorials/vpc-anywhere.rst.txt @@ -0,0 +1,28 @@ +================================== +VPC Anywhere Getting Started Guide +================================== + + +.. toctree:: + :maxdepth: 2 + + vpc-anywhere-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + vpc-anywhere-controller-installation + vpc-anywhere-check-default-site + vpc-anywhere-softgate-installation + vpc-anywhere-upstream-peering + vpc-anywhere-ipam-setup + +.. toctree:: + :maxdepth: 2 + :caption: Use + + vpc-anywhere-using-nat + vpc-anywhere-using-vnet + vpc-anywhere-using-l4lb + vpc-anywhere-using-multi-interface-softgate diff --git a/en/3.4/_sources/visibility.rst.txt b/en/3.4/_sources/visibility.rst.txt new file mode 100644 index 0000000000..cba565de9d --- /dev/null +++ b/en/3.4/_sources/visibility.rst.txt @@ -0,0 +1,84 @@ +.. meta:: + :description: Netris System Visibility, Monitoring & Telemetry + +********************** +Visibility (Telescope) +********************** + +Graph Boards +================= +You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view. + +To start with Graph Boards, first, you need to add a new Graph Board. + +1. Navigate to Telescope→Graph Boards, open the dropdown menu in the top left corner, then click +Add board. + +.. image:: images/telescope.png + :align: center + +2. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants. + +.. image:: images/createboard.png + :align: center + +Now you can add graphs by clicking +Add graph. + +Description of +Add graph fields: + +- **Title** - Title for the new graph. +- **Type** - Type of data source. + + - Bps - Traffic bits per second. + - Pps - Traffic packets per second. + - Errors - Errors per second. + - Optical - Optical signal statistics/history. + - MAC Count - History of the number of MAC addresses on the port. +- **Function** - Currently, only summing is supported. +- **+Member** - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port. + +Example: Sum of traffic on two ISP(Iris1 + Iris2) links. + +.. image:: images/ISP_Iris.png + :align: center + +Example: Sum of the traffic on all ports under the service called “my V-NET” + +.. image:: images/V_NET.png + :align: center + +Screenshot: Listing of a Graph Board with the explanation of the controls. + +.. image:: images/graphboard.png + :align: center + +API Logs +======== +Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type. + +Dashboard +========= +Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems. + +Telescope→Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner. + +Description of the pie charts. + +* **Hardware Health** - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions. +* **E-BGP** - Statuses of external BGP sessions. +* **LB VIP** - Statuses of Load Balancer frontend / VIP availability. +* **LB Members** - Statuses of Load Balancer backend members. + +By clicking on each title you can see the details of the checks on the right side. + +Screenshot: Dashboard showing details of “Hardware Health.” + +.. image:: images/hardware_health.png + :align: center + +Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state. + +Screenshot: “Save as normal” on selected ports. + +.. image:: images/saveasnormal.png + :align: center + diff --git a/en/3.4/_sources/vnet.rst.txt b/en/3.4/_sources/vnet.rst.txt new file mode 100644 index 0000000000..99b1ceaa41 --- /dev/null +++ b/en/3.4/_sources/vnet.rst.txt @@ -0,0 +1,52 @@ +.. meta:: + :description: V-Net + +.. _v-net_def: + +##### +V-Net +##### +V-Net is a virtual networking service that provides a Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or they can be created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization). +Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes. + + +V-Net Fields +============ + +- **Name** - Unique name for the V-Net +- **Owner** - Tenant, who can make any changes to current V-Net +- **V-Net state** - Active/Disable state for entire V-Net +- **VLAN aware** - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible +- **Guest tenants** - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters +- **Sites** - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites). +- **IPv4 Gateway** - IPv4 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site where V-Net is intended to span. +- **IPv6 Gateway** - IPv6 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site or sites where V-Net is intended to span. +- **Port** - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports. + + - **Enabled** - Enable or disable individual Switch Port under current V-Net + - **Port Name** - Switch Port format: (swp)@ + - **VLAN ID / Untag** - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port's ingress/egress unless VLAN aware mode is used. + - **LAG Mode** - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes). + +.. tip:: Many switches can't autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gbps switch ports, you'll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +.. image:: images/add-vnet.png + :align: center + :class: with-shadow + +.. centered:: + Example: Adding a new V-Net. + +.. image:: images/list-vnet.png + :align: center + :class: with-shadow + +.. centered:: + Example: Listing of V-Nets. + +.. image:: images/list-vnet-expanded.png + :align: center + :class: with-shadow + +.. centered:: + Example: Expanded view of a V-Net listing. \ No newline at end of file diff --git a/en/3.4/_static/_sphinx_javascript_frameworks_compat.js b/en/3.4/_static/_sphinx_javascript_frameworks_compat.js new file mode 100644 index 0000000000..8549469dc2 --- /dev/null +++ b/en/3.4/_static/_sphinx_javascript_frameworks_compat.js @@ -0,0 +1,134 @@ +/* + * _sphinx_javascript_frameworks_compat.js + * ~~~~~~~~~~ + * + * Compatability shim for jQuery and underscores.js. + * + * WILL BE REMOVED IN Sphinx 6.0 + * xref RemovedInSphinx60Warning + * + */ + +/** + * select a different prefix for underscore + */ +$u = _.noConflict(); + + +/** + * small helper function to urldecode strings + * + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL + */ +jQuery.urldecode = function(x) { + if (!x) { + return x + } + return decodeURIComponent(x.replace(/\+/g, ' ')); +}; + +/** + * small helper function to urlencode strings + */ +jQuery.urlencode = encodeURIComponent; + +/** + * This function returns the parsed url parameters of the + * current request. Multiple values per key are supported, + * it will always return arrays of strings for the value parts. + */ +jQuery.getQueryParameters = function(s) { + if (typeof s === 'undefined') + s = document.location.search; + var parts = s.substr(s.indexOf('?') + 1).split('&'); + var result = {}; + for (var i = 0; i < parts.length; i++) { + var tmp = parts[i].split('=', 2); + var key = jQuery.urldecode(tmp[0]); + var value = jQuery.urldecode(tmp[1]); + if (key in result) + result[key].push(value); + else + result[key] = [value]; + } + return result; +}; + +/** + * highlight a given string on a jquery object by wrapping it in + * span elements with the given class name. + */ +jQuery.fn.highlightText = function(text, className) { + function highlight(node, addItems) { + if (node.nodeType === 3) { + var val = node.nodeValue; + var pos = val.toLowerCase().indexOf(text); + if (pos >= 0 && + !jQuery(node.parentNode).hasClass(className) && + !jQuery(node.parentNode).hasClass("nohighlight")) { + var span; + var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.className = className; + } + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + node.parentNode.insertBefore(span, node.parentNode.insertBefore( + document.createTextNode(val.substr(pos + text.length)), + node.nextSibling)); + node.nodeValue = val.substr(0, pos); + if (isInSVG) { + var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect"); + var bbox = node.parentElement.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute('class', className); + addItems.push({ + "parent": node.parentNode, + "target": rect}); + } + } + } + else if (!jQuery(node).is("button, select, textarea")) { + jQuery.each(node.childNodes, function() { + highlight(this, addItems); + }); + } + } + var addItems = []; + var result = this.each(function() { + highlight(this, addItems); + }); + for (var i = 0; i < addItems.length; ++i) { + jQuery(addItems[i].parent).before(addItems[i].target); + } + return result; +}; + +/* + * backward compatibility for jQuery.browser + * This will be supported until firefox bug is fixed. + */ +if (!jQuery.browser) { + jQuery.uaMatch = function(ua) { + ua = ua.toLowerCase(); + + var match = /(chrome)[ \/]([\w.]+)/.exec(ua) || + /(webkit)[ \/]([\w.]+)/.exec(ua) || + /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) || + /(msie) ([\w.]+)/.exec(ua) || + ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) || + []; + + return { + browser: match[ 1 ] || "", + version: match[ 2 ] || "0" + }; + }; + jQuery.browser = {}; + jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true; +} diff --git a/en/3.4/_static/basic.css b/en/3.4/_static/basic.css new file mode 100644 index 0000000000..eeb0519a69 --- /dev/null +++ b/en/3.4/_static/basic.css @@ -0,0 +1,899 @@ +/* + * basic.css + * ~~~~~~~~~ + * + * Sphinx stylesheet -- basic theme. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +div.section::after { + display: block; + content: ''; + clear: left; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; + word-wrap: break-word; + overflow-wrap : break-word; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar #searchbox form.search { + overflow: hidden; +} + +div.sphinxsidebar #searchbox input[type="text"] { + float: left; + width: 80%; + padding: 0.25em; + box-sizing: border-box; +} + +div.sphinxsidebar #searchbox input[type="submit"] { + float: left; + width: 20%; + border-left: none; + padding: 0.25em; + box-sizing: border-box; +} + + +img { + border: 0; + max-width: 100%; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin: 10px 0 0 20px; + padding: 0; +} + +ul.search li { + padding: 5px 0 5px 20px; + background-image: url(file.png); + background-repeat: no-repeat; + background-position: 0 7px; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li p.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; + margin-left: auto; + margin-right: auto; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable ul { + margin-top: 0; + margin-bottom: 0; + list-style-type: none; +} + +table.indextable > tbody > tr > td > ul { + padding-left: 0em; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- domain module index --------------------------------------------------- */ + +table.modindextable td { + padding: 2px; + border-collapse: collapse; +} + +/* -- general body styles --------------------------------------------------- */ + +div.body { + min-width: 360px; + max-width: 800px; +} + +div.body p, div.body dd, div.body li, div.body blockquote { + -moz-hyphens: auto; + -ms-hyphens: auto; + -webkit-hyphens: auto; + hyphens: auto; +} + +a.headerlink { + visibility: hidden; +} +a.brackets:before, +span.brackets > a:before{ + content: "["; +} + +a.brackets:after, +span.brackets > a:after { + content: "]"; +} + + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { + visibility: visible; +} + +div.body p.caption { + text-align: inherit; +} + +div.body td { + text-align: left; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +img.align-left, figure.align-left, .figure.align-left, object.align-left { + clear: left; + float: left; + margin-right: 1em; +} + +img.align-right, figure.align-right, .figure.align-right, object.align-right { + clear: right; + float: right; + margin-left: 1em; +} + +img.align-center, figure.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +img.align-default, figure.align-default, .figure.align-default { + display: block; + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left; +} + +.align-center { + text-align: center; +} + +.align-default { + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar, +aside.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px; + background-color: #ffe; + width: 40%; + float: right; + clear: right; + overflow-x: auto; +} + +p.sidebar-title { + font-weight: bold; +} +div.admonition, div.topic, blockquote { + clear: left; +} + +/* -- topics ---------------------------------------------------------------- */ +div.topic { + border: 1px solid #ccc; + padding: 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- content of sidebars/topics/admonitions -------------------------------- */ + +div.sidebar > :last-child, +aside.sidebar > :last-child, +div.topic > :last-child, +div.admonition > :last-child { + margin-bottom: 0; +} + +div.sidebar::after, +aside.sidebar::after, +div.topic::after, +div.admonition::after, +blockquote::after { + display: block; + content: ''; + clear: both; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + margin-top: 10px; + margin-bottom: 10px; + border: 0; + border-collapse: collapse; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +table.align-default { + margin-left: auto; + margin-right: auto; +} + +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +th > :first-child, +td > :first-child { + margin-top: 0px; +} + +th > :last-child, +td > :last-child { + margin-bottom: 0px; +} + +/* -- figures --------------------------------------------------------------- */ + +div.figure, figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption, figcaption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number, +figcaption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text, +figcaption span.caption-text { +} + +/* -- field list styles ----------------------------------------------------- */ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.field-name { + -moz-hyphens: manual; + -ms-hyphens: manual; + -webkit-hyphens: manual; + hyphens: manual; +} + +/* -- hlist styles ---------------------------------------------------------- */ + +table.hlist { + margin: 1em 0; +} + +table.hlist td { + vertical-align: top; +} + +/* -- object description styles --------------------------------------------- */ + +.sig { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; +} + +.sig-name, code.descname { + background-color: transparent; + font-weight: bold; +} + +.sig-name { + font-size: 1.1em; +} + +code.descname { + font-size: 1.2em; +} + +.sig-prename, code.descclassname { + background-color: transparent; +} + +.optional { + font-size: 1.3em; +} + +.sig-paren { + font-size: larger; +} + +.sig-param.n { + font-style: italic; +} + +/* C++ specific styling */ + +.sig-inline.c-texpr, +.sig-inline.cpp-texpr { + font-family: unset; +} + +.sig.c .k, .sig.c .kt, +.sig.cpp .k, .sig.cpp .kt { + color: #0033B3; +} + +.sig.c .m, +.sig.cpp .m { + color: #1750EB; +} + +.sig.c .s, .sig.c .sc, +.sig.cpp .s, .sig.cpp .sc { + color: #067D17; +} + + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +:not(li) > ol > li:first-child > :first-child, +:not(li) > ul > li:first-child > :first-child { + margin-top: 0px; +} + +:not(li) > ol > li:last-child > :last-child, +:not(li) > ul > li:last-child > :last-child { + margin-bottom: 0px; +} + +ol.simple ol p, +ol.simple ul p, +ul.simple ol p, +ul.simple ul p { + margin-top: 0; +} + +ol.simple > li:not(:first-child) > p, +ul.simple > li:not(:first-child) > p { + margin-top: 0; +} + +ol.simple p, +ul.simple p { + margin-bottom: 0; +} +dl.footnote > dt, +dl.citation > dt { + float: left; + margin-right: 0.5em; +} + +dl.footnote > dd, +dl.citation > dd { + margin-bottom: 0em; +} + +dl.footnote > dd:after, +dl.citation > dd:after { + content: ""; + clear: both; +} + +dl.field-list { + display: grid; + grid-template-columns: fit-content(30%) auto; +} + +dl.field-list > dt { + font-weight: bold; + word-break: break-word; + padding-left: 0.5em; + padding-right: 5px; +} +dl.field-list > dt:after { + content: ":"; +} + + +dl.field-list > dd { + padding-left: 0.5em; + margin-top: 0em; + margin-left: 0em; + margin-bottom: 0em; +} + +dl { + margin-bottom: 15px; +} + +dd > :first-child { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +dl > dd:last-child, +dl > dd:last-child > :last-child { + margin-bottom: 0; +} + +dt:target, span.highlighted { + background-color: #fbe54e; +} + +rect.highlighted { + fill: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa; +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +.classifier:before { + font-style: normal; + margin: 0 0.5em; + content: ":"; + display: inline-block; +} + +abbr, acronym { + border-bottom: dotted 1px; + cursor: help; +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; + overflow-y: hidden; /* fixes display issues on Chrome browsers */ +} + +pre, div[class*="highlight-"] { + clear: both; +} + +span.pre { + -moz-hyphens: none; + -ms-hyphens: none; + -webkit-hyphens: none; + hyphens: none; + white-space: nowrap; +} + +div[class*="highlight-"] { + margin: 1em 0; +} + +td.linenos pre { + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + display: block; +} + +table.highlighttable tbody { + display: block; +} + +table.highlighttable tr { + display: flex; +} + +table.highlighttable td { + margin: 0; + padding: 0; +} + +table.highlighttable td.linenos { + padding-right: 0.5em; +} + +table.highlighttable td.code { + flex: 1; + overflow: hidden; +} + +.highlight .hll { + display: block; +} + +div.highlight pre, +table.highlighttable pre { + margin: 0; +} + +div.code-block-caption + div { + margin-top: 0; +} + +div.code-block-caption { + margin-top: 1em; + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +table.highlighttable td.linenos, +span.linenos, +div.highlight span.gp { /* gp: Generic.Prompt */ + user-select: none; + -webkit-user-select: text; /* Safari fallback only */ + -webkit-user-select: none; /* Chrome/Safari */ + -moz-user-select: none; /* Firefox */ + -ms-user-select: none; /* IE10+ */ +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + margin: 1em 0; +} + +code.xref, a code { + background-color: transparent; + font-weight: bold; +} + +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.body div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +span.eqno a.headerlink { + position: absolute; + z-index: 1; +} + +div.math:hover a.headerlink { + visibility: visible; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} \ No newline at end of file diff --git a/en/3.4/_static/check-solid.svg b/en/3.4/_static/check-solid.svg new file mode 100644 index 0000000000..92fad4b5c0 --- /dev/null +++ b/en/3.4/_static/check-solid.svg @@ -0,0 +1,4 @@ + + + + diff --git a/en/3.4/_static/clipboard.min.js b/en/3.4/_static/clipboard.min.js new file mode 100644 index 0000000000..54b3c46381 --- /dev/null +++ b/en/3.4/_static/clipboard.min.js @@ -0,0 +1,7 @@ +/*! + * clipboard.js v2.0.8 + * https://clipboardjs.com/ + * + * Licensed MIT © Zeno Rocha + */ +!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1 + + + + diff --git a/en/3.4/_static/copybutton.css b/en/3.4/_static/copybutton.css new file mode 100644 index 0000000000..f1916ec7d1 --- /dev/null +++ b/en/3.4/_static/copybutton.css @@ -0,0 +1,94 @@ +/* Copy buttons */ +button.copybtn { + position: absolute; + display: flex; + top: .3em; + right: .3em; + width: 1.7em; + height: 1.7em; + opacity: 0; + transition: opacity 0.3s, border .3s, background-color .3s; + user-select: none; + padding: 0; + border: none; + outline: none; + border-radius: 0.4em; + /* The colors that GitHub uses */ + border: #1b1f2426 1px solid; + background-color: #f6f8fa; + color: #57606a; +} + +button.copybtn.success { + border-color: #22863a; + color: #22863a; +} + +button.copybtn svg { + stroke: currentColor; + width: 1.5em; + height: 1.5em; + padding: 0.1em; +} + +div.highlight { + position: relative; +} + +/* Show the copybutton */ +.highlight:hover button.copybtn, button.copybtn.success { + opacity: 1; +} + +.highlight button.copybtn:hover { + background-color: rgb(235, 235, 235); +} + +.highlight button.copybtn:active { + background-color: rgb(187, 187, 187); +} + +/** + * A minimal CSS-only tooltip copied from: + * https://codepen.io/mildrenben/pen/rVBrpK + * + * To use, write HTML like the following: + * + *

Short

+ */ + .o-tooltip--left { + position: relative; + } + + .o-tooltip--left:after { + opacity: 0; + visibility: hidden; + position: absolute; + content: attr(data-tooltip); + padding: .2em; + font-size: .8em; + left: -.2em; + background: grey; + color: white; + white-space: nowrap; + z-index: 2; + border-radius: 2px; + transform: translateX(-102%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); +} + +.o-tooltip--left:hover:after { + display: block; + opacity: 1; + visibility: visible; + transform: translateX(-100%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); + transition-delay: .5s; +} + +/* By default the copy button shouldn't show up when printing a page */ +@media print { + button.copybtn { + display: none; + } +} diff --git a/en/3.4/_static/copybutton.js b/en/3.4/_static/copybutton.js new file mode 100644 index 0000000000..2ea7ff3e21 --- /dev/null +++ b/en/3.4/_static/copybutton.js @@ -0,0 +1,248 @@ +// Localization support +const messages = { + 'en': { + 'copy': 'Copy', + 'copy_to_clipboard': 'Copy to clipboard', + 'copy_success': 'Copied!', + 'copy_failure': 'Failed to copy', + }, + 'es' : { + 'copy': 'Copiar', + 'copy_to_clipboard': 'Copiar al portapapeles', + 'copy_success': '¡Copiado!', + 'copy_failure': 'Error al copiar', + }, + 'de' : { + 'copy': 'Kopieren', + 'copy_to_clipboard': 'In die Zwischenablage kopieren', + 'copy_success': 'Kopiert!', + 'copy_failure': 'Fehler beim Kopieren', + }, + 'fr' : { + 'copy': 'Copier', + 'copy_to_clipboard': 'Copier dans le presse-papier', + 'copy_success': 'Copié !', + 'copy_failure': 'Échec de la copie', + }, + 'ru': { + 'copy': 'Скопировать', + 'copy_to_clipboard': 'Скопировать в буфер', + 'copy_success': 'Скопировано!', + 'copy_failure': 'Не удалось скопировать', + }, + 'zh-CN': { + 'copy': '复制', + 'copy_to_clipboard': '复制到剪贴板', + 'copy_success': '复制成功!', + 'copy_failure': '复制失败', + }, + 'it' : { + 'copy': 'Copiare', + 'copy_to_clipboard': 'Copiato negli appunti', + 'copy_success': 'Copiato!', + 'copy_failure': 'Errore durante la copia', + } +} + +let locale = 'en' +if( document.documentElement.lang !== undefined + && messages[document.documentElement.lang] !== undefined ) { + locale = document.documentElement.lang +} + +let doc_url_root = DOCUMENTATION_OPTIONS.URL_ROOT; +if (doc_url_root == '#') { + doc_url_root = ''; +} + +/** + * SVG files for our copy buttons + */ +let iconCheck = ` + ${messages[locale]['copy_success']} + + +` + +// If the user specified their own SVG use that, otherwise use the default +let iconCopy = ``; +if (!iconCopy) { + iconCopy = ` + ${messages[locale]['copy_to_clipboard']} + + + +` +} + +/** + * Set up copy/paste for code blocks + */ + +const runWhenDOMLoaded = cb => { + if (document.readyState != 'loading') { + cb() + } else if (document.addEventListener) { + document.addEventListener('DOMContentLoaded', cb) + } else { + document.attachEvent('onreadystatechange', function() { + if (document.readyState == 'complete') cb() + }) + } +} + +const codeCellId = index => `codecell${index}` + +// Clears selected text since ClipboardJS will select the text when copying +const clearSelection = () => { + if (window.getSelection) { + window.getSelection().removeAllRanges() + } else if (document.selection) { + document.selection.empty() + } +} + +// Changes tooltip text for a moment, then changes it back +// We want the timeout of our `success` class to be a bit shorter than the +// tooltip and icon change, so that we can hide the icon before changing back. +var timeoutIcon = 2000; +var timeoutSuccessClass = 1500; + +const temporarilyChangeTooltip = (el, oldText, newText) => { + el.setAttribute('data-tooltip', newText) + el.classList.add('success') + // Remove success a little bit sooner than we change the tooltip + // So that we can use CSS to hide the copybutton first + setTimeout(() => el.classList.remove('success'), timeoutSuccessClass) + setTimeout(() => el.setAttribute('data-tooltip', oldText), timeoutIcon) +} + +// Changes the copy button icon for two seconds, then changes it back +const temporarilyChangeIcon = (el) => { + el.innerHTML = iconCheck; + setTimeout(() => {el.innerHTML = iconCopy}, timeoutIcon) +} + +const addCopyButtonToCodeCells = () => { + // If ClipboardJS hasn't loaded, wait a bit and try again. This + // happens because we load ClipboardJS asynchronously. + if (window.ClipboardJS === undefined) { + setTimeout(addCopyButtonToCodeCells, 250) + return + } + + // Add copybuttons to all of our code cells + const COPYBUTTON_SELECTOR = 'div.highlight pre'; + const codeCells = document.querySelectorAll(COPYBUTTON_SELECTOR) + codeCells.forEach((codeCell, index) => { + const id = codeCellId(index) + codeCell.setAttribute('id', id) + + const clipboardButton = id => + `` + codeCell.insertAdjacentHTML('afterend', clipboardButton(id)) + }) + +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} + + +var copyTargetText = (trigger) => { + var target = document.querySelector(trigger.attributes['data-clipboard-target'].value); + + // get filtered text + let exclude = '.linenos'; + + let text = filterText(target, exclude); + return formatCopyText(text, '', false, true, true, true, '', '') +} + + // Initialize with a callback so we can modify the text before copy + const clipboard = new ClipboardJS('.copybtn', {text: copyTargetText}) + + // Update UI with error/success messages + clipboard.on('success', event => { + clearSelection() + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_success']) + temporarilyChangeIcon(event.trigger) + }) + + clipboard.on('error', event => { + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_failure']) + }) +} + +runWhenDOMLoaded(addCopyButtonToCodeCells) \ No newline at end of file diff --git a/en/3.4/_static/copybutton_funcs.js b/en/3.4/_static/copybutton_funcs.js new file mode 100644 index 0000000000..dbe1aaad79 --- /dev/null +++ b/en/3.4/_static/copybutton_funcs.js @@ -0,0 +1,73 @@ +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +export function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +export function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} diff --git a/en/3.4/_static/css/badge_only.css b/en/3.4/_static/css/badge_only.css new file mode 100644 index 0000000000..e380325bc6 --- /dev/null +++ b/en/3.4/_static/css/badge_only.css @@ -0,0 +1 @@ +.fa:before{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-style:normal;font-weight:400;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#FontAwesome) format("svg")}.fa:before{font-family:FontAwesome;font-style:normal;font-weight:400;line-height:1}.fa:before,a .fa{text-decoration:inherit}.fa:before,a .fa,li .fa{display:inline-block}li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-.8em}ul.fas li .fa{width:.8em}ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before,.icon-book:before{content:"\f02d"}.fa-caret-down:before,.icon-caret-down:before{content:"\f0d7"}.fa-caret-up:before,.icon-caret-up:before{content:"\f0d8"}.fa-caret-left:before,.icon-caret-left:before{content:"\f0d9"}.fa-caret-right:before,.icon-caret-right:before{content:"\f0da"}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60}.rst-versions .rst-current-version:after{clear:both;content:"";display:block}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}} \ No newline at end of file diff --git a/en/3.4/_static/css/fonts/Roboto-Slab-Bold.woff b/en/3.4/_static/css/fonts/Roboto-Slab-Bold.woff new file mode 100644 index 0000000000..6cb6000018 Binary files /dev/null and b/en/3.4/_static/css/fonts/Roboto-Slab-Bold.woff differ diff --git a/en/3.4/_static/css/fonts/Roboto-Slab-Bold.woff2 b/en/3.4/_static/css/fonts/Roboto-Slab-Bold.woff2 new file mode 100644 index 0000000000..7059e23142 Binary files /dev/null and b/en/3.4/_static/css/fonts/Roboto-Slab-Bold.woff2 differ diff --git a/en/3.4/_static/css/fonts/Roboto-Slab-Regular.woff b/en/3.4/_static/css/fonts/Roboto-Slab-Regular.woff new file mode 100644 index 0000000000..f815f63f99 Binary files /dev/null and b/en/3.4/_static/css/fonts/Roboto-Slab-Regular.woff differ diff --git a/en/3.4/_static/css/fonts/Roboto-Slab-Regular.woff2 b/en/3.4/_static/css/fonts/Roboto-Slab-Regular.woff2 new file mode 100644 index 0000000000..f2c76e5bda Binary files /dev/null and b/en/3.4/_static/css/fonts/Roboto-Slab-Regular.woff2 differ diff --git a/en/3.4/_static/css/fonts/fontawesome-webfont.eot b/en/3.4/_static/css/fonts/fontawesome-webfont.eot new file mode 100644 index 0000000000..e9f60ca953 Binary files /dev/null and b/en/3.4/_static/css/fonts/fontawesome-webfont.eot differ diff --git a/en/3.4/_static/css/fonts/fontawesome-webfont.svg b/en/3.4/_static/css/fonts/fontawesome-webfont.svg new file mode 100644 index 0000000000..855c845e53 --- /dev/null +++ b/en/3.4/_static/css/fonts/fontawesome-webfont.svg @@ -0,0 +1,2671 @@ + + + + +Created by FontForge 20120731 at Mon Oct 24 17:37:40 2016 + By ,,, +Copyright Dave Gandy 2016. All rights reserved. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/en/3.4/_static/css/fonts/fontawesome-webfont.ttf b/en/3.4/_static/css/fonts/fontawesome-webfont.ttf new file mode 100644 index 0000000000..35acda2fa1 Binary files /dev/null and b/en/3.4/_static/css/fonts/fontawesome-webfont.ttf differ diff --git a/en/3.4/_static/css/fonts/fontawesome-webfont.woff b/en/3.4/_static/css/fonts/fontawesome-webfont.woff new file mode 100644 index 0000000000..400014a4b0 Binary files /dev/null and b/en/3.4/_static/css/fonts/fontawesome-webfont.woff differ diff --git a/en/3.4/_static/css/fonts/fontawesome-webfont.woff2 b/en/3.4/_static/css/fonts/fontawesome-webfont.woff2 new file mode 100644 index 0000000000..4d13fc6040 Binary files /dev/null and b/en/3.4/_static/css/fonts/fontawesome-webfont.woff2 differ diff --git a/en/3.4/_static/css/fonts/lato-bold-italic.woff b/en/3.4/_static/css/fonts/lato-bold-italic.woff new file mode 100644 index 0000000000..88ad05b9ff Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-bold-italic.woff differ diff --git a/en/3.4/_static/css/fonts/lato-bold-italic.woff2 b/en/3.4/_static/css/fonts/lato-bold-italic.woff2 new file mode 100644 index 0000000000..c4e3d804b5 Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-bold-italic.woff2 differ diff --git a/en/3.4/_static/css/fonts/lato-bold.woff b/en/3.4/_static/css/fonts/lato-bold.woff new file mode 100644 index 0000000000..c6dff51f06 Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-bold.woff differ diff --git a/en/3.4/_static/css/fonts/lato-bold.woff2 b/en/3.4/_static/css/fonts/lato-bold.woff2 new file mode 100644 index 0000000000..bb195043cf Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-bold.woff2 differ diff --git a/en/3.4/_static/css/fonts/lato-normal-italic.woff b/en/3.4/_static/css/fonts/lato-normal-italic.woff new file mode 100644 index 0000000000..76114bc033 Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-normal-italic.woff differ diff --git a/en/3.4/_static/css/fonts/lato-normal-italic.woff2 b/en/3.4/_static/css/fonts/lato-normal-italic.woff2 new file mode 100644 index 0000000000..3404f37e2e Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-normal-italic.woff2 differ diff --git a/en/3.4/_static/css/fonts/lato-normal.woff b/en/3.4/_static/css/fonts/lato-normal.woff new file mode 100644 index 0000000000..ae1307ff5f Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-normal.woff differ diff --git a/en/3.4/_static/css/fonts/lato-normal.woff2 b/en/3.4/_static/css/fonts/lato-normal.woff2 new file mode 100644 index 0000000000..3bf9843328 Binary files /dev/null and b/en/3.4/_static/css/fonts/lato-normal.woff2 differ diff --git a/en/3.4/_static/css/theme.css b/en/3.4/_static/css/theme.css new file mode 100644 index 0000000000..8cd4f101a9 --- /dev/null +++ b/en/3.4/_static/css/theme.css @@ -0,0 +1,4 @@ +html{box-sizing:border-box}*,:after,:before{box-sizing:inherit}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}[hidden],audio:not([controls]){display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:active,a:hover{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:700}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;text-decoration:none}ins,mark{color:#000}mark{background:#ff0;font-style:italic;font-weight:700}.rst-content code,.rst-content tt,code,kbd,pre,samp{font-family:monospace,serif;_font-family:courier new,monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:after,q:before{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sup{top:-.5em}sub{bottom:-.25em}dl,ol,ul{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure,form{margin:0}label{cursor:pointer}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type=button],input[type=reset],input[type=submit]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type=search]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}textarea{resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:.2em 0;background:#ccc;color:#000;padding:.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none!important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{body,html,section{background:none!important}*{box-shadow:none!important;text-shadow:none!important;filter:none!important;-ms-filter:none!important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="#"]:after,a[href^="javascript:"]:after{content:""}blockquote,pre{page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}img{max-width:100%!important}@page{margin:.5cm}.rst-content .toctree-wrapper>p.caption,h2,h3,p{orphans:3;widows:3}.rst-content .toctree-wrapper>p.caption,h2,h3{page-break-after:avoid}}.btn,.fa:before,.icon:before,.rst-content .admonition,.rst-content .admonition-title:before,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .code-block-caption .headerlink:before,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-alert,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before,.wy-nav-top a,.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a,input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week],select,textarea{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}/*! + * Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */@font-face{font-family:FontAwesome;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713);src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix&v=4.7.0) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#fontawesomeregular) format("svg");font-weight:400;font-style:normal}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14286em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14286em;width:2.14286em;top:.14286em;text-align:center}.fa-li.fa-lg{left:-1.85714em}.fa-border{padding:.2em .25em .15em;border:.08em solid #eee;border-radius:.1em}.fa-pull-left{float:left}.fa-pull-right{float:right}.fa-pull-left.icon,.fa.fa-pull-left,.rst-content .code-block-caption .fa-pull-left.headerlink,.rst-content .fa-pull-left.admonition-title,.rst-content code.download span.fa-pull-left:first-child,.rst-content dl dt .fa-pull-left.headerlink,.rst-content h1 .fa-pull-left.headerlink,.rst-content h2 .fa-pull-left.headerlink,.rst-content h3 .fa-pull-left.headerlink,.rst-content h4 .fa-pull-left.headerlink,.rst-content h5 .fa-pull-left.headerlink,.rst-content h6 .fa-pull-left.headerlink,.rst-content p.caption .fa-pull-left.headerlink,.rst-content table>caption .fa-pull-left.headerlink,.rst-content tt.download span.fa-pull-left:first-child,.wy-menu-vertical li.current>a span.fa-pull-left.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-left.toctree-expand,.wy-menu-vertical li span.fa-pull-left.toctree-expand{margin-right:.3em}.fa-pull-right.icon,.fa.fa-pull-right,.rst-content .code-block-caption .fa-pull-right.headerlink,.rst-content .fa-pull-right.admonition-title,.rst-content code.download span.fa-pull-right:first-child,.rst-content dl dt .fa-pull-right.headerlink,.rst-content h1 .fa-pull-right.headerlink,.rst-content h2 .fa-pull-right.headerlink,.rst-content h3 .fa-pull-right.headerlink,.rst-content h4 .fa-pull-right.headerlink,.rst-content h5 .fa-pull-right.headerlink,.rst-content h6 .fa-pull-right.headerlink,.rst-content p.caption .fa-pull-right.headerlink,.rst-content table>caption .fa-pull-right.headerlink,.rst-content tt.download span.fa-pull-right:first-child,.wy-menu-vertical li.current>a span.fa-pull-right.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-right.toctree-expand,.wy-menu-vertical li span.fa-pull-right.toctree-expand{margin-left:.3em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left,.pull-left.icon,.rst-content .code-block-caption .pull-left.headerlink,.rst-content .pull-left.admonition-title,.rst-content code.download span.pull-left:first-child,.rst-content dl dt .pull-left.headerlink,.rst-content h1 .pull-left.headerlink,.rst-content h2 .pull-left.headerlink,.rst-content h3 .pull-left.headerlink,.rst-content h4 .pull-left.headerlink,.rst-content h5 .pull-left.headerlink,.rst-content h6 .pull-left.headerlink,.rst-content p.caption .pull-left.headerlink,.rst-content table>caption .pull-left.headerlink,.rst-content tt.download span.pull-left:first-child,.wy-menu-vertical li.current>a span.pull-left.toctree-expand,.wy-menu-vertical li.on a span.pull-left.toctree-expand,.wy-menu-vertical li span.pull-left.toctree-expand{margin-right:.3em}.fa.pull-right,.pull-right.icon,.rst-content .code-block-caption .pull-right.headerlink,.rst-content .pull-right.admonition-title,.rst-content code.download span.pull-right:first-child,.rst-content dl dt .pull-right.headerlink,.rst-content h1 .pull-right.headerlink,.rst-content h2 .pull-right.headerlink,.rst-content h3 .pull-right.headerlink,.rst-content h4 .pull-right.headerlink,.rst-content h5 .pull-right.headerlink,.rst-content h6 .pull-right.headerlink,.rst-content p.caption .pull-right.headerlink,.rst-content table>caption .pull-right.headerlink,.rst-content tt.download span.pull-right:first-child,.wy-menu-vertical li.current>a span.pull-right.toctree-expand,.wy-menu-vertical li.on a span.pull-right.toctree-expand,.wy-menu-vertical li span.pull-right.toctree-expand{margin-left:.3em}.fa-spin{-webkit-animation:fa-spin 2s linear infinite;animation:fa-spin 2s linear infinite}.fa-pulse{-webkit-animation:fa-spin 1s steps(8) infinite;animation:fa-spin 1s steps(8) infinite}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";-webkit-transform:scaleX(-1);-ms-transform:scaleX(-1);transform:scaleX(-1)}.fa-flip-vertical{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";-webkit-transform:scaleY(-1);-ms-transform:scaleY(-1);transform:scaleY(-1)}:root .fa-flip-horizontal,:root .fa-flip-vertical,:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:""}.fa-music:before{content:""}.fa-search:before,.icon-search:before{content:""}.fa-envelope-o:before{content:""}.fa-heart:before{content:""}.fa-star:before{content:""}.fa-star-o:before{content:""}.fa-user:before{content:""}.fa-film:before{content:""}.fa-th-large:before{content:""}.fa-th:before{content:""}.fa-th-list:before{content:""}.fa-check:before{content:""}.fa-close:before,.fa-remove:before,.fa-times:before{content:""}.fa-search-plus:before{content:""}.fa-search-minus:before{content:""}.fa-power-off:before{content:""}.fa-signal:before{content:""}.fa-cog:before,.fa-gear:before{content:""}.fa-trash-o:before{content:""}.fa-home:before,.icon-home:before{content:""}.fa-file-o:before{content:""}.fa-clock-o:before{content:""}.fa-road:before{content:""}.fa-download:before,.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{content:""}.fa-arrow-circle-o-down:before{content:""}.fa-arrow-circle-o-up:before{content:""}.fa-inbox:before{content:""}.fa-play-circle-o:before{content:""}.fa-repeat:before,.fa-rotate-right:before{content:""}.fa-refresh:before{content:""}.fa-list-alt:before{content:""}.fa-lock:before{content:""}.fa-flag:before{content:""}.fa-headphones:before{content:""}.fa-volume-off:before{content:""}.fa-volume-down:before{content:""}.fa-volume-up:before{content:""}.fa-qrcode:before{content:""}.fa-barcode:before{content:""}.fa-tag:before{content:""}.fa-tags:before{content:""}.fa-book:before,.icon-book:before{content:""}.fa-bookmark:before{content:""}.fa-print:before{content:""}.fa-camera:before{content:""}.fa-font:before{content:""}.fa-bold:before{content:""}.fa-italic:before{content:""}.fa-text-height:before{content:""}.fa-text-width:before{content:""}.fa-align-left:before{content:""}.fa-align-center:before{content:""}.fa-align-right:before{content:""}.fa-align-justify:before{content:""}.fa-list:before{content:""}.fa-dedent:before,.fa-outdent:before{content:""}.fa-indent:before{content:""}.fa-video-camera:before{content:""}.fa-image:before,.fa-photo:before,.fa-picture-o:before{content:""}.fa-pencil:before{content:""}.fa-map-marker:before{content:""}.fa-adjust:before{content:""}.fa-tint:before{content:""}.fa-edit:before,.fa-pencil-square-o:before{content:""}.fa-share-square-o:before{content:""}.fa-check-square-o:before{content:""}.fa-arrows:before{content:""}.fa-step-backward:before{content:""}.fa-fast-backward:before{content:""}.fa-backward:before{content:""}.fa-play:before{content:""}.fa-pause:before{content:""}.fa-stop:before{content:""}.fa-forward:before{content:""}.fa-fast-forward:before{content:""}.fa-step-forward:before{content:""}.fa-eject:before{content:""}.fa-chevron-left:before{content:""}.fa-chevron-right:before{content:""}.fa-plus-circle:before{content:""}.fa-minus-circle:before{content:""}.fa-times-circle:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before{content:""}.fa-check-circle:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before{content:""}.fa-question-circle:before{content:""}.fa-info-circle:before{content:""}.fa-crosshairs:before{content:""}.fa-times-circle-o:before{content:""}.fa-check-circle-o:before{content:""}.fa-ban:before{content:""}.fa-arrow-left:before{content:""}.fa-arrow-right:before{content:""}.fa-arrow-up:before{content:""}.fa-arrow-down:before{content:""}.fa-mail-forward:before,.fa-share:before{content:""}.fa-expand:before{content:""}.fa-compress:before{content:""}.fa-plus:before{content:""}.fa-minus:before{content:""}.fa-asterisk:before{content:""}.fa-exclamation-circle:before,.rst-content .admonition-title:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before{content:""}.fa-gift:before{content:""}.fa-leaf:before{content:""}.fa-fire:before,.icon-fire:before{content:""}.fa-eye:before{content:""}.fa-eye-slash:before{content:""}.fa-exclamation-triangle:before,.fa-warning:before{content:""}.fa-plane:before{content:""}.fa-calendar:before{content:""}.fa-random:before{content:""}.fa-comment:before{content:""}.fa-magnet:before{content:""}.fa-chevron-up:before{content:""}.fa-chevron-down:before{content:""}.fa-retweet:before{content:""}.fa-shopping-cart:before{content:""}.fa-folder:before{content:""}.fa-folder-open:before{content:""}.fa-arrows-v:before{content:""}.fa-arrows-h:before{content:""}.fa-bar-chart-o:before,.fa-bar-chart:before{content:""}.fa-twitter-square:before{content:""}.fa-facebook-square:before{content:""}.fa-camera-retro:before{content:""}.fa-key:before{content:""}.fa-cogs:before,.fa-gears:before{content:""}.fa-comments:before{content:""}.fa-thumbs-o-up:before{content:""}.fa-thumbs-o-down:before{content:""}.fa-star-half:before{content:""}.fa-heart-o:before{content:""}.fa-sign-out:before{content:""}.fa-linkedin-square:before{content:""}.fa-thumb-tack:before{content:""}.fa-external-link:before{content:""}.fa-sign-in:before{content:""}.fa-trophy:before{content:""}.fa-github-square:before{content:""}.fa-upload:before{content:""}.fa-lemon-o:before{content:""}.fa-phone:before{content:""}.fa-square-o:before{content:""}.fa-bookmark-o:before{content:""}.fa-phone-square:before{content:""}.fa-twitter:before{content:""}.fa-facebook-f:before,.fa-facebook:before{content:""}.fa-github:before,.icon-github:before{content:""}.fa-unlock:before{content:""}.fa-credit-card:before{content:""}.fa-feed:before,.fa-rss:before{content:""}.fa-hdd-o:before{content:""}.fa-bullhorn:before{content:""}.fa-bell:before{content:""}.fa-certificate:before{content:""}.fa-hand-o-right:before{content:""}.fa-hand-o-left:before{content:""}.fa-hand-o-up:before{content:""}.fa-hand-o-down:before{content:""}.fa-arrow-circle-left:before,.icon-circle-arrow-left:before{content:""}.fa-arrow-circle-right:before,.icon-circle-arrow-right:before{content:""}.fa-arrow-circle-up:before{content:""}.fa-arrow-circle-down:before{content:""}.fa-globe:before{content:""}.fa-wrench:before{content:""}.fa-tasks:before{content:""}.fa-filter:before{content:""}.fa-briefcase:before{content:""}.fa-arrows-alt:before{content:""}.fa-group:before,.fa-users:before{content:""}.fa-chain:before,.fa-link:before,.icon-link:before{content:""}.fa-cloud:before{content:""}.fa-flask:before{content:""}.fa-cut:before,.fa-scissors:before{content:""}.fa-copy:before,.fa-files-o:before{content:""}.fa-paperclip:before{content:""}.fa-floppy-o:before,.fa-save:before{content:""}.fa-square:before{content:""}.fa-bars:before,.fa-navicon:before,.fa-reorder:before{content:""}.fa-list-ul:before{content:""}.fa-list-ol:before{content:""}.fa-strikethrough:before{content:""}.fa-underline:before{content:""}.fa-table:before{content:""}.fa-magic:before{content:""}.fa-truck:before{content:""}.fa-pinterest:before{content:""}.fa-pinterest-square:before{content:""}.fa-google-plus-square:before{content:""}.fa-google-plus:before{content:""}.fa-money:before{content:""}.fa-caret-down:before,.icon-caret-down:before,.wy-dropdown .caret:before{content:""}.fa-caret-up:before{content:""}.fa-caret-left:before{content:""}.fa-caret-right:before{content:""}.fa-columns:before{content:""}.fa-sort:before,.fa-unsorted:before{content:""}.fa-sort-desc:before,.fa-sort-down:before{content:""}.fa-sort-asc:before,.fa-sort-up:before{content:""}.fa-envelope:before{content:""}.fa-linkedin:before{content:""}.fa-rotate-left:before,.fa-undo:before{content:""}.fa-gavel:before,.fa-legal:before{content:""}.fa-dashboard:before,.fa-tachometer:before{content:""}.fa-comment-o:before{content:""}.fa-comments-o:before{content:""}.fa-bolt:before,.fa-flash:before{content:""}.fa-sitemap:before{content:""}.fa-umbrella:before{content:""}.fa-clipboard:before,.fa-paste:before{content:""}.fa-lightbulb-o:before{content:""}.fa-exchange:before{content:""}.fa-cloud-download:before{content:""}.fa-cloud-upload:before{content:""}.fa-user-md:before{content:""}.fa-stethoscope:before{content:""}.fa-suitcase:before{content:""}.fa-bell-o:before{content:""}.fa-coffee:before{content:""}.fa-cutlery:before{content:""}.fa-file-text-o:before{content:""}.fa-building-o:before{content:""}.fa-hospital-o:before{content:""}.fa-ambulance:before{content:""}.fa-medkit:before{content:""}.fa-fighter-jet:before{content:""}.fa-beer:before{content:""}.fa-h-square:before{content:""}.fa-plus-square:before{content:""}.fa-angle-double-left:before{content:""}.fa-angle-double-right:before{content:""}.fa-angle-double-up:before{content:""}.fa-angle-double-down:before{content:""}.fa-angle-left:before{content:""}.fa-angle-right:before{content:""}.fa-angle-up:before{content:""}.fa-angle-down:before{content:""}.fa-desktop:before{content:""}.fa-laptop:before{content:""}.fa-tablet:before{content:""}.fa-mobile-phone:before,.fa-mobile:before{content:""}.fa-circle-o:before{content:""}.fa-quote-left:before{content:""}.fa-quote-right:before{content:""}.fa-spinner:before{content:""}.fa-circle:before{content:""}.fa-mail-reply:before,.fa-reply:before{content:""}.fa-github-alt:before{content:""}.fa-folder-o:before{content:""}.fa-folder-open-o:before{content:""}.fa-smile-o:before{content:""}.fa-frown-o:before{content:""}.fa-meh-o:before{content:""}.fa-gamepad:before{content:""}.fa-keyboard-o:before{content:""}.fa-flag-o:before{content:""}.fa-flag-checkered:before{content:""}.fa-terminal:before{content:""}.fa-code:before{content:""}.fa-mail-reply-all:before,.fa-reply-all:before{content:""}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:""}.fa-location-arrow:before{content:""}.fa-crop:before{content:""}.fa-code-fork:before{content:""}.fa-chain-broken:before,.fa-unlink:before{content:""}.fa-question:before{content:""}.fa-info:before{content:""}.fa-exclamation:before{content:""}.fa-superscript:before{content:""}.fa-subscript:before{content:""}.fa-eraser:before{content:""}.fa-puzzle-piece:before{content:""}.fa-microphone:before{content:""}.fa-microphone-slash:before{content:""}.fa-shield:before{content:""}.fa-calendar-o:before{content:""}.fa-fire-extinguisher:before{content:""}.fa-rocket:before{content:""}.fa-maxcdn:before{content:""}.fa-chevron-circle-left:before{content:""}.fa-chevron-circle-right:before{content:""}.fa-chevron-circle-up:before{content:""}.fa-chevron-circle-down:before{content:""}.fa-html5:before{content:""}.fa-css3:before{content:""}.fa-anchor:before{content:""}.fa-unlock-alt:before{content:""}.fa-bullseye:before{content:""}.fa-ellipsis-h:before{content:""}.fa-ellipsis-v:before{content:""}.fa-rss-square:before{content:""}.fa-play-circle:before{content:""}.fa-ticket:before{content:""}.fa-minus-square:before{content:""}.fa-minus-square-o:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before{content:""}.fa-level-up:before{content:""}.fa-level-down:before{content:""}.fa-check-square:before{content:""}.fa-pencil-square:before{content:""}.fa-external-link-square:before{content:""}.fa-share-square:before{content:""}.fa-compass:before{content:""}.fa-caret-square-o-down:before,.fa-toggle-down:before{content:""}.fa-caret-square-o-up:before,.fa-toggle-up:before{content:""}.fa-caret-square-o-right:before,.fa-toggle-right:before{content:""}.fa-eur:before,.fa-euro:before{content:""}.fa-gbp:before{content:""}.fa-dollar:before,.fa-usd:before{content:""}.fa-inr:before,.fa-rupee:before{content:""}.fa-cny:before,.fa-jpy:before,.fa-rmb:before,.fa-yen:before{content:""}.fa-rouble:before,.fa-rub:before,.fa-ruble:before{content:""}.fa-krw:before,.fa-won:before{content:""}.fa-bitcoin:before,.fa-btc:before{content:""}.fa-file:before{content:""}.fa-file-text:before{content:""}.fa-sort-alpha-asc:before{content:""}.fa-sort-alpha-desc:before{content:""}.fa-sort-amount-asc:before{content:""}.fa-sort-amount-desc:before{content:""}.fa-sort-numeric-asc:before{content:""}.fa-sort-numeric-desc:before{content:""}.fa-thumbs-up:before{content:""}.fa-thumbs-down:before{content:""}.fa-youtube-square:before{content:""}.fa-youtube:before{content:""}.fa-xing:before{content:""}.fa-xing-square:before{content:""}.fa-youtube-play:before{content:""}.fa-dropbox:before{content:""}.fa-stack-overflow:before{content:""}.fa-instagram:before{content:""}.fa-flickr:before{content:""}.fa-adn:before{content:""}.fa-bitbucket:before,.icon-bitbucket:before{content:""}.fa-bitbucket-square:before{content:""}.fa-tumblr:before{content:""}.fa-tumblr-square:before{content:""}.fa-long-arrow-down:before{content:""}.fa-long-arrow-up:before{content:""}.fa-long-arrow-left:before{content:""}.fa-long-arrow-right:before{content:""}.fa-apple:before{content:""}.fa-windows:before{content:""}.fa-android:before{content:""}.fa-linux:before{content:""}.fa-dribbble:before{content:""}.fa-skype:before{content:""}.fa-foursquare:before{content:""}.fa-trello:before{content:""}.fa-female:before{content:""}.fa-male:before{content:""}.fa-gittip:before,.fa-gratipay:before{content:""}.fa-sun-o:before{content:""}.fa-moon-o:before{content:""}.fa-archive:before{content:""}.fa-bug:before{content:""}.fa-vk:before{content:""}.fa-weibo:before{content:""}.fa-renren:before{content:""}.fa-pagelines:before{content:""}.fa-stack-exchange:before{content:""}.fa-arrow-circle-o-right:before{content:""}.fa-arrow-circle-o-left:before{content:""}.fa-caret-square-o-left:before,.fa-toggle-left:before{content:""}.fa-dot-circle-o:before{content:""}.fa-wheelchair:before{content:""}.fa-vimeo-square:before{content:""}.fa-try:before,.fa-turkish-lira:before{content:""}.fa-plus-square-o:before,.wy-menu-vertical li span.toctree-expand:before{content:""}.fa-space-shuttle:before{content:""}.fa-slack:before{content:""}.fa-envelope-square:before{content:""}.fa-wordpress:before{content:""}.fa-openid:before{content:""}.fa-bank:before,.fa-institution:before,.fa-university:before{content:""}.fa-graduation-cap:before,.fa-mortar-board:before{content:""}.fa-yahoo:before{content:""}.fa-google:before{content:""}.fa-reddit:before{content:""}.fa-reddit-square:before{content:""}.fa-stumbleupon-circle:before{content:""}.fa-stumbleupon:before{content:""}.fa-delicious:before{content:""}.fa-digg:before{content:""}.fa-pied-piper-pp:before{content:""}.fa-pied-piper-alt:before{content:""}.fa-drupal:before{content:""}.fa-joomla:before{content:""}.fa-language:before{content:""}.fa-fax:before{content:""}.fa-building:before{content:""}.fa-child:before{content:""}.fa-paw:before{content:""}.fa-spoon:before{content:""}.fa-cube:before{content:""}.fa-cubes:before{content:""}.fa-behance:before{content:""}.fa-behance-square:before{content:""}.fa-steam:before{content:""}.fa-steam-square:before{content:""}.fa-recycle:before{content:""}.fa-automobile:before,.fa-car:before{content:""}.fa-cab:before,.fa-taxi:before{content:""}.fa-tree:before{content:""}.fa-spotify:before{content:""}.fa-deviantart:before{content:""}.fa-soundcloud:before{content:""}.fa-database:before{content:""}.fa-file-pdf-o:before{content:""}.fa-file-word-o:before{content:""}.fa-file-excel-o:before{content:""}.fa-file-powerpoint-o:before{content:""}.fa-file-image-o:before,.fa-file-photo-o:before,.fa-file-picture-o:before{content:""}.fa-file-archive-o:before,.fa-file-zip-o:before{content:""}.fa-file-audio-o:before,.fa-file-sound-o:before{content:""}.fa-file-movie-o:before,.fa-file-video-o:before{content:""}.fa-file-code-o:before{content:""}.fa-vine:before{content:""}.fa-codepen:before{content:""}.fa-jsfiddle:before{content:""}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-ring:before,.fa-life-saver:before,.fa-support:before{content:""}.fa-circle-o-notch:before{content:""}.fa-ra:before,.fa-rebel:before,.fa-resistance:before{content:""}.fa-empire:before,.fa-ge:before{content:""}.fa-git-square:before{content:""}.fa-git:before{content:""}.fa-hacker-news:before,.fa-y-combinator-square:before,.fa-yc-square:before{content:""}.fa-tencent-weibo:before{content:""}.fa-qq:before{content:""}.fa-wechat:before,.fa-weixin:before{content:""}.fa-paper-plane:before,.fa-send:before{content:""}.fa-paper-plane-o:before,.fa-send-o:before{content:""}.fa-history:before{content:""}.fa-circle-thin:before{content:""}.fa-header:before{content:""}.fa-paragraph:before{content:""}.fa-sliders:before{content:""}.fa-share-alt:before{content:""}.fa-share-alt-square:before{content:""}.fa-bomb:before{content:""}.fa-futbol-o:before,.fa-soccer-ball-o:before{content:""}.fa-tty:before{content:""}.fa-binoculars:before{content:""}.fa-plug:before{content:""}.fa-slideshare:before{content:""}.fa-twitch:before{content:""}.fa-yelp:before{content:""}.fa-newspaper-o:before{content:""}.fa-wifi:before{content:""}.fa-calculator:before{content:""}.fa-paypal:before{content:""}.fa-google-wallet:before{content:""}.fa-cc-visa:before{content:""}.fa-cc-mastercard:before{content:""}.fa-cc-discover:before{content:""}.fa-cc-amex:before{content:""}.fa-cc-paypal:before{content:""}.fa-cc-stripe:before{content:""}.fa-bell-slash:before{content:""}.fa-bell-slash-o:before{content:""}.fa-trash:before{content:""}.fa-copyright:before{content:""}.fa-at:before{content:""}.fa-eyedropper:before{content:""}.fa-paint-brush:before{content:""}.fa-birthday-cake:before{content:""}.fa-area-chart:before{content:""}.fa-pie-chart:before{content:""}.fa-line-chart:before{content:""}.fa-lastfm:before{content:""}.fa-lastfm-square:before{content:""}.fa-toggle-off:before{content:""}.fa-toggle-on:before{content:""}.fa-bicycle:before{content:""}.fa-bus:before{content:""}.fa-ioxhost:before{content:""}.fa-angellist:before{content:""}.fa-cc:before{content:""}.fa-ils:before,.fa-shekel:before,.fa-sheqel:before{content:""}.fa-meanpath:before{content:""}.fa-buysellads:before{content:""}.fa-connectdevelop:before{content:""}.fa-dashcube:before{content:""}.fa-forumbee:before{content:""}.fa-leanpub:before{content:""}.fa-sellsy:before{content:""}.fa-shirtsinbulk:before{content:""}.fa-simplybuilt:before{content:""}.fa-skyatlas:before{content:""}.fa-cart-plus:before{content:""}.fa-cart-arrow-down:before{content:""}.fa-diamond:before{content:""}.fa-ship:before{content:""}.fa-user-secret:before{content:""}.fa-motorcycle:before{content:""}.fa-street-view:before{content:""}.fa-heartbeat:before{content:""}.fa-venus:before{content:""}.fa-mars:before{content:""}.fa-mercury:before{content:""}.fa-intersex:before,.fa-transgender:before{content:""}.fa-transgender-alt:before{content:""}.fa-venus-double:before{content:""}.fa-mars-double:before{content:""}.fa-venus-mars:before{content:""}.fa-mars-stroke:before{content:""}.fa-mars-stroke-v:before{content:""}.fa-mars-stroke-h:before{content:""}.fa-neuter:before{content:""}.fa-genderless:before{content:""}.fa-facebook-official:before{content:""}.fa-pinterest-p:before{content:""}.fa-whatsapp:before{content:""}.fa-server:before{content:""}.fa-user-plus:before{content:""}.fa-user-times:before{content:""}.fa-bed:before,.fa-hotel:before{content:""}.fa-viacoin:before{content:""}.fa-train:before{content:""}.fa-subway:before{content:""}.fa-medium:before{content:""}.fa-y-combinator:before,.fa-yc:before{content:""}.fa-optin-monster:before{content:""}.fa-opencart:before{content:""}.fa-expeditedssl:before{content:""}.fa-battery-4:before,.fa-battery-full:before,.fa-battery:before{content:""}.fa-battery-3:before,.fa-battery-three-quarters:before{content:""}.fa-battery-2:before,.fa-battery-half:before{content:""}.fa-battery-1:before,.fa-battery-quarter:before{content:""}.fa-battery-0:before,.fa-battery-empty:before{content:""}.fa-mouse-pointer:before{content:""}.fa-i-cursor:before{content:""}.fa-object-group:before{content:""}.fa-object-ungroup:before{content:""}.fa-sticky-note:before{content:""}.fa-sticky-note-o:before{content:""}.fa-cc-jcb:before{content:""}.fa-cc-diners-club:before{content:""}.fa-clone:before{content:""}.fa-balance-scale:before{content:""}.fa-hourglass-o:before{content:""}.fa-hourglass-1:before,.fa-hourglass-start:before{content:""}.fa-hourglass-2:before,.fa-hourglass-half:before{content:""}.fa-hourglass-3:before,.fa-hourglass-end:before{content:""}.fa-hourglass:before{content:""}.fa-hand-grab-o:before,.fa-hand-rock-o:before{content:""}.fa-hand-paper-o:before,.fa-hand-stop-o:before{content:""}.fa-hand-scissors-o:before{content:""}.fa-hand-lizard-o:before{content:""}.fa-hand-spock-o:before{content:""}.fa-hand-pointer-o:before{content:""}.fa-hand-peace-o:before{content:""}.fa-trademark:before{content:""}.fa-registered:before{content:""}.fa-creative-commons:before{content:""}.fa-gg:before{content:""}.fa-gg-circle:before{content:""}.fa-tripadvisor:before{content:""}.fa-odnoklassniki:before{content:""}.fa-odnoklassniki-square:before{content:""}.fa-get-pocket:before{content:""}.fa-wikipedia-w:before{content:""}.fa-safari:before{content:""}.fa-chrome:before{content:""}.fa-firefox:before{content:""}.fa-opera:before{content:""}.fa-internet-explorer:before{content:""}.fa-television:before,.fa-tv:before{content:""}.fa-contao:before{content:""}.fa-500px:before{content:""}.fa-amazon:before{content:""}.fa-calendar-plus-o:before{content:""}.fa-calendar-minus-o:before{content:""}.fa-calendar-times-o:before{content:""}.fa-calendar-check-o:before{content:""}.fa-industry:before{content:""}.fa-map-pin:before{content:""}.fa-map-signs:before{content:""}.fa-map-o:before{content:""}.fa-map:before{content:""}.fa-commenting:before{content:""}.fa-commenting-o:before{content:""}.fa-houzz:before{content:""}.fa-vimeo:before{content:""}.fa-black-tie:before{content:""}.fa-fonticons:before{content:""}.fa-reddit-alien:before{content:""}.fa-edge:before{content:""}.fa-credit-card-alt:before{content:""}.fa-codiepie:before{content:""}.fa-modx:before{content:""}.fa-fort-awesome:before{content:""}.fa-usb:before{content:""}.fa-product-hunt:before{content:""}.fa-mixcloud:before{content:""}.fa-scribd:before{content:""}.fa-pause-circle:before{content:""}.fa-pause-circle-o:before{content:""}.fa-stop-circle:before{content:""}.fa-stop-circle-o:before{content:""}.fa-shopping-bag:before{content:""}.fa-shopping-basket:before{content:""}.fa-hashtag:before{content:""}.fa-bluetooth:before{content:""}.fa-bluetooth-b:before{content:""}.fa-percent:before{content:""}.fa-gitlab:before,.icon-gitlab:before{content:""}.fa-wpbeginner:before{content:""}.fa-wpforms:before{content:""}.fa-envira:before{content:""}.fa-universal-access:before{content:""}.fa-wheelchair-alt:before{content:""}.fa-question-circle-o:before{content:""}.fa-blind:before{content:""}.fa-audio-description:before{content:""}.fa-volume-control-phone:before{content:""}.fa-braille:before{content:""}.fa-assistive-listening-systems:before{content:""}.fa-american-sign-language-interpreting:before,.fa-asl-interpreting:before{content:""}.fa-deaf:before,.fa-deafness:before,.fa-hard-of-hearing:before{content:""}.fa-glide:before{content:""}.fa-glide-g:before{content:""}.fa-sign-language:before,.fa-signing:before{content:""}.fa-low-vision:before{content:""}.fa-viadeo:before{content:""}.fa-viadeo-square:before{content:""}.fa-snapchat:before{content:""}.fa-snapchat-ghost:before{content:""}.fa-snapchat-square:before{content:""}.fa-pied-piper:before{content:""}.fa-first-order:before{content:""}.fa-yoast:before{content:""}.fa-themeisle:before{content:""}.fa-google-plus-circle:before,.fa-google-plus-official:before{content:""}.fa-fa:before,.fa-font-awesome:before{content:""}.fa-handshake-o:before{content:""}.fa-envelope-open:before{content:""}.fa-envelope-open-o:before{content:""}.fa-linode:before{content:""}.fa-address-book:before{content:""}.fa-address-book-o:before{content:""}.fa-address-card:before,.fa-vcard:before{content:""}.fa-address-card-o:before,.fa-vcard-o:before{content:""}.fa-user-circle:before{content:""}.fa-user-circle-o:before{content:""}.fa-user-o:before{content:""}.fa-id-badge:before{content:""}.fa-drivers-license:before,.fa-id-card:before{content:""}.fa-drivers-license-o:before,.fa-id-card-o:before{content:""}.fa-quora:before{content:""}.fa-free-code-camp:before{content:""}.fa-telegram:before{content:""}.fa-thermometer-4:before,.fa-thermometer-full:before,.fa-thermometer:before{content:""}.fa-thermometer-3:before,.fa-thermometer-three-quarters:before{content:""}.fa-thermometer-2:before,.fa-thermometer-half:before{content:""}.fa-thermometer-1:before,.fa-thermometer-quarter:before{content:""}.fa-thermometer-0:before,.fa-thermometer-empty:before{content:""}.fa-shower:before{content:""}.fa-bath:before,.fa-bathtub:before,.fa-s15:before{content:""}.fa-podcast:before{content:""}.fa-window-maximize:before{content:""}.fa-window-minimize:before{content:""}.fa-window-restore:before{content:""}.fa-times-rectangle:before,.fa-window-close:before{content:""}.fa-times-rectangle-o:before,.fa-window-close-o:before{content:""}.fa-bandcamp:before{content:""}.fa-grav:before{content:""}.fa-etsy:before{content:""}.fa-imdb:before{content:""}.fa-ravelry:before{content:""}.fa-eercast:before{content:""}.fa-microchip:before{content:""}.fa-snowflake-o:before{content:""}.fa-superpowers:before{content:""}.fa-wpexplorer:before{content:""}.fa-meetup:before{content:""}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-dropdown .caret,.wy-inline-validate.wy-inline-validate-danger .wy-input-context,.wy-inline-validate.wy-inline-validate-info .wy-input-context,.wy-inline-validate.wy-inline-validate-success .wy-input-context,.wy-inline-validate.wy-inline-validate-warning .wy-input-context,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{font-family:inherit}.fa:before,.icon:before,.rst-content .admonition-title:before,.rst-content .code-block-caption .headerlink:before,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before{font-family:FontAwesome;display:inline-block;font-style:normal;font-weight:400;line-height:1;text-decoration:inherit}.rst-content .code-block-caption a .headerlink,.rst-content a .admonition-title,.rst-content code.download a span:first-child,.rst-content dl dt a .headerlink,.rst-content h1 a .headerlink,.rst-content h2 a .headerlink,.rst-content h3 a .headerlink,.rst-content h4 a .headerlink,.rst-content h5 a .headerlink,.rst-content h6 a .headerlink,.rst-content p.caption a .headerlink,.rst-content table>caption a .headerlink,.rst-content tt.download a span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li a span.toctree-expand,a .fa,a .icon,a .rst-content .admonition-title,a .rst-content .code-block-caption .headerlink,a .rst-content code.download span:first-child,a .rst-content dl dt .headerlink,a .rst-content h1 .headerlink,a .rst-content h2 .headerlink,a .rst-content h3 .headerlink,a .rst-content h4 .headerlink,a .rst-content h5 .headerlink,a .rst-content h6 .headerlink,a .rst-content p.caption .headerlink,a .rst-content table>caption .headerlink,a .rst-content tt.download span:first-child,a .wy-menu-vertical li span.toctree-expand{display:inline-block;text-decoration:inherit}.btn .fa,.btn .icon,.btn .rst-content .admonition-title,.btn .rst-content .code-block-caption .headerlink,.btn .rst-content code.download span:first-child,.btn .rst-content dl dt .headerlink,.btn .rst-content h1 .headerlink,.btn .rst-content h2 .headerlink,.btn .rst-content h3 .headerlink,.btn .rst-content h4 .headerlink,.btn .rst-content h5 .headerlink,.btn .rst-content h6 .headerlink,.btn .rst-content p.caption .headerlink,.btn .rst-content table>caption .headerlink,.btn .rst-content tt.download span:first-child,.btn .wy-menu-vertical li.current>a span.toctree-expand,.btn .wy-menu-vertical li.on a span.toctree-expand,.btn .wy-menu-vertical li span.toctree-expand,.nav .fa,.nav .icon,.nav .rst-content .admonition-title,.nav .rst-content .code-block-caption .headerlink,.nav .rst-content code.download span:first-child,.nav .rst-content dl dt .headerlink,.nav .rst-content h1 .headerlink,.nav .rst-content h2 .headerlink,.nav .rst-content h3 .headerlink,.nav .rst-content h4 .headerlink,.nav .rst-content h5 .headerlink,.nav .rst-content h6 .headerlink,.nav .rst-content p.caption .headerlink,.nav .rst-content table>caption .headerlink,.nav .rst-content tt.download span:first-child,.nav .wy-menu-vertical li.current>a span.toctree-expand,.nav .wy-menu-vertical li.on a span.toctree-expand,.nav .wy-menu-vertical li span.toctree-expand,.rst-content .btn .admonition-title,.rst-content .code-block-caption .btn .headerlink,.rst-content .code-block-caption .nav .headerlink,.rst-content .nav .admonition-title,.rst-content code.download .btn span:first-child,.rst-content code.download .nav span:first-child,.rst-content dl dt .btn .headerlink,.rst-content dl dt .nav .headerlink,.rst-content h1 .btn .headerlink,.rst-content h1 .nav .headerlink,.rst-content h2 .btn .headerlink,.rst-content h2 .nav .headerlink,.rst-content h3 .btn .headerlink,.rst-content h3 .nav .headerlink,.rst-content h4 .btn .headerlink,.rst-content h4 .nav .headerlink,.rst-content h5 .btn .headerlink,.rst-content h5 .nav .headerlink,.rst-content h6 .btn .headerlink,.rst-content h6 .nav .headerlink,.rst-content p.caption .btn .headerlink,.rst-content p.caption .nav .headerlink,.rst-content table>caption .btn .headerlink,.rst-content table>caption .nav .headerlink,.rst-content tt.download .btn span:first-child,.rst-content tt.download .nav span:first-child,.wy-menu-vertical li .btn span.toctree-expand,.wy-menu-vertical li.current>a .btn span.toctree-expand,.wy-menu-vertical li.current>a .nav span.toctree-expand,.wy-menu-vertical li .nav span.toctree-expand,.wy-menu-vertical li.on a .btn span.toctree-expand,.wy-menu-vertical li.on a .nav span.toctree-expand{display:inline}.btn .fa-large.icon,.btn .fa.fa-large,.btn .rst-content .code-block-caption .fa-large.headerlink,.btn .rst-content .fa-large.admonition-title,.btn .rst-content code.download span.fa-large:first-child,.btn .rst-content dl dt .fa-large.headerlink,.btn .rst-content h1 .fa-large.headerlink,.btn .rst-content h2 .fa-large.headerlink,.btn .rst-content h3 .fa-large.headerlink,.btn .rst-content h4 .fa-large.headerlink,.btn .rst-content h5 .fa-large.headerlink,.btn .rst-content h6 .fa-large.headerlink,.btn .rst-content p.caption .fa-large.headerlink,.btn .rst-content table>caption .fa-large.headerlink,.btn .rst-content tt.download span.fa-large:first-child,.btn .wy-menu-vertical li span.fa-large.toctree-expand,.nav .fa-large.icon,.nav .fa.fa-large,.nav .rst-content .code-block-caption .fa-large.headerlink,.nav .rst-content .fa-large.admonition-title,.nav .rst-content code.download span.fa-large:first-child,.nav .rst-content dl dt .fa-large.headerlink,.nav .rst-content h1 .fa-large.headerlink,.nav .rst-content h2 .fa-large.headerlink,.nav .rst-content h3 .fa-large.headerlink,.nav .rst-content h4 .fa-large.headerlink,.nav .rst-content h5 .fa-large.headerlink,.nav .rst-content h6 .fa-large.headerlink,.nav .rst-content p.caption .fa-large.headerlink,.nav .rst-content table>caption .fa-large.headerlink,.nav .rst-content tt.download span.fa-large:first-child,.nav .wy-menu-vertical li span.fa-large.toctree-expand,.rst-content .btn .fa-large.admonition-title,.rst-content .code-block-caption .btn .fa-large.headerlink,.rst-content .code-block-caption .nav .fa-large.headerlink,.rst-content .nav .fa-large.admonition-title,.rst-content code.download .btn span.fa-large:first-child,.rst-content code.download .nav span.fa-large:first-child,.rst-content dl dt .btn .fa-large.headerlink,.rst-content dl dt .nav .fa-large.headerlink,.rst-content h1 .btn .fa-large.headerlink,.rst-content h1 .nav .fa-large.headerlink,.rst-content h2 .btn .fa-large.headerlink,.rst-content h2 .nav .fa-large.headerlink,.rst-content h3 .btn .fa-large.headerlink,.rst-content h3 .nav .fa-large.headerlink,.rst-content h4 .btn .fa-large.headerlink,.rst-content h4 .nav .fa-large.headerlink,.rst-content h5 .btn .fa-large.headerlink,.rst-content h5 .nav .fa-large.headerlink,.rst-content h6 .btn .fa-large.headerlink,.rst-content h6 .nav .fa-large.headerlink,.rst-content p.caption .btn .fa-large.headerlink,.rst-content p.caption .nav .fa-large.headerlink,.rst-content table>caption .btn .fa-large.headerlink,.rst-content table>caption .nav .fa-large.headerlink,.rst-content tt.download .btn span.fa-large:first-child,.rst-content tt.download .nav span.fa-large:first-child,.wy-menu-vertical li .btn span.fa-large.toctree-expand,.wy-menu-vertical li .nav span.fa-large.toctree-expand{line-height:.9em}.btn .fa-spin.icon,.btn .fa.fa-spin,.btn .rst-content .code-block-caption .fa-spin.headerlink,.btn .rst-content .fa-spin.admonition-title,.btn .rst-content code.download span.fa-spin:first-child,.btn .rst-content dl dt .fa-spin.headerlink,.btn .rst-content h1 .fa-spin.headerlink,.btn .rst-content h2 .fa-spin.headerlink,.btn .rst-content h3 .fa-spin.headerlink,.btn .rst-content h4 .fa-spin.headerlink,.btn .rst-content h5 .fa-spin.headerlink,.btn .rst-content h6 .fa-spin.headerlink,.btn .rst-content p.caption .fa-spin.headerlink,.btn .rst-content table>caption .fa-spin.headerlink,.btn .rst-content tt.download span.fa-spin:first-child,.btn .wy-menu-vertical li span.fa-spin.toctree-expand,.nav .fa-spin.icon,.nav .fa.fa-spin,.nav .rst-content .code-block-caption .fa-spin.headerlink,.nav .rst-content .fa-spin.admonition-title,.nav .rst-content code.download span.fa-spin:first-child,.nav .rst-content dl dt .fa-spin.headerlink,.nav .rst-content h1 .fa-spin.headerlink,.nav .rst-content h2 .fa-spin.headerlink,.nav .rst-content h3 .fa-spin.headerlink,.nav .rst-content h4 .fa-spin.headerlink,.nav .rst-content h5 .fa-spin.headerlink,.nav .rst-content h6 .fa-spin.headerlink,.nav .rst-content p.caption .fa-spin.headerlink,.nav .rst-content table>caption .fa-spin.headerlink,.nav .rst-content tt.download span.fa-spin:first-child,.nav .wy-menu-vertical li span.fa-spin.toctree-expand,.rst-content .btn .fa-spin.admonition-title,.rst-content .code-block-caption .btn .fa-spin.headerlink,.rst-content .code-block-caption .nav .fa-spin.headerlink,.rst-content .nav .fa-spin.admonition-title,.rst-content code.download .btn span.fa-spin:first-child,.rst-content code.download .nav span.fa-spin:first-child,.rst-content dl dt .btn .fa-spin.headerlink,.rst-content dl dt .nav .fa-spin.headerlink,.rst-content h1 .btn .fa-spin.headerlink,.rst-content h1 .nav .fa-spin.headerlink,.rst-content h2 .btn .fa-spin.headerlink,.rst-content h2 .nav .fa-spin.headerlink,.rst-content h3 .btn .fa-spin.headerlink,.rst-content h3 .nav .fa-spin.headerlink,.rst-content h4 .btn .fa-spin.headerlink,.rst-content h4 .nav .fa-spin.headerlink,.rst-content h5 .btn .fa-spin.headerlink,.rst-content h5 .nav .fa-spin.headerlink,.rst-content h6 .btn .fa-spin.headerlink,.rst-content h6 .nav .fa-spin.headerlink,.rst-content p.caption .btn .fa-spin.headerlink,.rst-content p.caption .nav .fa-spin.headerlink,.rst-content table>caption .btn .fa-spin.headerlink,.rst-content table>caption .nav .fa-spin.headerlink,.rst-content tt.download .btn span.fa-spin:first-child,.rst-content tt.download .nav span.fa-spin:first-child,.wy-menu-vertical li .btn span.fa-spin.toctree-expand,.wy-menu-vertical li .nav span.fa-spin.toctree-expand{display:inline-block}.btn.fa:before,.btn.icon:before,.rst-content .btn.admonition-title:before,.rst-content .code-block-caption .btn.headerlink:before,.rst-content code.download span.btn:first-child:before,.rst-content dl dt .btn.headerlink:before,.rst-content h1 .btn.headerlink:before,.rst-content h2 .btn.headerlink:before,.rst-content h3 .btn.headerlink:before,.rst-content h4 .btn.headerlink:before,.rst-content h5 .btn.headerlink:before,.rst-content h6 .btn.headerlink:before,.rst-content p.caption .btn.headerlink:before,.rst-content table>caption .btn.headerlink:before,.rst-content tt.download span.btn:first-child:before,.wy-menu-vertical li span.btn.toctree-expand:before{opacity:.5;-webkit-transition:opacity .05s ease-in;-moz-transition:opacity .05s ease-in;transition:opacity .05s ease-in}.btn.fa:hover:before,.btn.icon:hover:before,.rst-content .btn.admonition-title:hover:before,.rst-content .code-block-caption .btn.headerlink:hover:before,.rst-content code.download span.btn:first-child:hover:before,.rst-content dl dt .btn.headerlink:hover:before,.rst-content h1 .btn.headerlink:hover:before,.rst-content h2 .btn.headerlink:hover:before,.rst-content h3 .btn.headerlink:hover:before,.rst-content h4 .btn.headerlink:hover:before,.rst-content h5 .btn.headerlink:hover:before,.rst-content h6 .btn.headerlink:hover:before,.rst-content p.caption .btn.headerlink:hover:before,.rst-content table>caption .btn.headerlink:hover:before,.rst-content tt.download span.btn:first-child:hover:before,.wy-menu-vertical li span.btn.toctree-expand:hover:before{opacity:1}.btn-mini .fa:before,.btn-mini .icon:before,.btn-mini .rst-content .admonition-title:before,.btn-mini .rst-content .code-block-caption .headerlink:before,.btn-mini .rst-content code.download span:first-child:before,.btn-mini .rst-content dl dt .headerlink:before,.btn-mini .rst-content h1 .headerlink:before,.btn-mini .rst-content h2 .headerlink:before,.btn-mini .rst-content h3 .headerlink:before,.btn-mini .rst-content h4 .headerlink:before,.btn-mini .rst-content h5 .headerlink:before,.btn-mini .rst-content h6 .headerlink:before,.btn-mini .rst-content p.caption .headerlink:before,.btn-mini .rst-content table>caption .headerlink:before,.btn-mini .rst-content tt.download span:first-child:before,.btn-mini .wy-menu-vertical li span.toctree-expand:before,.rst-content .btn-mini .admonition-title:before,.rst-content .code-block-caption .btn-mini .headerlink:before,.rst-content code.download .btn-mini span:first-child:before,.rst-content dl dt .btn-mini .headerlink:before,.rst-content h1 .btn-mini .headerlink:before,.rst-content h2 .btn-mini .headerlink:before,.rst-content h3 .btn-mini .headerlink:before,.rst-content h4 .btn-mini .headerlink:before,.rst-content h5 .btn-mini .headerlink:before,.rst-content h6 .btn-mini .headerlink:before,.rst-content p.caption .btn-mini .headerlink:before,.rst-content table>caption .btn-mini .headerlink:before,.rst-content tt.download .btn-mini span:first-child:before,.wy-menu-vertical li .btn-mini span.toctree-expand:before{font-size:14px;vertical-align:-15%}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.wy-alert{padding:12px;line-height:24px;margin-bottom:24px;background:#e7f2fa}.rst-content .admonition-title,.wy-alert-title{font-weight:700;display:block;color:#fff;background:#6ab0de;padding:6px 12px;margin:-12px -12px 12px}.rst-content .danger,.rst-content .error,.rst-content .wy-alert-danger.admonition,.rst-content .wy-alert-danger.admonition-todo,.rst-content .wy-alert-danger.attention,.rst-content .wy-alert-danger.caution,.rst-content .wy-alert-danger.hint,.rst-content .wy-alert-danger.important,.rst-content .wy-alert-danger.note,.rst-content .wy-alert-danger.seealso,.rst-content .wy-alert-danger.tip,.rst-content .wy-alert-danger.warning,.wy-alert.wy-alert-danger{background:#fdf3f2}.rst-content .danger .admonition-title,.rst-content .danger .wy-alert-title,.rst-content .error .admonition-title,.rst-content .error .wy-alert-title,.rst-content .wy-alert-danger.admonition-todo .admonition-title,.rst-content .wy-alert-danger.admonition-todo .wy-alert-title,.rst-content .wy-alert-danger.admonition .admonition-title,.rst-content .wy-alert-danger.admonition .wy-alert-title,.rst-content .wy-alert-danger.attention .admonition-title,.rst-content .wy-alert-danger.attention .wy-alert-title,.rst-content .wy-alert-danger.caution .admonition-title,.rst-content .wy-alert-danger.caution .wy-alert-title,.rst-content .wy-alert-danger.hint .admonition-title,.rst-content .wy-alert-danger.hint .wy-alert-title,.rst-content .wy-alert-danger.important .admonition-title,.rst-content .wy-alert-danger.important .wy-alert-title,.rst-content .wy-alert-danger.note .admonition-title,.rst-content .wy-alert-danger.note .wy-alert-title,.rst-content .wy-alert-danger.seealso .admonition-title,.rst-content .wy-alert-danger.seealso .wy-alert-title,.rst-content .wy-alert-danger.tip .admonition-title,.rst-content .wy-alert-danger.tip .wy-alert-title,.rst-content .wy-alert-danger.warning .admonition-title,.rst-content .wy-alert-danger.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-danger .admonition-title,.wy-alert.wy-alert-danger .rst-content .admonition-title,.wy-alert.wy-alert-danger .wy-alert-title{background:#f29f97}.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .warning,.rst-content .wy-alert-warning.admonition,.rst-content .wy-alert-warning.danger,.rst-content .wy-alert-warning.error,.rst-content .wy-alert-warning.hint,.rst-content .wy-alert-warning.important,.rst-content .wy-alert-warning.note,.rst-content .wy-alert-warning.seealso,.rst-content .wy-alert-warning.tip,.wy-alert.wy-alert-warning{background:#ffedcc}.rst-content .admonition-todo .admonition-title,.rst-content .admonition-todo .wy-alert-title,.rst-content .attention .admonition-title,.rst-content .attention .wy-alert-title,.rst-content .caution .admonition-title,.rst-content .caution .wy-alert-title,.rst-content .warning .admonition-title,.rst-content .warning .wy-alert-title,.rst-content .wy-alert-warning.admonition .admonition-title,.rst-content .wy-alert-warning.admonition .wy-alert-title,.rst-content .wy-alert-warning.danger .admonition-title,.rst-content .wy-alert-warning.danger .wy-alert-title,.rst-content .wy-alert-warning.error .admonition-title,.rst-content .wy-alert-warning.error .wy-alert-title,.rst-content .wy-alert-warning.hint .admonition-title,.rst-content .wy-alert-warning.hint .wy-alert-title,.rst-content .wy-alert-warning.important .admonition-title,.rst-content .wy-alert-warning.important .wy-alert-title,.rst-content .wy-alert-warning.note .admonition-title,.rst-content .wy-alert-warning.note .wy-alert-title,.rst-content .wy-alert-warning.seealso .admonition-title,.rst-content .wy-alert-warning.seealso .wy-alert-title,.rst-content .wy-alert-warning.tip .admonition-title,.rst-content .wy-alert-warning.tip .wy-alert-title,.rst-content .wy-alert.wy-alert-warning .admonition-title,.wy-alert.wy-alert-warning .rst-content .admonition-title,.wy-alert.wy-alert-warning .wy-alert-title{background:#f0b37e}.rst-content .note,.rst-content .seealso,.rst-content .wy-alert-info.admonition,.rst-content .wy-alert-info.admonition-todo,.rst-content .wy-alert-info.attention,.rst-content .wy-alert-info.caution,.rst-content .wy-alert-info.danger,.rst-content .wy-alert-info.error,.rst-content .wy-alert-info.hint,.rst-content .wy-alert-info.important,.rst-content .wy-alert-info.tip,.rst-content .wy-alert-info.warning,.wy-alert.wy-alert-info{background:#e7f2fa}.rst-content .note .admonition-title,.rst-content .note .wy-alert-title,.rst-content .seealso .admonition-title,.rst-content .seealso .wy-alert-title,.rst-content .wy-alert-info.admonition-todo .admonition-title,.rst-content .wy-alert-info.admonition-todo .wy-alert-title,.rst-content .wy-alert-info.admonition .admonition-title,.rst-content .wy-alert-info.admonition .wy-alert-title,.rst-content .wy-alert-info.attention .admonition-title,.rst-content .wy-alert-info.attention .wy-alert-title,.rst-content .wy-alert-info.caution .admonition-title,.rst-content .wy-alert-info.caution .wy-alert-title,.rst-content .wy-alert-info.danger .admonition-title,.rst-content .wy-alert-info.danger .wy-alert-title,.rst-content .wy-alert-info.error .admonition-title,.rst-content .wy-alert-info.error .wy-alert-title,.rst-content .wy-alert-info.hint .admonition-title,.rst-content .wy-alert-info.hint .wy-alert-title,.rst-content .wy-alert-info.important .admonition-title,.rst-content .wy-alert-info.important .wy-alert-title,.rst-content .wy-alert-info.tip .admonition-title,.rst-content .wy-alert-info.tip .wy-alert-title,.rst-content .wy-alert-info.warning .admonition-title,.rst-content .wy-alert-info.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-info .admonition-title,.wy-alert.wy-alert-info .rst-content .admonition-title,.wy-alert.wy-alert-info .wy-alert-title{background:#6ab0de}.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .wy-alert-success.admonition,.rst-content .wy-alert-success.admonition-todo,.rst-content .wy-alert-success.attention,.rst-content .wy-alert-success.caution,.rst-content .wy-alert-success.danger,.rst-content .wy-alert-success.error,.rst-content .wy-alert-success.note,.rst-content .wy-alert-success.seealso,.rst-content .wy-alert-success.warning,.wy-alert.wy-alert-success{background:#dbfaf4}.rst-content .hint .admonition-title,.rst-content .hint .wy-alert-title,.rst-content .important .admonition-title,.rst-content .important .wy-alert-title,.rst-content .tip .admonition-title,.rst-content .tip .wy-alert-title,.rst-content .wy-alert-success.admonition-todo .admonition-title,.rst-content .wy-alert-success.admonition-todo .wy-alert-title,.rst-content .wy-alert-success.admonition .admonition-title,.rst-content .wy-alert-success.admonition .wy-alert-title,.rst-content .wy-alert-success.attention .admonition-title,.rst-content .wy-alert-success.attention .wy-alert-title,.rst-content .wy-alert-success.caution .admonition-title,.rst-content .wy-alert-success.caution .wy-alert-title,.rst-content .wy-alert-success.danger .admonition-title,.rst-content .wy-alert-success.danger .wy-alert-title,.rst-content .wy-alert-success.error .admonition-title,.rst-content .wy-alert-success.error .wy-alert-title,.rst-content .wy-alert-success.note .admonition-title,.rst-content .wy-alert-success.note .wy-alert-title,.rst-content .wy-alert-success.seealso .admonition-title,.rst-content .wy-alert-success.seealso .wy-alert-title,.rst-content .wy-alert-success.warning .admonition-title,.rst-content .wy-alert-success.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-success .admonition-title,.wy-alert.wy-alert-success .rst-content .admonition-title,.wy-alert.wy-alert-success .wy-alert-title{background:#1abc9c}.rst-content .wy-alert-neutral.admonition,.rst-content .wy-alert-neutral.admonition-todo,.rst-content .wy-alert-neutral.attention,.rst-content .wy-alert-neutral.caution,.rst-content .wy-alert-neutral.danger,.rst-content .wy-alert-neutral.error,.rst-content .wy-alert-neutral.hint,.rst-content .wy-alert-neutral.important,.rst-content .wy-alert-neutral.note,.rst-content .wy-alert-neutral.seealso,.rst-content .wy-alert-neutral.tip,.rst-content .wy-alert-neutral.warning,.wy-alert.wy-alert-neutral{background:#f3f6f6}.rst-content .wy-alert-neutral.admonition-todo .admonition-title,.rst-content .wy-alert-neutral.admonition-todo .wy-alert-title,.rst-content .wy-alert-neutral.admonition .admonition-title,.rst-content .wy-alert-neutral.admonition .wy-alert-title,.rst-content .wy-alert-neutral.attention .admonition-title,.rst-content .wy-alert-neutral.attention .wy-alert-title,.rst-content .wy-alert-neutral.caution .admonition-title,.rst-content .wy-alert-neutral.caution .wy-alert-title,.rst-content .wy-alert-neutral.danger .admonition-title,.rst-content .wy-alert-neutral.danger .wy-alert-title,.rst-content .wy-alert-neutral.error .admonition-title,.rst-content .wy-alert-neutral.error .wy-alert-title,.rst-content .wy-alert-neutral.hint .admonition-title,.rst-content .wy-alert-neutral.hint .wy-alert-title,.rst-content .wy-alert-neutral.important .admonition-title,.rst-content .wy-alert-neutral.important .wy-alert-title,.rst-content .wy-alert-neutral.note .admonition-title,.rst-content .wy-alert-neutral.note .wy-alert-title,.rst-content .wy-alert-neutral.seealso .admonition-title,.rst-content .wy-alert-neutral.seealso .wy-alert-title,.rst-content .wy-alert-neutral.tip .admonition-title,.rst-content .wy-alert-neutral.tip .wy-alert-title,.rst-content .wy-alert-neutral.warning .admonition-title,.rst-content .wy-alert-neutral.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-neutral .admonition-title,.wy-alert.wy-alert-neutral .rst-content .admonition-title,.wy-alert.wy-alert-neutral .wy-alert-title{color:#404040;background:#e1e4e5}.rst-content .wy-alert-neutral.admonition-todo a,.rst-content .wy-alert-neutral.admonition a,.rst-content .wy-alert-neutral.attention a,.rst-content .wy-alert-neutral.caution a,.rst-content .wy-alert-neutral.danger a,.rst-content .wy-alert-neutral.error a,.rst-content .wy-alert-neutral.hint a,.rst-content .wy-alert-neutral.important a,.rst-content .wy-alert-neutral.note a,.rst-content .wy-alert-neutral.seealso a,.rst-content .wy-alert-neutral.tip a,.rst-content .wy-alert-neutral.warning a,.wy-alert.wy-alert-neutral a{color:#2980b9}.rst-content .admonition-todo p:last-child,.rst-content .admonition p:last-child,.rst-content .attention p:last-child,.rst-content .caution p:last-child,.rst-content .danger p:last-child,.rst-content .error p:last-child,.rst-content .hint p:last-child,.rst-content .important p:last-child,.rst-content .note p:last-child,.rst-content .seealso p:last-child,.rst-content .tip p:last-child,.rst-content .warning p:last-child,.wy-alert p:last-child{margin-bottom:0}.wy-tray-container{position:fixed;bottom:0;left:0;z-index:600}.wy-tray-container li{display:block;width:300px;background:transparent;color:#fff;text-align:center;box-shadow:0 5px 5px 0 rgba(0,0,0,.1);padding:0 24px;min-width:20%;opacity:0;height:0;line-height:56px;overflow:hidden;-webkit-transition:all .3s ease-in;-moz-transition:all .3s ease-in;transition:all .3s ease-in}.wy-tray-container li.wy-tray-item-success{background:#27ae60}.wy-tray-container li.wy-tray-item-info{background:#2980b9}.wy-tray-container li.wy-tray-item-warning{background:#e67e22}.wy-tray-container li.wy-tray-item-danger{background:#e74c3c}.wy-tray-container li.on{opacity:1;height:56px}@media screen and (max-width:768px){.wy-tray-container{bottom:auto;top:0;width:100%}.wy-tray-container li{width:100%}}button{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle;cursor:pointer;line-height:normal;-webkit-appearance:button;*overflow:visible}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}button[disabled]{cursor:default}.btn{display:inline-block;border-radius:2px;line-height:normal;white-space:nowrap;text-align:center;cursor:pointer;font-size:100%;padding:6px 12px 8px;color:#fff;border:1px solid rgba(0,0,0,.1);background-color:#27ae60;text-decoration:none;font-weight:400;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 2px -1px hsla(0,0%,100%,.5),inset 0 -2px 0 0 rgba(0,0,0,.1);outline-none:false;vertical-align:middle;*display:inline;zoom:1;-webkit-user-drag:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transition:all .1s linear;-moz-transition:all .1s linear;transition:all .1s linear}.btn-hover{background:#2e8ece;color:#fff}.btn:hover{background:#2cc36b;color:#fff}.btn:focus{background:#2cc36b;outline:0}.btn:active{box-shadow:inset 0 -1px 0 0 rgba(0,0,0,.05),inset 0 2px 0 0 rgba(0,0,0,.1);padding:8px 12px 6px}.btn:visited{color:#fff}.btn-disabled,.btn-disabled:active,.btn-disabled:focus,.btn-disabled:hover,.btn:disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:.4;cursor:not-allowed;box-shadow:none}.btn::-moz-focus-inner{padding:0;border:0}.btn-small{font-size:80%}.btn-info{background-color:#2980b9!important}.btn-info:hover{background-color:#2e8ece!important}.btn-neutral{background-color:#f3f6f6!important;color:#404040!important}.btn-neutral:hover{background-color:#e5ebeb!important;color:#404040}.btn-neutral:visited{color:#404040!important}.btn-success{background-color:#27ae60!important}.btn-success:hover{background-color:#295!important}.btn-danger{background-color:#e74c3c!important}.btn-danger:hover{background-color:#ea6153!important}.btn-warning{background-color:#e67e22!important}.btn-warning:hover{background-color:#e98b39!important}.btn-invert{background-color:#222}.btn-invert:hover{background-color:#2f2f2f!important}.btn-link{background-color:transparent!important;color:#2980b9;box-shadow:none;border-color:transparent!important}.btn-link:active,.btn-link:hover{background-color:transparent!important;color:#409ad5!important;box-shadow:none}.btn-link:visited{color:#9b59b6}.wy-btn-group .btn,.wy-control .btn{vertical-align:middle}.wy-btn-group{margin-bottom:24px;*zoom:1}.wy-btn-group:after,.wy-btn-group:before{display:table;content:""}.wy-btn-group:after{clear:both}.wy-dropdown{position:relative;display:inline-block}.wy-dropdown-active .wy-dropdown-menu{display:block}.wy-dropdown-menu{position:absolute;left:0;display:none;float:left;top:100%;min-width:100%;background:#fcfcfc;z-index:100;border:1px solid #cfd7dd;box-shadow:0 2px 2px 0 rgba(0,0,0,.1);padding:12px}.wy-dropdown-menu>dd>a{display:block;clear:both;color:#404040;white-space:nowrap;font-size:90%;padding:0 12px;cursor:pointer}.wy-dropdown-menu>dd>a:hover{background:#2980b9;color:#fff}.wy-dropdown-menu>dd.divider{border-top:1px solid #cfd7dd;margin:6px 0}.wy-dropdown-menu>dd.search{padding-bottom:12px}.wy-dropdown-menu>dd.search input[type=search]{width:100%}.wy-dropdown-menu>dd.call-to-action{background:#e3e3e3;text-transform:uppercase;font-weight:500;font-size:80%}.wy-dropdown-menu>dd.call-to-action:hover{background:#e3e3e3}.wy-dropdown-menu>dd.call-to-action .btn{color:#fff}.wy-dropdown.wy-dropdown-up .wy-dropdown-menu{bottom:100%;top:auto;left:auto;right:0}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu{background:#fcfcfc;margin-top:2px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a{padding:6px 12px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a:hover{background:#2980b9;color:#fff}.wy-dropdown.wy-dropdown-left .wy-dropdown-menu{right:0;left:auto;text-align:right}.wy-dropdown-arrow:before{content:" ";border-bottom:5px solid #f5f5f5;border-left:5px solid transparent;border-right:5px solid transparent;position:absolute;display:block;top:-4px;left:50%;margin-left:-3px}.wy-dropdown-arrow.wy-dropdown-arrow-left:before{left:11px}.wy-form-stacked select{display:block}.wy-form-aligned .wy-help-inline,.wy-form-aligned input,.wy-form-aligned label,.wy-form-aligned select,.wy-form-aligned textarea{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-form-aligned .wy-control-group>label{display:inline-block;vertical-align:middle;width:10em;margin:6px 12px 0 0;float:left}.wy-form-aligned .wy-control{float:left}.wy-form-aligned .wy-control label{display:block}.wy-form-aligned .wy-control select{margin-top:6px}fieldset{margin:0}fieldset,legend{border:0;padding:0}legend{width:100%;white-space:normal;margin-bottom:24px;font-size:150%;*margin-left:-7px}label,legend{display:block}label{margin:0 0 .3125em;color:#333;font-size:90%}input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}.wy-control-group{margin-bottom:24px;max-width:1200px;margin-left:auto;margin-right:auto;*zoom:1}.wy-control-group:after,.wy-control-group:before{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group.wy-control-group-required>label:after{content:" *";color:#e74c3c}.wy-control-group .wy-form-full,.wy-control-group .wy-form-halves,.wy-control-group .wy-form-thirds{padding-bottom:12px}.wy-control-group .wy-form-full input[type=color],.wy-control-group .wy-form-full input[type=date],.wy-control-group .wy-form-full input[type=datetime-local],.wy-control-group .wy-form-full input[type=datetime],.wy-control-group .wy-form-full input[type=email],.wy-control-group .wy-form-full input[type=month],.wy-control-group .wy-form-full input[type=number],.wy-control-group .wy-form-full input[type=password],.wy-control-group .wy-form-full input[type=search],.wy-control-group .wy-form-full input[type=tel],.wy-control-group .wy-form-full input[type=text],.wy-control-group .wy-form-full input[type=time],.wy-control-group .wy-form-full input[type=url],.wy-control-group .wy-form-full input[type=week],.wy-control-group .wy-form-full select,.wy-control-group .wy-form-halves input[type=color],.wy-control-group .wy-form-halves input[type=date],.wy-control-group .wy-form-halves input[type=datetime-local],.wy-control-group .wy-form-halves input[type=datetime],.wy-control-group .wy-form-halves input[type=email],.wy-control-group .wy-form-halves input[type=month],.wy-control-group .wy-form-halves input[type=number],.wy-control-group .wy-form-halves input[type=password],.wy-control-group .wy-form-halves input[type=search],.wy-control-group .wy-form-halves input[type=tel],.wy-control-group .wy-form-halves input[type=text],.wy-control-group .wy-form-halves input[type=time],.wy-control-group .wy-form-halves input[type=url],.wy-control-group .wy-form-halves input[type=week],.wy-control-group .wy-form-halves select,.wy-control-group .wy-form-thirds input[type=color],.wy-control-group .wy-form-thirds input[type=date],.wy-control-group .wy-form-thirds input[type=datetime-local],.wy-control-group .wy-form-thirds input[type=datetime],.wy-control-group .wy-form-thirds input[type=email],.wy-control-group .wy-form-thirds input[type=month],.wy-control-group .wy-form-thirds input[type=number],.wy-control-group .wy-form-thirds input[type=password],.wy-control-group .wy-form-thirds input[type=search],.wy-control-group .wy-form-thirds input[type=tel],.wy-control-group .wy-form-thirds input[type=text],.wy-control-group .wy-form-thirds input[type=time],.wy-control-group .wy-form-thirds input[type=url],.wy-control-group .wy-form-thirds input[type=week],.wy-control-group .wy-form-thirds select{width:100%}.wy-control-group .wy-form-full{float:left;display:block;width:100%;margin-right:0}.wy-control-group .wy-form-full:last-child{margin-right:0}.wy-control-group .wy-form-halves{float:left;display:block;margin-right:2.35765%;width:48.82117%}.wy-control-group .wy-form-halves:last-child,.wy-control-group .wy-form-halves:nth-of-type(2n){margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(odd){clear:left}.wy-control-group .wy-form-thirds{float:left;display:block;margin-right:2.35765%;width:31.76157%}.wy-control-group .wy-form-thirds:last-child,.wy-control-group .wy-form-thirds:nth-of-type(3n){margin-right:0}.wy-control-group .wy-form-thirds:nth-of-type(3n+1){clear:left}.wy-control-group.wy-control-group-no-input .wy-control,.wy-control-no-input{margin:6px 0 0;font-size:90%}.wy-control-no-input{display:inline-block}.wy-control-group.fluid-input input[type=color],.wy-control-group.fluid-input input[type=date],.wy-control-group.fluid-input input[type=datetime-local],.wy-control-group.fluid-input input[type=datetime],.wy-control-group.fluid-input input[type=email],.wy-control-group.fluid-input input[type=month],.wy-control-group.fluid-input input[type=number],.wy-control-group.fluid-input input[type=password],.wy-control-group.fluid-input input[type=search],.wy-control-group.fluid-input input[type=tel],.wy-control-group.fluid-input input[type=text],.wy-control-group.fluid-input input[type=time],.wy-control-group.fluid-input input[type=url],.wy-control-group.fluid-input input[type=week]{width:100%}.wy-form-message-inline{padding-left:.3em;color:#666;font-size:90%}.wy-form-message{display:block;color:#999;font-size:70%;margin-top:.3125em;font-style:italic}.wy-form-message p{font-size:inherit;font-style:italic;margin-bottom:6px}.wy-form-message p:last-child{margin-bottom:0}input{line-height:normal}input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;*overflow:visible}input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week]{-webkit-appearance:none;padding:6px;display:inline-block;border:1px solid #ccc;font-size:80%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 3px #ddd;border-radius:0;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}input[type=datetime-local]{padding:.34375em .625em}input[disabled]{cursor:default}input[type=checkbox],input[type=radio]{padding:0;margin-right:.3125em;*height:13px;*width:13px}input[type=checkbox],input[type=radio],input[type=search]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}input[type=search]::-webkit-search-cancel-button,input[type=search]::-webkit-search-decoration{-webkit-appearance:none}input[type=color]:focus,input[type=date]:focus,input[type=datetime-local]:focus,input[type=datetime]:focus,input[type=email]:focus,input[type=month]:focus,input[type=number]:focus,input[type=password]:focus,input[type=search]:focus,input[type=tel]:focus,input[type=text]:focus,input[type=time]:focus,input[type=url]:focus,input[type=week]:focus{outline:0;outline:thin dotted\9;border-color:#333}input.no-focus:focus{border-color:#ccc!important}input[type=checkbox]:focus,input[type=file]:focus,input[type=radio]:focus{outline:thin dotted #333;outline:1px auto #129fea}input[type=color][disabled],input[type=date][disabled],input[type=datetime-local][disabled],input[type=datetime][disabled],input[type=email][disabled],input[type=month][disabled],input[type=number][disabled],input[type=password][disabled],input[type=search][disabled],input[type=tel][disabled],input[type=text][disabled],input[type=time][disabled],input[type=url][disabled],input[type=week][disabled]{cursor:not-allowed;background-color:#fafafa}input:focus:invalid,select:focus:invalid,textarea:focus:invalid{color:#e74c3c;border:1px solid #e74c3c}input:focus:invalid:focus,select:focus:invalid:focus,textarea:focus:invalid:focus{border-color:#e74c3c}input[type=checkbox]:focus:invalid:focus,input[type=file]:focus:invalid:focus,input[type=radio]:focus:invalid:focus{outline-color:#e74c3c}input.wy-input-large{padding:12px;font-size:100%}textarea{overflow:auto;vertical-align:top;width:100%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif}select,textarea{padding:.5em .625em;display:inline-block;border:1px solid #ccc;font-size:80%;box-shadow:inset 0 1px 3px #ddd;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}select{border:1px solid #ccc;background-color:#fff}select[multiple]{height:auto}select:focus,textarea:focus{outline:0}input[readonly],select[disabled],select[readonly],textarea[disabled],textarea[readonly]{cursor:not-allowed;background-color:#fafafa}input[type=checkbox][disabled],input[type=radio][disabled]{cursor:not-allowed}.wy-checkbox,.wy-radio{margin:6px 0;color:#404040;display:block}.wy-checkbox input,.wy-radio input{vertical-align:baseline}.wy-form-message-inline{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-input-prefix,.wy-input-suffix{white-space:nowrap;padding:6px}.wy-input-prefix .wy-input-context,.wy-input-suffix .wy-input-context{line-height:27px;padding:0 8px;display:inline-block;font-size:80%;background-color:#f3f6f6;border:1px solid #ccc;color:#999}.wy-input-suffix .wy-input-context{border-left:0}.wy-input-prefix .wy-input-context{border-right:0}.wy-switch{position:relative;display:block;height:24px;margin-top:12px;cursor:pointer}.wy-switch:before{left:0;top:0;width:36px;height:12px;background:#ccc}.wy-switch:after,.wy-switch:before{position:absolute;content:"";display:block;border-radius:4px;-webkit-transition:all .2s ease-in-out;-moz-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.wy-switch:after{width:18px;height:18px;background:#999;left:-3px;top:-3px}.wy-switch span{position:absolute;left:48px;display:block;font-size:12px;color:#ccc;line-height:1}.wy-switch.active:before{background:#1e8449}.wy-switch.active:after{left:24px;background:#27ae60}.wy-switch.disabled{cursor:not-allowed;opacity:.8}.wy-control-group.wy-control-group-error .wy-form-message,.wy-control-group.wy-control-group-error>label{color:#e74c3c}.wy-control-group.wy-control-group-error input[type=color],.wy-control-group.wy-control-group-error input[type=date],.wy-control-group.wy-control-group-error input[type=datetime-local],.wy-control-group.wy-control-group-error input[type=datetime],.wy-control-group.wy-control-group-error input[type=email],.wy-control-group.wy-control-group-error input[type=month],.wy-control-group.wy-control-group-error input[type=number],.wy-control-group.wy-control-group-error input[type=password],.wy-control-group.wy-control-group-error input[type=search],.wy-control-group.wy-control-group-error input[type=tel],.wy-control-group.wy-control-group-error input[type=text],.wy-control-group.wy-control-group-error input[type=time],.wy-control-group.wy-control-group-error input[type=url],.wy-control-group.wy-control-group-error input[type=week],.wy-control-group.wy-control-group-error textarea{border:1px solid #e74c3c}.wy-inline-validate{white-space:nowrap}.wy-inline-validate .wy-input-context{padding:.5em .625em;display:inline-block;font-size:80%}.wy-inline-validate.wy-inline-validate-success .wy-input-context{color:#27ae60}.wy-inline-validate.wy-inline-validate-danger .wy-input-context{color:#e74c3c}.wy-inline-validate.wy-inline-validate-warning .wy-input-context{color:#e67e22}.wy-inline-validate.wy-inline-validate-info .wy-input-context{color:#2980b9}.rotate-90{-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.rotate-180{-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.rotate-270{-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.mirror{-webkit-transform:scaleX(-1);-moz-transform:scaleX(-1);-ms-transform:scaleX(-1);-o-transform:scaleX(-1);transform:scaleX(-1)}.mirror.rotate-90{-webkit-transform:scaleX(-1) rotate(90deg);-moz-transform:scaleX(-1) rotate(90deg);-ms-transform:scaleX(-1) rotate(90deg);-o-transform:scaleX(-1) rotate(90deg);transform:scaleX(-1) rotate(90deg)}.mirror.rotate-180{-webkit-transform:scaleX(-1) rotate(180deg);-moz-transform:scaleX(-1) rotate(180deg);-ms-transform:scaleX(-1) rotate(180deg);-o-transform:scaleX(-1) rotate(180deg);transform:scaleX(-1) rotate(180deg)}.mirror.rotate-270{-webkit-transform:scaleX(-1) rotate(270deg);-moz-transform:scaleX(-1) rotate(270deg);-ms-transform:scaleX(-1) rotate(270deg);-o-transform:scaleX(-1) rotate(270deg);transform:scaleX(-1) rotate(270deg)}@media only screen and (max-width:480px){.wy-form button[type=submit]{margin:.7em 0 0}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=text],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week],.wy-form label{margin-bottom:.3em;display:block}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week]{margin-bottom:0}.wy-form-aligned .wy-control-group label{margin-bottom:.3em;text-align:left;display:block;width:100%}.wy-form-aligned .wy-control{margin:1.5em 0 0}.wy-form-message,.wy-form-message-inline,.wy-form .wy-help-inline{display:block;font-size:80%;padding:6px 0}}@media screen and (max-width:768px){.tablet-hide{display:none}}@media screen and (max-width:480px){.mobile-hide{display:none}}.float-left{float:left}.float-right{float:right}.full-width{width:100%}.rst-content table.docutils,.rst-content table.field-list,.wy-table{border-collapse:collapse;border-spacing:0;empty-cells:show;margin-bottom:24px}.rst-content table.docutils caption,.rst-content table.field-list caption,.wy-table caption{color:#000;font:italic 85%/1 arial,sans-serif;padding:1em 0;text-align:center}.rst-content table.docutils td,.rst-content table.docutils th,.rst-content table.field-list td,.rst-content table.field-list th,.wy-table td,.wy-table th{font-size:90%;margin:0;overflow:visible;padding:8px 16px}.rst-content table.docutils td:first-child,.rst-content table.docutils th:first-child,.rst-content table.field-list td:first-child,.rst-content table.field-list th:first-child,.wy-table td:first-child,.wy-table th:first-child{border-left-width:0}.rst-content table.docutils thead,.rst-content table.field-list thead,.wy-table thead{color:#000;text-align:left;vertical-align:bottom;white-space:nowrap}.rst-content table.docutils thead th,.rst-content table.field-list thead th,.wy-table thead th{font-weight:700;border-bottom:2px solid #e1e4e5}.rst-content table.docutils td,.rst-content table.field-list td,.wy-table td{background-color:transparent;vertical-align:middle}.rst-content table.docutils td p,.rst-content table.field-list td p,.wy-table td p{line-height:18px}.rst-content table.docutils td p:last-child,.rst-content table.field-list td p:last-child,.wy-table td p:last-child{margin-bottom:0}.rst-content table.docutils .wy-table-cell-min,.rst-content table.field-list .wy-table-cell-min,.wy-table .wy-table-cell-min{width:1%;padding-right:0}.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-content table.field-list .wy-table-cell-min input[type=checkbox],.wy-table .wy-table-cell-min input[type=checkbox]{margin:0}.wy-table-secondary{color:grey;font-size:90%}.wy-table-tertiary{color:grey;font-size:80%}.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td,.wy-table-backed,.wy-table-odd td,.wy-table-striped tr:nth-child(2n-1) td{background-color:#f3f6f6}.rst-content table.docutils,.wy-table-bordered-all{border:1px solid #e1e4e5}.rst-content table.docutils td,.wy-table-bordered-all td{border-bottom:1px solid #e1e4e5;border-left:1px solid #e1e4e5}.rst-content table.docutils tbody>tr:last-child td,.wy-table-bordered-all tbody>tr:last-child td{border-bottom-width:0}.wy-table-bordered{border:1px solid #e1e4e5}.wy-table-bordered-rows td{border-bottom:1px solid #e1e4e5}.wy-table-bordered-rows tbody>tr:last-child td{border-bottom-width:0}.wy-table-horizontal td,.wy-table-horizontal th{border-width:0 0 1px;border-bottom:1px solid #e1e4e5}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-responsive{margin-bottom:24px;max-width:100%;overflow:auto}.wy-table-responsive table{margin-bottom:0!important}.wy-table-responsive table td,.wy-table-responsive table th{white-space:nowrap}a{color:#2980b9;text-decoration:none;cursor:pointer}a:hover{color:#3091d1}a:visited{color:#9b59b6}html{height:100%}body,html{overflow-x:hidden}body{font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;font-weight:400;color:#404040;min-height:100%;background:#edf0f2}.wy-text-left{text-align:left}.wy-text-center{text-align:center}.wy-text-right{text-align:right}.wy-text-large{font-size:120%}.wy-text-normal{font-size:100%}.wy-text-small,small{font-size:80%}.wy-text-strike{text-decoration:line-through}.wy-text-warning{color:#e67e22!important}a.wy-text-warning:hover{color:#eb9950!important}.wy-text-info{color:#2980b9!important}a.wy-text-info:hover{color:#409ad5!important}.wy-text-success{color:#27ae60!important}a.wy-text-success:hover{color:#36d278!important}.wy-text-danger{color:#e74c3c!important}a.wy-text-danger:hover{color:#ed7669!important}.wy-text-neutral{color:#404040!important}a.wy-text-neutral:hover{color:#595959!important}.rst-content .toctree-wrapper>p.caption,h1,h2,h3,h4,h5,h6,legend{margin-top:0;font-weight:700;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif}p{line-height:24px;font-size:16px;margin:0 0 24px}h1{font-size:175%}.rst-content .toctree-wrapper>p.caption,h2{font-size:150%}h3{font-size:125%}h4{font-size:115%}h5{font-size:110%}h6{font-size:100%}hr{display:block;height:1px;border:0;border-top:1px solid #e1e4e5;margin:24px 0;padding:0}.rst-content code,.rst-content tt,code{white-space:nowrap;max-width:100%;background:#fff;border:1px solid #e1e4e5;font-size:75%;padding:0 5px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#e74c3c;overflow-x:auto}.rst-content tt.code-large,code.code-large{font-size:90%}.rst-content .section ul,.rst-content .toctree-wrapper ul,.wy-plain-list-disc,article ul{list-style:disc;line-height:24px;margin-bottom:24px}.rst-content .section ul li,.rst-content .toctree-wrapper ul li,.wy-plain-list-disc li,article ul li{list-style:disc;margin-left:24px}.rst-content .section ul li p:last-child,.rst-content .section ul li ul,.rst-content .toctree-wrapper ul li p:last-child,.rst-content .toctree-wrapper ul li ul,.wy-plain-list-disc li p:last-child,.wy-plain-list-disc li ul,article ul li p:last-child,article ul li ul{margin-bottom:0}.rst-content .section ul li li,.rst-content .toctree-wrapper ul li li,.wy-plain-list-disc li li,article ul li li{list-style:circle}.rst-content .section ul li li li,.rst-content .toctree-wrapper ul li li li,.wy-plain-list-disc li li li,article ul li li li{list-style:square}.rst-content .section ul li ol li,.rst-content .toctree-wrapper ul li ol li,.wy-plain-list-disc li ol li,article ul li ol li{list-style:decimal}.rst-content .section ol,.rst-content ol.arabic,.wy-plain-list-decimal,article ol{list-style:decimal;line-height:24px;margin-bottom:24px}.rst-content .section ol li,.rst-content ol.arabic li,.wy-plain-list-decimal li,article ol li{list-style:decimal;margin-left:24px}.rst-content .section ol li p:last-child,.rst-content .section ol li ul,.rst-content ol.arabic li p:last-child,.rst-content ol.arabic li ul,.wy-plain-list-decimal li p:last-child,.wy-plain-list-decimal li ul,article ol li p:last-child,article ol li ul{margin-bottom:0}.rst-content .section ol li ul li,.rst-content ol.arabic li ul li,.wy-plain-list-decimal li ul li,article ol li ul li{list-style:disc}.wy-breadcrumbs{*zoom:1}.wy-breadcrumbs:after,.wy-breadcrumbs:before{display:table;content:""}.wy-breadcrumbs:after{clear:both}.wy-breadcrumbs li{display:inline-block}.wy-breadcrumbs li.wy-breadcrumbs-aside{float:right}.wy-breadcrumbs li a{display:inline-block;padding:5px}.wy-breadcrumbs li a:first-child{padding-left:0}.rst-content .wy-breadcrumbs li tt,.wy-breadcrumbs li .rst-content tt,.wy-breadcrumbs li code{padding:5px;border:none;background:none}.rst-content .wy-breadcrumbs li tt.literal,.wy-breadcrumbs li .rst-content tt.literal,.wy-breadcrumbs li code.literal{color:#404040}.wy-breadcrumbs-extra{margin-bottom:0;color:#b3b3b3;font-size:80%;display:inline-block}@media screen and (max-width:480px){.wy-breadcrumbs-extra,.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}@media print{.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}html{font-size:16px}.wy-affix{position:fixed;top:1.618em}.wy-menu a:hover{text-decoration:none}.wy-menu-horiz{*zoom:1}.wy-menu-horiz:after,.wy-menu-horiz:before{display:table;content:""}.wy-menu-horiz:after{clear:both}.wy-menu-horiz li,.wy-menu-horiz ul{display:inline-block}.wy-menu-horiz li:hover{background:hsla(0,0%,100%,.1)}.wy-menu-horiz li.divide-left{border-left:1px solid #404040}.wy-menu-horiz li.divide-right{border-right:1px solid #404040}.wy-menu-horiz a{height:32px;display:inline-block;line-height:32px;padding:0 16px}.wy-menu-vertical{width:300px}.wy-menu-vertical header,.wy-menu-vertical p.caption{color:#55a5d9;height:32px;line-height:32px;padding:0 1.618em;margin:12px 0 0;display:block;font-weight:700;text-transform:uppercase;font-size:85%;white-space:nowrap}.wy-menu-vertical ul{margin-bottom:0}.wy-menu-vertical li.divide-top{border-top:1px solid #404040}.wy-menu-vertical li.divide-bottom{border-bottom:1px solid #404040}.wy-menu-vertical li.current{background:#e3e3e3}.wy-menu-vertical li.current a{color:grey;border-right:1px solid #c9c9c9;padding:.4045em 2.427em}.wy-menu-vertical li.current a:hover{background:#d6d6d6}.rst-content .wy-menu-vertical li tt,.wy-menu-vertical li .rst-content tt,.wy-menu-vertical li code{border:none;background:inherit;color:inherit;padding-left:0;padding-right:0}.wy-menu-vertical li span.toctree-expand{display:block;float:left;margin-left:-1.2em;font-size:.8em;line-height:1.6em;color:#4d4d4d}.wy-menu-vertical li.current>a,.wy-menu-vertical li.on a{color:#404040;font-weight:700;position:relative;background:#fcfcfc;border:none;padding:.4045em 1.618em}.wy-menu-vertical li.current>a:hover,.wy-menu-vertical li.on a:hover{background:#fcfcfc}.wy-menu-vertical li.current>a:hover span.toctree-expand,.wy-menu-vertical li.on a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand{display:block;font-size:.8em;line-height:1.6em;color:#333}.wy-menu-vertical li.toctree-l1.current>a{border-bottom:1px solid #c9c9c9;border-top:1px solid #c9c9c9}.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .toctree-l11>ul{display:none}.wy-menu-vertical .toctree-l1.current .current.toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .current.toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .current.toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .current.toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .current.toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .current.toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .current.toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .current.toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .current.toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .current.toctree-l11>ul{display:block}.wy-menu-vertical li.toctree-l3,.wy-menu-vertical li.toctree-l4{font-size:.9em}.wy-menu-vertical li.toctree-l2 a,.wy-menu-vertical li.toctree-l3 a,.wy-menu-vertical li.toctree-l4 a,.wy-menu-vertical li.toctree-l5 a,.wy-menu-vertical li.toctree-l6 a,.wy-menu-vertical li.toctree-l7 a,.wy-menu-vertical li.toctree-l8 a,.wy-menu-vertical li.toctree-l9 a,.wy-menu-vertical li.toctree-l10 a{color:#404040}.wy-menu-vertical li.toctree-l2 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l3 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l4 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l5 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l6 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l7 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l8 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l9 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l10 a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{display:block}.wy-menu-vertical li.toctree-l2.current>a{padding:.4045em 2.427em}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current>a{padding:.4045em 4.045em}.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current>a{padding:.4045em 5.663em}.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current>a{padding:.4045em 7.281em}.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current>a{padding:.4045em 8.899em}.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current>a{padding:.4045em 10.517em}.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current>a{padding:.4045em 12.135em}.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current>a{padding:.4045em 13.753em}.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current>a{padding:.4045em 15.371em}.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{padding:.4045em 16.989em}.wy-menu-vertical li.toctree-l2.current>a,.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{background:#c9c9c9}.wy-menu-vertical li.toctree-l2 span.toctree-expand{color:#a3a3a3}.wy-menu-vertical li.toctree-l3.current>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{background:#bdbdbd}.wy-menu-vertical li.toctree-l3 span.toctree-expand{color:#969696}.wy-menu-vertical li.current ul{display:block}.wy-menu-vertical li ul{margin-bottom:0;display:none}.wy-menu-vertical li ul li a{margin-bottom:0;color:#d9d9d9;font-weight:400}.wy-menu-vertical a{line-height:18px;padding:.4045em 1.618em;display:block;position:relative;font-size:90%;color:#d9d9d9}.wy-menu-vertical a:hover{background-color:#4e4a4a;cursor:pointer}.wy-menu-vertical a:hover span.toctree-expand{color:#d9d9d9}.wy-menu-vertical a:active{background-color:#2980b9;cursor:pointer;color:#fff}.wy-menu-vertical a:active span.toctree-expand{color:#fff}.wy-side-nav-search{display:block;width:300px;padding:.809em;margin-bottom:.809em;z-index:200;background-color:#2980b9;text-align:center;color:#fcfcfc}.wy-side-nav-search input[type=text]{width:100%;border-radius:50px;padding:6px 12px;border-color:#2472a4}.wy-side-nav-search img{display:block;margin:auto auto .809em;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a{color:#fcfcfc;font-size:100%;font-weight:700;display:inline-block;padding:4px 6px;margin-bottom:.809em}.wy-side-nav-search .wy-dropdown>a:hover,.wy-side-nav-search>a:hover{background:hsla(0,0%,100%,.1)}.wy-side-nav-search .wy-dropdown>a img.logo,.wy-side-nav-search>a img.logo{display:block;margin:0 auto;height:auto;width:auto;border-radius:0;max-width:100%;background:transparent}.wy-side-nav-search .wy-dropdown>a.icon img.logo,.wy-side-nav-search>a.icon img.logo{margin-top:.85em}.wy-side-nav-search>div.version{margin-top:-.4045em;margin-bottom:.809em;font-weight:400;color:hsla(0,0%,100%,.3)}.wy-nav .wy-menu-vertical header{color:#2980b9}.wy-nav .wy-menu-vertical a{color:#b3b3b3}.wy-nav .wy-menu-vertical a:hover{background-color:#2980b9;color:#fff}[data-menu-wrap]{-webkit-transition:all .2s ease-in;-moz-transition:all .2s ease-in;transition:all .2s ease-in;position:absolute;opacity:1;width:100%;opacity:0}[data-menu-wrap].move-center{left:0;right:auto;opacity:1}[data-menu-wrap].move-left{right:auto;left:-100%;opacity:0}[data-menu-wrap].move-right{right:-100%;left:auto;opacity:0}.wy-body-for-nav{background:#fcfcfc}.wy-grid-for-nav{position:absolute;width:100%;height:100%}.wy-nav-side{position:fixed;top:0;bottom:0;left:0;padding-bottom:2em;width:300px;overflow-x:hidden;overflow-y:hidden;min-height:100%;color:#9b9b9b;background:#343131;z-index:200}.wy-side-scroll{width:320px;position:relative;overflow-x:hidden;overflow-y:scroll;height:100%}.wy-nav-top{display:none;background:#2980b9;color:#fff;padding:.4045em .809em;position:relative;line-height:50px;text-align:center;font-size:100%;*zoom:1}.wy-nav-top:after,.wy-nav-top:before{display:table;content:""}.wy-nav-top:after{clear:both}.wy-nav-top a{color:#fff;font-weight:700}.wy-nav-top img{margin-right:12px;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-nav-top i{font-size:30px;float:left;cursor:pointer;padding-top:inherit}.wy-nav-content-wrap{margin-left:300px;background:#fcfcfc;min-height:100%}.wy-nav-content{padding:1.618em 3.236em;height:100%;max-width:800px;margin:auto}.wy-body-mask{position:fixed;width:100%;height:100%;background:rgba(0,0,0,.2);display:none;z-index:499}.wy-body-mask.on{display:block}footer{color:grey}footer p{margin-bottom:12px}.rst-content footer span.commit tt,footer span.commit .rst-content tt,footer span.commit code{padding:0;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:1em;background:none;border:none;color:grey}.rst-footer-buttons{*zoom:1}.rst-footer-buttons:after,.rst-footer-buttons:before{width:100%;display:table;content:""}.rst-footer-buttons:after{clear:both}.rst-breadcrumbs-buttons{margin-top:12px;*zoom:1}.rst-breadcrumbs-buttons:after,.rst-breadcrumbs-buttons:before{display:table;content:""}.rst-breadcrumbs-buttons:after{clear:both}#search-results .search li{margin-bottom:24px;border-bottom:1px solid #e1e4e5;padding-bottom:24px}#search-results .search li:first-child{border-top:1px solid #e1e4e5;padding-top:24px}#search-results .search li a{font-size:120%;margin-bottom:12px;display:inline-block}#search-results .context{color:grey;font-size:90%}.genindextable li>ul{margin-left:24px}@media screen and (max-width:768px){.wy-body-for-nav{background:#fcfcfc}.wy-nav-top{display:block}.wy-nav-side{left:-300px}.wy-nav-side.shift{width:85%;left:0}.wy-menu.wy-menu-vertical,.wy-side-nav-search,.wy-side-scroll{width:auto}.wy-nav-content-wrap{margin-left:0}.wy-nav-content-wrap .wy-nav-content{padding:1.618em}.wy-nav-content-wrap.shift{position:fixed;min-width:100%;left:85%;top:0;height:100%;overflow:hidden}}@media screen and (min-width:1100px){.wy-nav-content-wrap{background:rgba(0,0,0,.05)}.wy-nav-content{margin:0;background:#fcfcfc}}@media print{.rst-versions,.wy-nav-side,footer{display:none}.wy-nav-content-wrap{margin-left:0}}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60;*zoom:1}.rst-versions .rst-current-version:after,.rst-versions .rst-current-version:before{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-content .code-block-caption .rst-versions .rst-current-version .headerlink,.rst-content .rst-versions .rst-current-version .admonition-title,.rst-content code.download .rst-versions .rst-current-version span:first-child,.rst-content dl dt .rst-versions .rst-current-version .headerlink,.rst-content h1 .rst-versions .rst-current-version .headerlink,.rst-content h2 .rst-versions .rst-current-version .headerlink,.rst-content h3 .rst-versions .rst-current-version .headerlink,.rst-content h4 .rst-versions .rst-current-version .headerlink,.rst-content h5 .rst-versions .rst-current-version .headerlink,.rst-content h6 .rst-versions .rst-current-version .headerlink,.rst-content p.caption .rst-versions .rst-current-version .headerlink,.rst-content table>caption .rst-versions .rst-current-version .headerlink,.rst-content tt.download .rst-versions .rst-current-version span:first-child,.rst-versions .rst-current-version .fa,.rst-versions .rst-current-version .icon,.rst-versions .rst-current-version .rst-content .admonition-title,.rst-versions .rst-current-version .rst-content .code-block-caption .headerlink,.rst-versions .rst-current-version .rst-content code.download span:first-child,.rst-versions .rst-current-version .rst-content dl dt .headerlink,.rst-versions .rst-current-version .rst-content h1 .headerlink,.rst-versions .rst-current-version .rst-content h2 .headerlink,.rst-versions .rst-current-version .rst-content h3 .headerlink,.rst-versions .rst-current-version .rst-content h4 .headerlink,.rst-versions .rst-current-version .rst-content h5 .headerlink,.rst-versions .rst-current-version .rst-content h6 .headerlink,.rst-versions .rst-current-version .rst-content p.caption .headerlink,.rst-versions .rst-current-version .rst-content table>caption .headerlink,.rst-versions .rst-current-version .rst-content tt.download span:first-child,.rst-versions .rst-current-version .wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li .rst-versions .rst-current-version span.toctree-expand{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}}.rst-content img{max-width:100%;height:auto}.rst-content div.figure{margin-bottom:24px}.rst-content div.figure p.caption{font-style:italic}.rst-content div.figure p:last-child.caption{margin-bottom:0}.rst-content div.figure.align-center{text-align:center}.rst-content .section>a>img,.rst-content .section>img{margin-bottom:24px}.rst-content abbr[title]{text-decoration:none}.rst-content.style-external-links a.reference.external:after{font-family:FontAwesome;content:"\f08e";color:#b3b3b3;vertical-align:super;font-size:60%;margin:0 .2em}.rst-content blockquote{margin-left:24px;line-height:24px;margin-bottom:24px}.rst-content pre.literal-block{white-space:pre;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;display:block;overflow:auto}.rst-content div[class^=highlight],.rst-content pre.literal-block{border:1px solid #e1e4e5;overflow-x:auto;margin:1px 0 24px}.rst-content div[class^=highlight] div[class^=highlight],.rst-content pre.literal-block div[class^=highlight]{padding:0;border:none;margin:0}.rst-content div[class^=highlight] td.code{width:100%}.rst-content .linenodiv pre{border-right:1px solid #e6e9ea;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;user-select:none;pointer-events:none}.rst-content div[class^=highlight] pre{white-space:pre;margin:0;padding:12px;display:block;overflow:auto}.rst-content div[class^=highlight] pre .hll{display:block;margin:0 -12px;padding:0 12px}.rst-content .linenodiv pre,.rst-content div[class^=highlight] pre,.rst-content pre.literal-block{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:12px;line-height:1.4}.rst-content div.highlight .gp{user-select:none;pointer-events:none}.rst-content .code-block-caption{font-style:italic;font-size:85%;line-height:1;padding:1em 0;text-align:center}@media print{.rst-content .codeblock,.rst-content div[class^=highlight],.rst-content div[class^=highlight] pre{white-space:pre-wrap}}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning{clear:both}.rst-content .admonition-todo .last,.rst-content .admonition-todo>:last-child,.rst-content .admonition .last,.rst-content .admonition>:last-child,.rst-content .attention .last,.rst-content .attention>:last-child,.rst-content .caution .last,.rst-content .caution>:last-child,.rst-content .danger .last,.rst-content .danger>:last-child,.rst-content .error .last,.rst-content .error>:last-child,.rst-content .hint .last,.rst-content .hint>:last-child,.rst-content .important .last,.rst-content .important>:last-child,.rst-content .note .last,.rst-content .note>:last-child,.rst-content .seealso .last,.rst-content .seealso>:last-child,.rst-content .tip .last,.rst-content .tip>:last-child,.rst-content .warning .last,.rst-content .warning>:last-child{margin-bottom:0}.rst-content .admonition-title:before{margin-right:4px}.rst-content .admonition table{border-color:rgba(0,0,0,.1)}.rst-content .admonition table td,.rst-content .admonition table th{background:transparent!important;border-color:rgba(0,0,0,.1)!important}.rst-content .section ol.loweralpha,.rst-content .section ol.loweralpha>li{list-style:lower-alpha}.rst-content .section ol.upperalpha,.rst-content .section ol.upperalpha>li{list-style:upper-alpha}.rst-content .section ol li>*,.rst-content .section ul li>*{margin-top:12px;margin-bottom:12px}.rst-content .section ol li>:first-child,.rst-content .section ul li>:first-child{margin-top:0}.rst-content .section ol li>p,.rst-content .section ol li>p:last-child,.rst-content .section ul li>p,.rst-content .section ul li>p:last-child{margin-bottom:12px}.rst-content .section ol li>p:only-child,.rst-content .section ol li>p:only-child:last-child,.rst-content .section ul li>p:only-child,.rst-content .section ul li>p:only-child:last-child{margin-bottom:0}.rst-content .section ol li>ol,.rst-content .section ol li>ul,.rst-content .section ul li>ol,.rst-content .section ul li>ul{margin-bottom:12px}.rst-content .section ol.simple li>*,.rst-content .section ol.simple li ol,.rst-content .section ol.simple li ul,.rst-content .section ul.simple li>*,.rst-content .section ul.simple li ol,.rst-content .section ul.simple li ul{margin-top:0;margin-bottom:0}.rst-content .line-block{margin-left:0;margin-bottom:24px;line-height:24px}.rst-content .line-block .line-block{margin-left:24px;margin-bottom:0}.rst-content .topic-title{font-weight:700;margin-bottom:12px}.rst-content .toc-backref{color:#404040}.rst-content .align-right{float:right;margin:0 0 24px 24px}.rst-content .align-left{float:left;margin:0 24px 24px 0}.rst-content .align-center{margin:auto}.rst-content .align-center:not(table){display:block}.rst-content .code-block-caption .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink{visibility:hidden;font-size:14px}.rst-content .code-block-caption .headerlink:after,.rst-content .toctree-wrapper>p.caption .headerlink:after,.rst-content dl dt .headerlink:after,.rst-content h1 .headerlink:after,.rst-content h2 .headerlink:after,.rst-content h3 .headerlink:after,.rst-content h4 .headerlink:after,.rst-content h5 .headerlink:after,.rst-content h6 .headerlink:after,.rst-content p.caption .headerlink:after,.rst-content table>caption .headerlink:after{content:"\f0c1";font-family:FontAwesome}.rst-content .code-block-caption:hover .headerlink:after,.rst-content .toctree-wrapper>p.caption:hover .headerlink:after,.rst-content dl dt:hover .headerlink:after,.rst-content h1:hover .headerlink:after,.rst-content h2:hover .headerlink:after,.rst-content h3:hover .headerlink:after,.rst-content h4:hover .headerlink:after,.rst-content h5:hover .headerlink:after,.rst-content h6:hover .headerlink:after,.rst-content p.caption:hover .headerlink:after,.rst-content table>caption:hover .headerlink:after{visibility:visible}.rst-content table>caption .headerlink:after{font-size:12px}.rst-content .centered{text-align:center}.rst-content .sidebar{float:right;width:40%;display:block;margin:0 0 24px 24px;padding:24px;background:#f3f6f6;border:1px solid #e1e4e5}.rst-content .sidebar dl,.rst-content .sidebar p,.rst-content .sidebar ul{font-size:90%}.rst-content .sidebar .last,.rst-content .sidebar>:last-child{margin-bottom:0}.rst-content .sidebar .sidebar-title{display:block;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif;font-weight:700;background:#e1e4e5;padding:6px 12px;margin:-24px -24px 24px;font-size:100%}.rst-content .highlighted{background:#f1c40f;box-shadow:0 0 0 2px #f1c40f;display:inline;font-weight:700}.rst-content .citation-reference,.rst-content .footnote-reference{vertical-align:baseline;position:relative;top:-.4em;line-height:0;font-size:90%}.rst-content .hlist{width:100%}html.writer-html4 .rst-content table.docutils.citation,html.writer-html4 .rst-content table.docutils.footnote{background:none;border:none}html.writer-html4 .rst-content table.docutils.citation td,html.writer-html4 .rst-content table.docutils.citation tr,html.writer-html4 .rst-content table.docutils.footnote td,html.writer-html4 .rst-content table.docutils.footnote tr{border:none;background-color:transparent!important;white-space:normal}html.writer-html4 .rst-content table.docutils.citation td.label,html.writer-html4 .rst-content table.docutils.footnote td.label{padding-left:0;padding-right:0;vertical-align:top}html.writer-html5 .rst-content dl dt span.classifier:before{content:" : "}html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{display:grid;grid-template-columns:max-content auto}html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{padding-left:1rem}html.writer-html5 .rst-content dl.field-list>dt:after,html.writer-html5 .rst-content dl.footnote>dt:after{content:":"}html.writer-html5 .rst-content dl.field-list>dd,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dd,html.writer-html5 .rst-content dl.footnote>dt{margin-bottom:0}html.writer-html5 .rst-content dl.footnote{font-size:.9rem}html.writer-html5 .rst-content dl.footnote>dt{margin:0 .5rem .5rem 0;line-height:1.2rem;word-break:break-all;font-weight:400}html.writer-html5 .rst-content dl.footnote>dt>span.brackets{margin-right:.5rem}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:before{content:"["}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:after{content:"]"}html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref{font-style:italic}html.writer-html5 .rst-content dl.footnote>dd{margin:0 0 .5rem;line-height:1.2rem}html.writer-html5 .rst-content dl.footnote>dd p,html.writer-html5 .rst-content dl.option-list kbd{font-size:.9rem}.rst-content table.docutils.footnote,html.writer-html4 .rst-content table.docutils.citation,html.writer-html5 .rst-content dl.footnote{color:grey}.rst-content table.docutils.footnote code,.rst-content table.docutils.footnote tt,html.writer-html4 .rst-content table.docutils.citation code,html.writer-html4 .rst-content table.docutils.citation tt,html.writer-html5 .rst-content dl.footnote code,html.writer-html5 .rst-content dl.footnote tt{color:#555}.rst-content .wy-table-responsive.citation,.rst-content .wy-table-responsive.footnote{margin-bottom:0}.rst-content .wy-table-responsive.citation+:not(.citation),.rst-content .wy-table-responsive.footnote+:not(.footnote){margin-top:24px}.rst-content .wy-table-responsive.citation:last-child,.rst-content .wy-table-responsive.footnote:last-child{margin-bottom:24px}.rst-content table.docutils th{border-color:#e1e4e5}html.writer-html5 .rst-content table.docutils th{border:1px solid #e1e4e5}html.writer-html5 .rst-content table.docutils td>p,html.writer-html5 .rst-content table.docutils th>p{line-height:1rem;margin-bottom:0;font-size:.9rem}.rst-content table.docutils td .last,.rst-content table.docutils td .last>:last-child{margin-bottom:0}.rst-content table.field-list,.rst-content table.field-list td{border:none}.rst-content table.field-list td p{font-size:inherit;line-height:inherit}.rst-content table.field-list td>strong{display:inline-block}.rst-content table.field-list .field-name{padding-right:10px;text-align:left;white-space:nowrap}.rst-content table.field-list .field-body{text-align:left}.rst-content code,.rst-content tt{color:#000;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;padding:2px 5px}.rst-content code big,.rst-content code em,.rst-content tt big,.rst-content tt em{font-size:100%!important;line-height:normal}.rst-content code.literal,.rst-content tt.literal{color:#e74c3c}.rst-content code.xref,.rst-content tt.xref,a .rst-content code,a .rst-content tt{font-weight:700;color:#404040}.rst-content kbd,.rst-content pre,.rst-content samp{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace}.rst-content a code,.rst-content a tt{color:#2980b9}.rst-content dl{margin-bottom:24px}.rst-content dl dt{font-weight:700;margin-bottom:12px}.rst-content dl ol,.rst-content dl p,.rst-content dl table,.rst-content dl ul{margin-bottom:12px}.rst-content dl dd{margin:0 0 12px 24px;line-height:24px}html.writer-html4 .rst-content dl:not(.docutils),html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple){margin-bottom:24px}html.writer-html4 .rst-content dl:not(.docutils)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt{display:table;margin:6px 0;font-size:90%;line-height:normal;background:#e7f2fa;color:#2980b9;border-top:3px solid #6ab0de;padding:6px;position:relative}html.writer-html4 .rst-content dl:not(.docutils)>dt:before,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:before{color:#6ab0de}html.writer-html4 .rst-content dl:not(.docutils)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt{margin-bottom:6px;border:none;border-left:3px solid #ccc;background:#f0f0f0;color:#555}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils)>dt:first-child,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:first-child{margin-top:0}html.writer-html4 .rst-content dl:not(.docutils) code,html.writer-html4 .rst-content dl:not(.docutils) tt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) code.descclassname,html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descclassname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{background-color:transparent;border:none;padding:0;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .optional,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .optional{display:inline-block;padding:0 4px;color:#000;font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .property,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .property{display:inline-block;padding-right:8px}.rst-content .viewcode-back,.rst-content .viewcode-link{display:inline-block;color:#27ae60;font-size:80%;padding-left:24px}.rst-content .viewcode-back{display:block;float:right}.rst-content p.rubric{margin-bottom:12px;font-weight:700}.rst-content code.download,.rst-content tt.download{background:inherit;padding:inherit;font-weight:400;font-family:inherit;font-size:inherit;color:inherit;border:inherit;white-space:inherit}.rst-content code.download span:first-child,.rst-content tt.download span:first-child{-webkit-font-smoothing:subpixel-antialiased}.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{margin-right:4px}.rst-content .guilabel{border:1px solid #7fbbe3;background:#e7f2fa;font-size:80%;font-weight:700;border-radius:4px;padding:2.4px 6px;margin:auto 2px}.rst-content .versionmodified{font-style:italic}@media screen and (max-width:480px){.rst-content .sidebar{width:100%}}span[id*=MathJax-Span]{color:#404040}.math{text-align:center}@font-face{font-family:Lato;src:url(fonts/lato-normal.woff2?bd03a2cc277bbbc338d464e679fe9942) format("woff2"),url(fonts/lato-normal.woff?27bd77b9162d388cb8d4c4217c7c5e2a) format("woff");font-weight:400;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold.woff2?cccb897485813c7c256901dbca54ecf2) format("woff2"),url(fonts/lato-bold.woff?d878b6c29b10beca227e9eef4246111b) format("woff");font-weight:700;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold-italic.woff2?0b6bb6725576b072c5d0b02ecdd1900d) format("woff2"),url(fonts/lato-bold-italic.woff?9c7e4e9eb485b4a121c760e61bc3707c) format("woff");font-weight:700;font-style:italic;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-normal-italic.woff2?4eb103b4d12be57cb1d040ed5e162e9d) format("woff2"),url(fonts/lato-normal-italic.woff?f28f2d6482446544ef1ea1ccc6dd5892) format("woff");font-weight:400;font-style:italic;font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:400;src:url(fonts/Roboto-Slab-Regular.woff2?7abf5b8d04d26a2cafea937019bca958) format("woff2"),url(fonts/Roboto-Slab-Regular.woff?c1be9284088d487c5e3ff0a10a92e58c) format("woff");font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:700;src:url(fonts/Roboto-Slab-Bold.woff2?9984f4a9bda09be08e83f2506954adbe) format("woff2"),url(fonts/Roboto-Slab-Bold.woff?bed5564a116b05148e3b3bea6fb1162a) format("woff");font-display:block} \ No newline at end of file diff --git a/en/3.4/_static/doctools.js b/en/3.4/_static/doctools.js new file mode 100644 index 0000000000..527b876ca6 --- /dev/null +++ b/en/3.4/_static/doctools.js @@ -0,0 +1,156 @@ +/* + * doctools.js + * ~~~~~~~~~~~ + * + * Base JavaScript utilities for all Sphinx HTML documentation. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ +"use strict"; + +const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([ + "TEXTAREA", + "INPUT", + "SELECT", + "BUTTON", +]); + +const _ready = (callback) => { + if (document.readyState !== "loading") { + callback(); + } else { + document.addEventListener("DOMContentLoaded", callback); + } +}; + +/** + * Small JavaScript module for the documentation. + */ +const Documentation = { + init: () => { + Documentation.initDomainIndexTable(); + Documentation.initOnKeyListeners(); + }, + + /** + * i18n support + */ + TRANSLATIONS: {}, + PLURAL_EXPR: (n) => (n === 1 ? 0 : 1), + LOCALE: "unknown", + + // gettext and ngettext don't access this so that the functions + // can safely bound to a different name (_ = Documentation.gettext) + gettext: (string) => { + const translated = Documentation.TRANSLATIONS[string]; + switch (typeof translated) { + case "undefined": + return string; // no translation + case "string": + return translated; // translation exists + default: + return translated[0]; // (singular, plural) translation tuple exists + } + }, + + ngettext: (singular, plural, n) => { + const translated = Documentation.TRANSLATIONS[singular]; + if (typeof translated !== "undefined") + return translated[Documentation.PLURAL_EXPR(n)]; + return n === 1 ? singular : plural; + }, + + addTranslations: (catalog) => { + Object.assign(Documentation.TRANSLATIONS, catalog.messages); + Documentation.PLURAL_EXPR = new Function( + "n", + `return (${catalog.plural_expr})` + ); + Documentation.LOCALE = catalog.locale; + }, + + /** + * helper function to focus on search bar + */ + focusSearchBar: () => { + document.querySelectorAll("input[name=q]")[0]?.focus(); + }, + + /** + * Initialise the domain index toggle buttons + */ + initDomainIndexTable: () => { + const toggler = (el) => { + const idNumber = el.id.substr(7); + const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`); + if (el.src.substr(-9) === "minus.png") { + el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`; + toggledRows.forEach((el) => (el.style.display = "none")); + } else { + el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`; + toggledRows.forEach((el) => (el.style.display = "")); + } + }; + + const togglerElements = document.querySelectorAll("img.toggler"); + togglerElements.forEach((el) => + el.addEventListener("click", (event) => toggler(event.currentTarget)) + ); + togglerElements.forEach((el) => (el.style.display = "")); + if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler); + }, + + initOnKeyListeners: () => { + // only install a listener if it is really needed + if ( + !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS && + !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS + ) + return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.altKey || event.ctrlKey || event.metaKey) return; + + if (!event.shiftKey) { + switch (event.key) { + case "ArrowLeft": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const prevLink = document.querySelector('link[rel="prev"]'); + if (prevLink && prevLink.href) { + window.location.href = prevLink.href; + event.preventDefault(); + } + break; + case "ArrowRight": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const nextLink = document.querySelector('link[rel="next"]'); + if (nextLink && nextLink.href) { + window.location.href = nextLink.href; + event.preventDefault(); + } + break; + } + } + + // some keyboard layouts may need Shift to get / + switch (event.key) { + case "/": + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break; + Documentation.focusSearchBar(); + event.preventDefault(); + } + }); + }, +}; + +// quick alias for translations +const _ = Documentation.gettext; + +_ready(Documentation.init); diff --git a/en/3.4/_static/documentation_options.js b/en/3.4/_static/documentation_options.js new file mode 100644 index 0000000000..f9961581ec --- /dev/null +++ b/en/3.4/_static/documentation_options.js @@ -0,0 +1,14 @@ +var DOCUMENTATION_OPTIONS = { + URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), + VERSION: 'Netris v3.0', + LANGUAGE: 'en', + COLLAPSE_INDEX: false, + BUILDER: 'html', + FILE_SUFFIX: '.html', + LINK_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt', + NAVIGATION_WITH_KEYS: false, + SHOW_SEARCH_SUMMARY: true, + ENABLE_SEARCH_SHORTCUTS: true, +}; \ No newline at end of file diff --git a/en/3.4/_static/file.png b/en/3.4/_static/file.png new file mode 100644 index 0000000000..a858a410e4 Binary files /dev/null and b/en/3.4/_static/file.png differ diff --git a/en/3.4/_static/jquery-3.6.0.js b/en/3.4/_static/jquery-3.6.0.js new file mode 100644 index 0000000000..fc6c299b73 --- /dev/null +++ b/en/3.4/_static/jquery-3.6.0.js @@ -0,0 +1,10881 @@ +/*! + * jQuery JavaScript Library v3.6.0 + * https://jquery.com/ + * + * Includes Sizzle.js + * https://sizzlejs.com/ + * + * Copyright OpenJS Foundation and other contributors + * Released under the MIT license + * https://jquery.org/license + * + * Date: 2021-03-02T17:08Z + */ +( function( global, factory ) { + + "use strict"; + + if ( typeof module === "object" && typeof module.exports === "object" ) { + + // For CommonJS and CommonJS-like environments where a proper `window` + // is present, execute the factory and get jQuery. + // For environments that do not have a `window` with a `document` + // (such as Node.js), expose a factory as module.exports. + // This accentuates the need for the creation of a real `window`. + // e.g. var jQuery = require("jquery")(window); + // See ticket #14549 for more info. + module.exports = global.document ? + factory( global, true ) : + function( w ) { + if ( !w.document ) { + throw new Error( "jQuery requires a window with a document" ); + } + return factory( w ); + }; + } else { + factory( global ); + } + +// Pass this if window is not defined yet +} )( typeof window !== "undefined" ? window : this, function( window, noGlobal ) { + +// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 - 9.1 +// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict mode +// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict mode should be common +// enough that all such attempts are guarded in a try block. +"use strict"; + +var arr = []; + +var getProto = Object.getPrototypeOf; + +var slice = arr.slice; + +var flat = arr.flat ? function( array ) { + return arr.flat.call( array ); +} : function( array ) { + return arr.concat.apply( [], array ); +}; + + +var push = arr.push; + +var indexOf = arr.indexOf; + +var class2type = {}; + +var toString = class2type.toString; + +var hasOwn = class2type.hasOwnProperty; + +var fnToString = hasOwn.toString; + +var ObjectFunctionString = fnToString.call( Object ); + +var support = {}; + +var isFunction = function isFunction( obj ) { + + // Support: Chrome <=57, Firefox <=52 + // In some browsers, typeof returns "function" for HTML elements + // (i.e., `typeof document.createElement( "object" ) === "function"`). + // We don't want to classify *any* DOM node as a function. + // Support: QtWeb <=3.8.5, WebKit <=534.34, wkhtmltopdf tool <=0.12.5 + // Plus for old WebKit, typeof returns "function" for HTML collections + // (e.g., `typeof document.getElementsByTagName("div") === "function"`). (gh-4756) + return typeof obj === "function" && typeof obj.nodeType !== "number" && + typeof obj.item !== "function"; + }; + + +var isWindow = function isWindow( obj ) { + return obj != null && obj === obj.window; + }; + + +var document = window.document; + + + + var preservedScriptAttributes = { + type: true, + src: true, + nonce: true, + noModule: true + }; + + function DOMEval( code, node, doc ) { + doc = doc || document; + + var i, val, + script = doc.createElement( "script" ); + + script.text = code; + if ( node ) { + for ( i in preservedScriptAttributes ) { + + // Support: Firefox 64+, Edge 18+ + // Some browsers don't support the "nonce" property on scripts. + // On the other hand, just using `getAttribute` is not enough as + // the `nonce` attribute is reset to an empty string whenever it + // becomes browsing-context connected. + // See https://github.com/whatwg/html/issues/2369 + // See https://html.spec.whatwg.org/#nonce-attributes + // The `node.getAttribute` check was added for the sake of + // `jQuery.globalEval` so that it can fake a nonce-containing node + // via an object. + val = node[ i ] || node.getAttribute && node.getAttribute( i ); + if ( val ) { + script.setAttribute( i, val ); + } + } + } + doc.head.appendChild( script ).parentNode.removeChild( script ); + } + + +function toType( obj ) { + if ( obj == null ) { + return obj + ""; + } + + // Support: Android <=2.3 only (functionish RegExp) + return typeof obj === "object" || typeof obj === "function" ? + class2type[ toString.call( obj ) ] || "object" : + typeof obj; +} +/* global Symbol */ +// Defining this global in .eslintrc.json would create a danger of using the global +// unguarded in another place, it seems safer to define global only for this module + + + +var + version = "3.6.0", + + // Define a local copy of jQuery + jQuery = function( selector, context ) { + + // The jQuery object is actually just the init constructor 'enhanced' + // Need init if jQuery is called (just allow error to be thrown if not included) + return new jQuery.fn.init( selector, context ); + }; + +jQuery.fn = jQuery.prototype = { + + // The current version of jQuery being used + jquery: version, + + constructor: jQuery, + + // The default length of a jQuery object is 0 + length: 0, + + toArray: function() { + return slice.call( this ); + }, + + // Get the Nth element in the matched element set OR + // Get the whole matched element set as a clean array + get: function( num ) { + + // Return all the elements in a clean array + if ( num == null ) { + return slice.call( this ); + } + + // Return just the one element from the set + return num < 0 ? this[ num + this.length ] : this[ num ]; + }, + + // Take an array of elements and push it onto the stack + // (returning the new matched element set) + pushStack: function( elems ) { + + // Build a new jQuery matched element set + var ret = jQuery.merge( this.constructor(), elems ); + + // Add the old object onto the stack (as a reference) + ret.prevObject = this; + + // Return the newly-formed element set + return ret; + }, + + // Execute a callback for every element in the matched set. + each: function( callback ) { + return jQuery.each( this, callback ); + }, + + map: function( callback ) { + return this.pushStack( jQuery.map( this, function( elem, i ) { + return callback.call( elem, i, elem ); + } ) ); + }, + + slice: function() { + return this.pushStack( slice.apply( this, arguments ) ); + }, + + first: function() { + return this.eq( 0 ); + }, + + last: function() { + return this.eq( -1 ); + }, + + even: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return ( i + 1 ) % 2; + } ) ); + }, + + odd: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return i % 2; + } ) ); + }, + + eq: function( i ) { + var len = this.length, + j = +i + ( i < 0 ? len : 0 ); + return this.pushStack( j >= 0 && j < len ? [ this[ j ] ] : [] ); + }, + + end: function() { + return this.prevObject || this.constructor(); + }, + + // For internal use only. + // Behaves like an Array's method, not like a jQuery method. + push: push, + sort: arr.sort, + splice: arr.splice +}; + +jQuery.extend = jQuery.fn.extend = function() { + var options, name, src, copy, copyIsArray, clone, + target = arguments[ 0 ] || {}, + i = 1, + length = arguments.length, + deep = false; + + // Handle a deep copy situation + if ( typeof target === "boolean" ) { + deep = target; + + // Skip the boolean and the target + target = arguments[ i ] || {}; + i++; + } + + // Handle case when target is a string or something (possible in deep copy) + if ( typeof target !== "object" && !isFunction( target ) ) { + target = {}; + } + + // Extend jQuery itself if only one argument is passed + if ( i === length ) { + target = this; + i--; + } + + for ( ; i < length; i++ ) { + + // Only deal with non-null/undefined values + if ( ( options = arguments[ i ] ) != null ) { + + // Extend the base object + for ( name in options ) { + copy = options[ name ]; + + // Prevent Object.prototype pollution + // Prevent never-ending loop + if ( name === "__proto__" || target === copy ) { + continue; + } + + // Recurse if we're merging plain objects or arrays + if ( deep && copy && ( jQuery.isPlainObject( copy ) || + ( copyIsArray = Array.isArray( copy ) ) ) ) { + src = target[ name ]; + + // Ensure proper type for the source value + if ( copyIsArray && !Array.isArray( src ) ) { + clone = []; + } else if ( !copyIsArray && !jQuery.isPlainObject( src ) ) { + clone = {}; + } else { + clone = src; + } + copyIsArray = false; + + // Never move original objects, clone them + target[ name ] = jQuery.extend( deep, clone, copy ); + + // Don't bring in undefined values + } else if ( copy !== undefined ) { + target[ name ] = copy; + } + } + } + } + + // Return the modified object + return target; +}; + +jQuery.extend( { + + // Unique for each copy of jQuery on the page + expando: "jQuery" + ( version + Math.random() ).replace( /\D/g, "" ), + + // Assume jQuery is ready without the ready module + isReady: true, + + error: function( msg ) { + throw new Error( msg ); + }, + + noop: function() {}, + + isPlainObject: function( obj ) { + var proto, Ctor; + + // Detect obvious negatives + // Use toString instead of jQuery.type to catch host objects + if ( !obj || toString.call( obj ) !== "[object Object]" ) { + return false; + } + + proto = getProto( obj ); + + // Objects with no prototype (e.g., `Object.create( null )`) are plain + if ( !proto ) { + return true; + } + + // Objects with prototype are plain iff they were constructed by a global Object function + Ctor = hasOwn.call( proto, "constructor" ) && proto.constructor; + return typeof Ctor === "function" && fnToString.call( Ctor ) === ObjectFunctionString; + }, + + isEmptyObject: function( obj ) { + var name; + + for ( name in obj ) { + return false; + } + return true; + }, + + // Evaluates a script in a provided context; falls back to the global one + // if not specified. + globalEval: function( code, options, doc ) { + DOMEval( code, { nonce: options && options.nonce }, doc ); + }, + + each: function( obj, callback ) { + var length, i = 0; + + if ( isArrayLike( obj ) ) { + length = obj.length; + for ( ; i < length; i++ ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } else { + for ( i in obj ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } + + return obj; + }, + + // results is for internal usage only + makeArray: function( arr, results ) { + var ret = results || []; + + if ( arr != null ) { + if ( isArrayLike( Object( arr ) ) ) { + jQuery.merge( ret, + typeof arr === "string" ? + [ arr ] : arr + ); + } else { + push.call( ret, arr ); + } + } + + return ret; + }, + + inArray: function( elem, arr, i ) { + return arr == null ? -1 : indexOf.call( arr, elem, i ); + }, + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + merge: function( first, second ) { + var len = +second.length, + j = 0, + i = first.length; + + for ( ; j < len; j++ ) { + first[ i++ ] = second[ j ]; + } + + first.length = i; + + return first; + }, + + grep: function( elems, callback, invert ) { + var callbackInverse, + matches = [], + i = 0, + length = elems.length, + callbackExpect = !invert; + + // Go through the array, only saving the items + // that pass the validator function + for ( ; i < length; i++ ) { + callbackInverse = !callback( elems[ i ], i ); + if ( callbackInverse !== callbackExpect ) { + matches.push( elems[ i ] ); + } + } + + return matches; + }, + + // arg is for internal usage only + map: function( elems, callback, arg ) { + var length, value, + i = 0, + ret = []; + + // Go through the array, translating each of the items to their new values + if ( isArrayLike( elems ) ) { + length = elems.length; + for ( ; i < length; i++ ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + + // Go through every key on the object, + } else { + for ( i in elems ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + } + + // Flatten any nested arrays + return flat( ret ); + }, + + // A global GUID counter for objects + guid: 1, + + // jQuery.support is not used in Core but other projects attach their + // properties to it so it needs to exist. + support: support +} ); + +if ( typeof Symbol === "function" ) { + jQuery.fn[ Symbol.iterator ] = arr[ Symbol.iterator ]; +} + +// Populate the class2type map +jQuery.each( "Boolean Number String Function Array Date RegExp Object Error Symbol".split( " " ), + function( _i, name ) { + class2type[ "[object " + name + "]" ] = name.toLowerCase(); + } ); + +function isArrayLike( obj ) { + + // Support: real iOS 8.2 only (not reproducible in simulator) + // `in` check used to prevent JIT error (gh-2145) + // hasOwn isn't used here due to false negatives + // regarding Nodelist length in IE + var length = !!obj && "length" in obj && obj.length, + type = toType( obj ); + + if ( isFunction( obj ) || isWindow( obj ) ) { + return false; + } + + return type === "array" || length === 0 || + typeof length === "number" && length > 0 && ( length - 1 ) in obj; +} +var Sizzle = +/*! + * Sizzle CSS Selector Engine v2.3.6 + * https://sizzlejs.com/ + * + * Copyright JS Foundation and other contributors + * Released under the MIT license + * https://js.foundation/ + * + * Date: 2021-02-16 + */ +( function( window ) { +var i, + support, + Expr, + getText, + isXML, + tokenize, + compile, + select, + outermostContext, + sortInput, + hasDuplicate, + + // Local document vars + setDocument, + document, + docElem, + documentIsHTML, + rbuggyQSA, + rbuggyMatches, + matches, + contains, + + // Instance-specific data + expando = "sizzle" + 1 * new Date(), + preferredDoc = window.document, + dirruns = 0, + done = 0, + classCache = createCache(), + tokenCache = createCache(), + compilerCache = createCache(), + nonnativeSelectorCache = createCache(), + sortOrder = function( a, b ) { + if ( a === b ) { + hasDuplicate = true; + } + return 0; + }, + + // Instance methods + hasOwn = ( {} ).hasOwnProperty, + arr = [], + pop = arr.pop, + pushNative = arr.push, + push = arr.push, + slice = arr.slice, + + // Use a stripped-down indexOf as it's faster than native + // https://jsperf.com/thor-indexof-vs-for/5 + indexOf = function( list, elem ) { + var i = 0, + len = list.length; + for ( ; i < len; i++ ) { + if ( list[ i ] === elem ) { + return i; + } + } + return -1; + }, + + booleans = "checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|" + + "ismap|loop|multiple|open|readonly|required|scoped", + + // Regular expressions + + // http://www.w3.org/TR/css3-selectors/#whitespace + whitespace = "[\\x20\\t\\r\\n\\f]", + + // https://www.w3.org/TR/css-syntax-3/#ident-token-diagram + identifier = "(?:\\\\[\\da-fA-F]{1,6}" + whitespace + + "?|\\\\[^\\r\\n\\f]|[\\w-]|[^\0-\\x7f])+", + + // Attribute selectors: http://www.w3.org/TR/selectors/#attribute-selectors + attributes = "\\[" + whitespace + "*(" + identifier + ")(?:" + whitespace + + + // Operator (capture 2) + "*([*^$|!~]?=)" + whitespace + + + // "Attribute values must be CSS identifiers [capture 5] + // or strings [capture 3 or capture 4]" + "*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|(" + identifier + "))|)" + + whitespace + "*\\]", + + pseudos = ":(" + identifier + ")(?:\\((" + + + // To reduce the number of selectors needing tokenize in the preFilter, prefer arguments: + // 1. quoted (capture 3; capture 4 or capture 5) + "('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|" + + + // 2. simple (capture 6) + "((?:\\\\.|[^\\\\()[\\]]|" + attributes + ")*)|" + + + // 3. anything else (capture 2) + ".*" + + ")\\)|)", + + // Leading and non-escaped trailing whitespace, capturing some non-whitespace characters preceding the latter + rwhitespace = new RegExp( whitespace + "+", "g" ), + rtrim = new RegExp( "^" + whitespace + "+|((?:^|[^\\\\])(?:\\\\.)*)" + + whitespace + "+$", "g" ), + + rcomma = new RegExp( "^" + whitespace + "*," + whitespace + "*" ), + rcombinators = new RegExp( "^" + whitespace + "*([>+~]|" + whitespace + ")" + whitespace + + "*" ), + rdescend = new RegExp( whitespace + "|>" ), + + rpseudo = new RegExp( pseudos ), + ridentifier = new RegExp( "^" + identifier + "$" ), + + matchExpr = { + "ID": new RegExp( "^#(" + identifier + ")" ), + "CLASS": new RegExp( "^\\.(" + identifier + ")" ), + "TAG": new RegExp( "^(" + identifier + "|[*])" ), + "ATTR": new RegExp( "^" + attributes ), + "PSEUDO": new RegExp( "^" + pseudos ), + "CHILD": new RegExp( "^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\(" + + whitespace + "*(even|odd|(([+-]|)(\\d*)n|)" + whitespace + "*(?:([+-]|)" + + whitespace + "*(\\d+)|))" + whitespace + "*\\)|)", "i" ), + "bool": new RegExp( "^(?:" + booleans + ")$", "i" ), + + // For use in libraries implementing .is() + // We use this for POS matching in `select` + "needsContext": new RegExp( "^" + whitespace + + "*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\(" + whitespace + + "*((?:-\\d)?\\d*)" + whitespace + "*\\)|)(?=[^-]|$)", "i" ) + }, + + rhtml = /HTML$/i, + rinputs = /^(?:input|select|textarea|button)$/i, + rheader = /^h\d$/i, + + rnative = /^[^{]+\{\s*\[native \w/, + + // Easily-parseable/retrievable ID or TAG or CLASS selectors + rquickExpr = /^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/, + + rsibling = /[+~]/, + + // CSS escapes + // http://www.w3.org/TR/CSS21/syndata.html#escaped-characters + runescape = new RegExp( "\\\\[\\da-fA-F]{1,6}" + whitespace + "?|\\\\([^\\r\\n\\f])", "g" ), + funescape = function( escape, nonHex ) { + var high = "0x" + escape.slice( 1 ) - 0x10000; + + return nonHex ? + + // Strip the backslash prefix from a non-hex escape sequence + nonHex : + + // Replace a hexadecimal escape sequence with the encoded Unicode code point + // Support: IE <=11+ + // For values outside the Basic Multilingual Plane (BMP), manually construct a + // surrogate pair + high < 0 ? + String.fromCharCode( high + 0x10000 ) : + String.fromCharCode( high >> 10 | 0xD800, high & 0x3FF | 0xDC00 ); + }, + + // CSS string/identifier serialization + // https://drafts.csswg.org/cssom/#common-serializing-idioms + rcssescape = /([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g, + fcssescape = function( ch, asCodePoint ) { + if ( asCodePoint ) { + + // U+0000 NULL becomes U+FFFD REPLACEMENT CHARACTER + if ( ch === "\0" ) { + return "\uFFFD"; + } + + // Control characters and (dependent upon position) numbers get escaped as code points + return ch.slice( 0, -1 ) + "\\" + + ch.charCodeAt( ch.length - 1 ).toString( 16 ) + " "; + } + + // Other potentially-special ASCII characters get backslash-escaped + return "\\" + ch; + }, + + // Used for iframes + // See setDocument() + // Removing the function wrapper causes a "Permission Denied" + // error in IE + unloadHandler = function() { + setDocument(); + }, + + inDisabledFieldset = addCombinator( + function( elem ) { + return elem.disabled === true && elem.nodeName.toLowerCase() === "fieldset"; + }, + { dir: "parentNode", next: "legend" } + ); + +// Optimize for push.apply( _, NodeList ) +try { + push.apply( + ( arr = slice.call( preferredDoc.childNodes ) ), + preferredDoc.childNodes + ); + + // Support: Android<4.0 + // Detect silently failing push.apply + // eslint-disable-next-line no-unused-expressions + arr[ preferredDoc.childNodes.length ].nodeType; +} catch ( e ) { + push = { apply: arr.length ? + + // Leverage slice if possible + function( target, els ) { + pushNative.apply( target, slice.call( els ) ); + } : + + // Support: IE<9 + // Otherwise append directly + function( target, els ) { + var j = target.length, + i = 0; + + // Can't trust NodeList.length + while ( ( target[ j++ ] = els[ i++ ] ) ) {} + target.length = j - 1; + } + }; +} + +function Sizzle( selector, context, results, seed ) { + var m, i, elem, nid, match, groups, newSelector, + newContext = context && context.ownerDocument, + + // nodeType defaults to 9, since context defaults to document + nodeType = context ? context.nodeType : 9; + + results = results || []; + + // Return early from calls with invalid selector or context + if ( typeof selector !== "string" || !selector || + nodeType !== 1 && nodeType !== 9 && nodeType !== 11 ) { + + return results; + } + + // Try to shortcut find operations (as opposed to filters) in HTML documents + if ( !seed ) { + setDocument( context ); + context = context || document; + + if ( documentIsHTML ) { + + // If the selector is sufficiently simple, try using a "get*By*" DOM method + // (excepting DocumentFragment context, where the methods don't exist) + if ( nodeType !== 11 && ( match = rquickExpr.exec( selector ) ) ) { + + // ID selector + if ( ( m = match[ 1 ] ) ) { + + // Document context + if ( nodeType === 9 ) { + if ( ( elem = context.getElementById( m ) ) ) { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( elem.id === m ) { + results.push( elem ); + return results; + } + } else { + return results; + } + + // Element context + } else { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( newContext && ( elem = newContext.getElementById( m ) ) && + contains( context, elem ) && + elem.id === m ) { + + results.push( elem ); + return results; + } + } + + // Type selector + } else if ( match[ 2 ] ) { + push.apply( results, context.getElementsByTagName( selector ) ); + return results; + + // Class selector + } else if ( ( m = match[ 3 ] ) && support.getElementsByClassName && + context.getElementsByClassName ) { + + push.apply( results, context.getElementsByClassName( m ) ); + return results; + } + } + + // Take advantage of querySelectorAll + if ( support.qsa && + !nonnativeSelectorCache[ selector + " " ] && + ( !rbuggyQSA || !rbuggyQSA.test( selector ) ) && + + // Support: IE 8 only + // Exclude object elements + ( nodeType !== 1 || context.nodeName.toLowerCase() !== "object" ) ) { + + newSelector = selector; + newContext = context; + + // qSA considers elements outside a scoping root when evaluating child or + // descendant combinators, which is not what we want. + // In such cases, we work around the behavior by prefixing every selector in the + // list with an ID selector referencing the scope context. + // The technique has to be used as well when a leading combinator is used + // as such selectors are not recognized by querySelectorAll. + // Thanks to Andrew Dupont for this technique. + if ( nodeType === 1 && + ( rdescend.test( selector ) || rcombinators.test( selector ) ) ) { + + // Expand context for sibling selectors + newContext = rsibling.test( selector ) && testContext( context.parentNode ) || + context; + + // We can use :scope instead of the ID hack if the browser + // supports it & if we're not changing the context. + if ( newContext !== context || !support.scope ) { + + // Capture the context ID, setting it first if necessary + if ( ( nid = context.getAttribute( "id" ) ) ) { + nid = nid.replace( rcssescape, fcssescape ); + } else { + context.setAttribute( "id", ( nid = expando ) ); + } + } + + // Prefix every selector in the list + groups = tokenize( selector ); + i = groups.length; + while ( i-- ) { + groups[ i ] = ( nid ? "#" + nid : ":scope" ) + " " + + toSelector( groups[ i ] ); + } + newSelector = groups.join( "," ); + } + + try { + push.apply( results, + newContext.querySelectorAll( newSelector ) + ); + return results; + } catch ( qsaError ) { + nonnativeSelectorCache( selector, true ); + } finally { + if ( nid === expando ) { + context.removeAttribute( "id" ); + } + } + } + } + } + + // All others + return select( selector.replace( rtrim, "$1" ), context, results, seed ); +} + +/** + * Create key-value caches of limited size + * @returns {function(string, object)} Returns the Object data after storing it on itself with + * property name the (space-suffixed) string and (if the cache is larger than Expr.cacheLength) + * deleting the oldest entry + */ +function createCache() { + var keys = []; + + function cache( key, value ) { + + // Use (key + " ") to avoid collision with native prototype properties (see Issue #157) + if ( keys.push( key + " " ) > Expr.cacheLength ) { + + // Only keep the most recent entries + delete cache[ keys.shift() ]; + } + return ( cache[ key + " " ] = value ); + } + return cache; +} + +/** + * Mark a function for special use by Sizzle + * @param {Function} fn The function to mark + */ +function markFunction( fn ) { + fn[ expando ] = true; + return fn; +} + +/** + * Support testing using an element + * @param {Function} fn Passed the created element and returns a boolean result + */ +function assert( fn ) { + var el = document.createElement( "fieldset" ); + + try { + return !!fn( el ); + } catch ( e ) { + return false; + } finally { + + // Remove from its parent by default + if ( el.parentNode ) { + el.parentNode.removeChild( el ); + } + + // release memory in IE + el = null; + } +} + +/** + * Adds the same handler for all of the specified attrs + * @param {String} attrs Pipe-separated list of attributes + * @param {Function} handler The method that will be applied + */ +function addHandle( attrs, handler ) { + var arr = attrs.split( "|" ), + i = arr.length; + + while ( i-- ) { + Expr.attrHandle[ arr[ i ] ] = handler; + } +} + +/** + * Checks document order of two siblings + * @param {Element} a + * @param {Element} b + * @returns {Number} Returns less than 0 if a precedes b, greater than 0 if a follows b + */ +function siblingCheck( a, b ) { + var cur = b && a, + diff = cur && a.nodeType === 1 && b.nodeType === 1 && + a.sourceIndex - b.sourceIndex; + + // Use IE sourceIndex if available on both nodes + if ( diff ) { + return diff; + } + + // Check if b follows a + if ( cur ) { + while ( ( cur = cur.nextSibling ) ) { + if ( cur === b ) { + return -1; + } + } + } + + return a ? 1 : -1; +} + +/** + * Returns a function to use in pseudos for input types + * @param {String} type + */ +function createInputPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for buttons + * @param {String} type + */ +function createButtonPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return ( name === "input" || name === "button" ) && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for :enabled/:disabled + * @param {Boolean} disabled true for :disabled; false for :enabled + */ +function createDisabledPseudo( disabled ) { + + // Known :disabled false positives: fieldset[disabled] > legend:nth-of-type(n+2) :can-disable + return function( elem ) { + + // Only certain elements can match :enabled or :disabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-enabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-disabled + if ( "form" in elem ) { + + // Check for inherited disabledness on relevant non-disabled elements: + // * listed form-associated elements in a disabled fieldset + // https://html.spec.whatwg.org/multipage/forms.html#category-listed + // https://html.spec.whatwg.org/multipage/forms.html#concept-fe-disabled + // * option elements in a disabled optgroup + // https://html.spec.whatwg.org/multipage/forms.html#concept-option-disabled + // All such elements have a "form" property. + if ( elem.parentNode && elem.disabled === false ) { + + // Option elements defer to a parent optgroup if present + if ( "label" in elem ) { + if ( "label" in elem.parentNode ) { + return elem.parentNode.disabled === disabled; + } else { + return elem.disabled === disabled; + } + } + + // Support: IE 6 - 11 + // Use the isDisabled shortcut property to check for disabled fieldset ancestors + return elem.isDisabled === disabled || + + // Where there is no isDisabled, check manually + /* jshint -W018 */ + elem.isDisabled !== !disabled && + inDisabledFieldset( elem ) === disabled; + } + + return elem.disabled === disabled; + + // Try to winnow out elements that can't be disabled before trusting the disabled property. + // Some victims get caught in our net (label, legend, menu, track), but it shouldn't + // even exist on them, let alone have a boolean value. + } else if ( "label" in elem ) { + return elem.disabled === disabled; + } + + // Remaining elements are neither :enabled nor :disabled + return false; + }; +} + +/** + * Returns a function to use in pseudos for positionals + * @param {Function} fn + */ +function createPositionalPseudo( fn ) { + return markFunction( function( argument ) { + argument = +argument; + return markFunction( function( seed, matches ) { + var j, + matchIndexes = fn( [], seed.length, argument ), + i = matchIndexes.length; + + // Match elements found at the specified indexes + while ( i-- ) { + if ( seed[ ( j = matchIndexes[ i ] ) ] ) { + seed[ j ] = !( matches[ j ] = seed[ j ] ); + } + } + } ); + } ); +} + +/** + * Checks a node for validity as a Sizzle context + * @param {Element|Object=} context + * @returns {Element|Object|Boolean} The input node if acceptable, otherwise a falsy value + */ +function testContext( context ) { + return context && typeof context.getElementsByTagName !== "undefined" && context; +} + +// Expose support vars for convenience +support = Sizzle.support = {}; + +/** + * Detects XML nodes + * @param {Element|Object} elem An element or a document + * @returns {Boolean} True iff elem is a non-HTML XML node + */ +isXML = Sizzle.isXML = function( elem ) { + var namespace = elem && elem.namespaceURI, + docElem = elem && ( elem.ownerDocument || elem ).documentElement; + + // Support: IE <=8 + // Assume HTML when documentElement doesn't yet exist, such as inside loading iframes + // https://bugs.jquery.com/ticket/4833 + return !rhtml.test( namespace || docElem && docElem.nodeName || "HTML" ); +}; + +/** + * Sets document-related variables once based on the current document + * @param {Element|Object} [doc] An element or document object to use to set the document + * @returns {Object} Returns the current document + */ +setDocument = Sizzle.setDocument = function( node ) { + var hasCompare, subWindow, + doc = node ? node.ownerDocument || node : preferredDoc; + + // Return early if doc is invalid or already selected + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( doc == document || doc.nodeType !== 9 || !doc.documentElement ) { + return document; + } + + // Update global variables + document = doc; + docElem = document.documentElement; + documentIsHTML = !isXML( document ); + + // Support: IE 9 - 11+, Edge 12 - 18+ + // Accessing iframe documents after unload throws "permission denied" errors (jQuery #13936) + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( preferredDoc != document && + ( subWindow = document.defaultView ) && subWindow.top !== subWindow ) { + + // Support: IE 11, Edge + if ( subWindow.addEventListener ) { + subWindow.addEventListener( "unload", unloadHandler, false ); + + // Support: IE 9 - 10 only + } else if ( subWindow.attachEvent ) { + subWindow.attachEvent( "onunload", unloadHandler ); + } + } + + // Support: IE 8 - 11+, Edge 12 - 18+, Chrome <=16 - 25 only, Firefox <=3.6 - 31 only, + // Safari 4 - 5 only, Opera <=11.6 - 12.x only + // IE/Edge & older browsers don't support the :scope pseudo-class. + // Support: Safari 6.0 only + // Safari 6.0 supports :scope but it's an alias of :root there. + support.scope = assert( function( el ) { + docElem.appendChild( el ).appendChild( document.createElement( "div" ) ); + return typeof el.querySelectorAll !== "undefined" && + !el.querySelectorAll( ":scope fieldset div" ).length; + } ); + + /* Attributes + ---------------------------------------------------------------------- */ + + // Support: IE<8 + // Verify that getAttribute really returns attributes and not properties + // (excepting IE8 booleans) + support.attributes = assert( function( el ) { + el.className = "i"; + return !el.getAttribute( "className" ); + } ); + + /* getElement(s)By* + ---------------------------------------------------------------------- */ + + // Check if getElementsByTagName("*") returns only elements + support.getElementsByTagName = assert( function( el ) { + el.appendChild( document.createComment( "" ) ); + return !el.getElementsByTagName( "*" ).length; + } ); + + // Support: IE<9 + support.getElementsByClassName = rnative.test( document.getElementsByClassName ); + + // Support: IE<10 + // Check if getElementById returns elements by name + // The broken getElementById methods don't pick up programmatically-set names, + // so use a roundabout getElementsByName test + support.getById = assert( function( el ) { + docElem.appendChild( el ).id = expando; + return !document.getElementsByName || !document.getElementsByName( expando ).length; + } ); + + // ID filter and find + if ( support.getById ) { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + return elem.getAttribute( "id" ) === attrId; + }; + }; + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var elem = context.getElementById( id ); + return elem ? [ elem ] : []; + } + }; + } else { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + var node = typeof elem.getAttributeNode !== "undefined" && + elem.getAttributeNode( "id" ); + return node && node.value === attrId; + }; + }; + + // Support: IE 6 - 7 only + // getElementById is not reliable as a find shortcut + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var node, i, elems, + elem = context.getElementById( id ); + + if ( elem ) { + + // Verify the id attribute + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + + // Fall back on getElementsByName + elems = context.getElementsByName( id ); + i = 0; + while ( ( elem = elems[ i++ ] ) ) { + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + } + } + + return []; + } + }; + } + + // Tag + Expr.find[ "TAG" ] = support.getElementsByTagName ? + function( tag, context ) { + if ( typeof context.getElementsByTagName !== "undefined" ) { + return context.getElementsByTagName( tag ); + + // DocumentFragment nodes don't have gEBTN + } else if ( support.qsa ) { + return context.querySelectorAll( tag ); + } + } : + + function( tag, context ) { + var elem, + tmp = [], + i = 0, + + // By happy coincidence, a (broken) gEBTN appears on DocumentFragment nodes too + results = context.getElementsByTagName( tag ); + + // Filter out possible comments + if ( tag === "*" ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem.nodeType === 1 ) { + tmp.push( elem ); + } + } + + return tmp; + } + return results; + }; + + // Class + Expr.find[ "CLASS" ] = support.getElementsByClassName && function( className, context ) { + if ( typeof context.getElementsByClassName !== "undefined" && documentIsHTML ) { + return context.getElementsByClassName( className ); + } + }; + + /* QSA/matchesSelector + ---------------------------------------------------------------------- */ + + // QSA and matchesSelector support + + // matchesSelector(:active) reports false when true (IE9/Opera 11.5) + rbuggyMatches = []; + + // qSa(:focus) reports false when true (Chrome 21) + // We allow this because of a bug in IE8/9 that throws an error + // whenever `document.activeElement` is accessed on an iframe + // So, we allow :focus to pass through QSA all the time to avoid the IE error + // See https://bugs.jquery.com/ticket/13378 + rbuggyQSA = []; + + if ( ( support.qsa = rnative.test( document.querySelectorAll ) ) ) { + + // Build QSA regex + // Regex strategy adopted from Diego Perini + assert( function( el ) { + + var input; + + // Select is set to empty string on purpose + // This is to test IE's treatment of not explicitly + // setting a boolean content attribute, + // since its presence should be enough + // https://bugs.jquery.com/ticket/12359 + docElem.appendChild( el ).innerHTML = "" + + ""; + + // Support: IE8, Opera 11-12.16 + // Nothing should be selected when empty strings follow ^= or $= or *= + // The test attribute must be unknown in Opera but "safe" for WinRT + // https://msdn.microsoft.com/en-us/library/ie/hh465388.aspx#attribute_section + if ( el.querySelectorAll( "[msallowcapture^='']" ).length ) { + rbuggyQSA.push( "[*^$]=" + whitespace + "*(?:''|\"\")" ); + } + + // Support: IE8 + // Boolean attributes and "value" are not treated correctly + if ( !el.querySelectorAll( "[selected]" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*(?:value|" + booleans + ")" ); + } + + // Support: Chrome<29, Android<4.4, Safari<7.0+, iOS<7.0+, PhantomJS<1.9.8+ + if ( !el.querySelectorAll( "[id~=" + expando + "-]" ).length ) { + rbuggyQSA.push( "~=" ); + } + + // Support: IE 11+, Edge 15 - 18+ + // IE 11/Edge don't find elements on a `[name='']` query in some cases. + // Adding a temporary attribute to the document before the selection works + // around the issue. + // Interestingly, IE 10 & older don't seem to have the issue. + input = document.createElement( "input" ); + input.setAttribute( "name", "" ); + el.appendChild( input ); + if ( !el.querySelectorAll( "[name='']" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*name" + whitespace + "*=" + + whitespace + "*(?:''|\"\")" ); + } + + // Webkit/Opera - :checked should return selected option elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + // IE8 throws error here and will not see later tests + if ( !el.querySelectorAll( ":checked" ).length ) { + rbuggyQSA.push( ":checked" ); + } + + // Support: Safari 8+, iOS 8+ + // https://bugs.webkit.org/show_bug.cgi?id=136851 + // In-page `selector#id sibling-combinator selector` fails + if ( !el.querySelectorAll( "a#" + expando + "+*" ).length ) { + rbuggyQSA.push( ".#.+[+~]" ); + } + + // Support: Firefox <=3.6 - 5 only + // Old Firefox doesn't throw on a badly-escaped identifier. + el.querySelectorAll( "\\\f" ); + rbuggyQSA.push( "[\\r\\n\\f]" ); + } ); + + assert( function( el ) { + el.innerHTML = "" + + ""; + + // Support: Windows 8 Native Apps + // The type and name attributes are restricted during .innerHTML assignment + var input = document.createElement( "input" ); + input.setAttribute( "type", "hidden" ); + el.appendChild( input ).setAttribute( "name", "D" ); + + // Support: IE8 + // Enforce case-sensitivity of name attribute + if ( el.querySelectorAll( "[name=d]" ).length ) { + rbuggyQSA.push( "name" + whitespace + "*[*^$|!~]?=" ); + } + + // FF 3.5 - :enabled/:disabled and hidden elements (hidden elements are still enabled) + // IE8 throws error here and will not see later tests + if ( el.querySelectorAll( ":enabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: IE9-11+ + // IE's :disabled selector does not pick up the children of disabled fieldsets + docElem.appendChild( el ).disabled = true; + if ( el.querySelectorAll( ":disabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: Opera 10 - 11 only + // Opera 10-11 does not throw on post-comma invalid pseudos + el.querySelectorAll( "*,:x" ); + rbuggyQSA.push( ",.*:" ); + } ); + } + + if ( ( support.matchesSelector = rnative.test( ( matches = docElem.matches || + docElem.webkitMatchesSelector || + docElem.mozMatchesSelector || + docElem.oMatchesSelector || + docElem.msMatchesSelector ) ) ) ) { + + assert( function( el ) { + + // Check to see if it's possible to do matchesSelector + // on a disconnected node (IE 9) + support.disconnectedMatch = matches.call( el, "*" ); + + // This should fail with an exception + // Gecko does not error, returns false instead + matches.call( el, "[s!='']:x" ); + rbuggyMatches.push( "!=", pseudos ); + } ); + } + + rbuggyQSA = rbuggyQSA.length && new RegExp( rbuggyQSA.join( "|" ) ); + rbuggyMatches = rbuggyMatches.length && new RegExp( rbuggyMatches.join( "|" ) ); + + /* Contains + ---------------------------------------------------------------------- */ + hasCompare = rnative.test( docElem.compareDocumentPosition ); + + // Element contains another + // Purposefully self-exclusive + // As in, an element does not contain itself + contains = hasCompare || rnative.test( docElem.contains ) ? + function( a, b ) { + var adown = a.nodeType === 9 ? a.documentElement : a, + bup = b && b.parentNode; + return a === bup || !!( bup && bup.nodeType === 1 && ( + adown.contains ? + adown.contains( bup ) : + a.compareDocumentPosition && a.compareDocumentPosition( bup ) & 16 + ) ); + } : + function( a, b ) { + if ( b ) { + while ( ( b = b.parentNode ) ) { + if ( b === a ) { + return true; + } + } + } + return false; + }; + + /* Sorting + ---------------------------------------------------------------------- */ + + // Document order sorting + sortOrder = hasCompare ? + function( a, b ) { + + // Flag for duplicate removal + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + // Sort on method existence if only one input has compareDocumentPosition + var compare = !a.compareDocumentPosition - !b.compareDocumentPosition; + if ( compare ) { + return compare; + } + + // Calculate position if both inputs belong to the same document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + compare = ( a.ownerDocument || a ) == ( b.ownerDocument || b ) ? + a.compareDocumentPosition( b ) : + + // Otherwise we know they are disconnected + 1; + + // Disconnected nodes + if ( compare & 1 || + ( !support.sortDetached && b.compareDocumentPosition( a ) === compare ) ) { + + // Choose the first element that is related to our preferred document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( a == document || a.ownerDocument == preferredDoc && + contains( preferredDoc, a ) ) { + return -1; + } + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( b == document || b.ownerDocument == preferredDoc && + contains( preferredDoc, b ) ) { + return 1; + } + + // Maintain original order + return sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + } + + return compare & 4 ? -1 : 1; + } : + function( a, b ) { + + // Exit early if the nodes are identical + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + var cur, + i = 0, + aup = a.parentNode, + bup = b.parentNode, + ap = [ a ], + bp = [ b ]; + + // Parentless nodes are either documents or disconnected + if ( !aup || !bup ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + return a == document ? -1 : + b == document ? 1 : + /* eslint-enable eqeqeq */ + aup ? -1 : + bup ? 1 : + sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + + // If the nodes are siblings, we can do a quick check + } else if ( aup === bup ) { + return siblingCheck( a, b ); + } + + // Otherwise we need full lists of their ancestors for comparison + cur = a; + while ( ( cur = cur.parentNode ) ) { + ap.unshift( cur ); + } + cur = b; + while ( ( cur = cur.parentNode ) ) { + bp.unshift( cur ); + } + + // Walk down the tree looking for a discrepancy + while ( ap[ i ] === bp[ i ] ) { + i++; + } + + return i ? + + // Do a sibling check if the nodes have a common ancestor + siblingCheck( ap[ i ], bp[ i ] ) : + + // Otherwise nodes in our document sort first + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + ap[ i ] == preferredDoc ? -1 : + bp[ i ] == preferredDoc ? 1 : + /* eslint-enable eqeqeq */ + 0; + }; + + return document; +}; + +Sizzle.matches = function( expr, elements ) { + return Sizzle( expr, null, null, elements ); +}; + +Sizzle.matchesSelector = function( elem, expr ) { + setDocument( elem ); + + if ( support.matchesSelector && documentIsHTML && + !nonnativeSelectorCache[ expr + " " ] && + ( !rbuggyMatches || !rbuggyMatches.test( expr ) ) && + ( !rbuggyQSA || !rbuggyQSA.test( expr ) ) ) { + + try { + var ret = matches.call( elem, expr ); + + // IE 9's matchesSelector returns false on disconnected nodes + if ( ret || support.disconnectedMatch || + + // As well, disconnected nodes are said to be in a document + // fragment in IE 9 + elem.document && elem.document.nodeType !== 11 ) { + return ret; + } + } catch ( e ) { + nonnativeSelectorCache( expr, true ); + } + } + + return Sizzle( expr, document, null, [ elem ] ).length > 0; +}; + +Sizzle.contains = function( context, elem ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( context.ownerDocument || context ) != document ) { + setDocument( context ); + } + return contains( context, elem ); +}; + +Sizzle.attr = function( elem, name ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( elem.ownerDocument || elem ) != document ) { + setDocument( elem ); + } + + var fn = Expr.attrHandle[ name.toLowerCase() ], + + // Don't get fooled by Object.prototype properties (jQuery #13807) + val = fn && hasOwn.call( Expr.attrHandle, name.toLowerCase() ) ? + fn( elem, name, !documentIsHTML ) : + undefined; + + return val !== undefined ? + val : + support.attributes || !documentIsHTML ? + elem.getAttribute( name ) : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; +}; + +Sizzle.escape = function( sel ) { + return ( sel + "" ).replace( rcssescape, fcssescape ); +}; + +Sizzle.error = function( msg ) { + throw new Error( "Syntax error, unrecognized expression: " + msg ); +}; + +/** + * Document sorting and removing duplicates + * @param {ArrayLike} results + */ +Sizzle.uniqueSort = function( results ) { + var elem, + duplicates = [], + j = 0, + i = 0; + + // Unless we *know* we can detect duplicates, assume their presence + hasDuplicate = !support.detectDuplicates; + sortInput = !support.sortStable && results.slice( 0 ); + results.sort( sortOrder ); + + if ( hasDuplicate ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem === results[ i ] ) { + j = duplicates.push( i ); + } + } + while ( j-- ) { + results.splice( duplicates[ j ], 1 ); + } + } + + // Clear input after sorting to release objects + // See https://github.com/jquery/sizzle/pull/225 + sortInput = null; + + return results; +}; + +/** + * Utility function for retrieving the text value of an array of DOM nodes + * @param {Array|Element} elem + */ +getText = Sizzle.getText = function( elem ) { + var node, + ret = "", + i = 0, + nodeType = elem.nodeType; + + if ( !nodeType ) { + + // If no nodeType, this is expected to be an array + while ( ( node = elem[ i++ ] ) ) { + + // Do not traverse comment nodes + ret += getText( node ); + } + } else if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) { + + // Use textContent for elements + // innerText usage removed for consistency of new lines (jQuery #11153) + if ( typeof elem.textContent === "string" ) { + return elem.textContent; + } else { + + // Traverse its children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + ret += getText( elem ); + } + } + } else if ( nodeType === 3 || nodeType === 4 ) { + return elem.nodeValue; + } + + // Do not include comment or processing instruction nodes + + return ret; +}; + +Expr = Sizzle.selectors = { + + // Can be adjusted by the user + cacheLength: 50, + + createPseudo: markFunction, + + match: matchExpr, + + attrHandle: {}, + + find: {}, + + relative: { + ">": { dir: "parentNode", first: true }, + " ": { dir: "parentNode" }, + "+": { dir: "previousSibling", first: true }, + "~": { dir: "previousSibling" } + }, + + preFilter: { + "ATTR": function( match ) { + match[ 1 ] = match[ 1 ].replace( runescape, funescape ); + + // Move the given value to match[3] whether quoted or unquoted + match[ 3 ] = ( match[ 3 ] || match[ 4 ] || + match[ 5 ] || "" ).replace( runescape, funescape ); + + if ( match[ 2 ] === "~=" ) { + match[ 3 ] = " " + match[ 3 ] + " "; + } + + return match.slice( 0, 4 ); + }, + + "CHILD": function( match ) { + + /* matches from matchExpr["CHILD"] + 1 type (only|nth|...) + 2 what (child|of-type) + 3 argument (even|odd|\d*|\d*n([+-]\d+)?|...) + 4 xn-component of xn+y argument ([+-]?\d*n|) + 5 sign of xn-component + 6 x of xn-component + 7 sign of y-component + 8 y of y-component + */ + match[ 1 ] = match[ 1 ].toLowerCase(); + + if ( match[ 1 ].slice( 0, 3 ) === "nth" ) { + + // nth-* requires argument + if ( !match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + // numeric x and y parameters for Expr.filter.CHILD + // remember that false/true cast respectively to 0/1 + match[ 4 ] = +( match[ 4 ] ? + match[ 5 ] + ( match[ 6 ] || 1 ) : + 2 * ( match[ 3 ] === "even" || match[ 3 ] === "odd" ) ); + match[ 5 ] = +( ( match[ 7 ] + match[ 8 ] ) || match[ 3 ] === "odd" ); + + // other types prohibit arguments + } else if ( match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + return match; + }, + + "PSEUDO": function( match ) { + var excess, + unquoted = !match[ 6 ] && match[ 2 ]; + + if ( matchExpr[ "CHILD" ].test( match[ 0 ] ) ) { + return null; + } + + // Accept quoted arguments as-is + if ( match[ 3 ] ) { + match[ 2 ] = match[ 4 ] || match[ 5 ] || ""; + + // Strip excess characters from unquoted arguments + } else if ( unquoted && rpseudo.test( unquoted ) && + + // Get excess from tokenize (recursively) + ( excess = tokenize( unquoted, true ) ) && + + // advance to the next closing parenthesis + ( excess = unquoted.indexOf( ")", unquoted.length - excess ) - unquoted.length ) ) { + + // excess is a negative index + match[ 0 ] = match[ 0 ].slice( 0, excess ); + match[ 2 ] = unquoted.slice( 0, excess ); + } + + // Return only captures needed by the pseudo filter method (type and argument) + return match.slice( 0, 3 ); + } + }, + + filter: { + + "TAG": function( nodeNameSelector ) { + var nodeName = nodeNameSelector.replace( runescape, funescape ).toLowerCase(); + return nodeNameSelector === "*" ? + function() { + return true; + } : + function( elem ) { + return elem.nodeName && elem.nodeName.toLowerCase() === nodeName; + }; + }, + + "CLASS": function( className ) { + var pattern = classCache[ className + " " ]; + + return pattern || + ( pattern = new RegExp( "(^|" + whitespace + + ")" + className + "(" + whitespace + "|$)" ) ) && classCache( + className, function( elem ) { + return pattern.test( + typeof elem.className === "string" && elem.className || + typeof elem.getAttribute !== "undefined" && + elem.getAttribute( "class" ) || + "" + ); + } ); + }, + + "ATTR": function( name, operator, check ) { + return function( elem ) { + var result = Sizzle.attr( elem, name ); + + if ( result == null ) { + return operator === "!="; + } + if ( !operator ) { + return true; + } + + result += ""; + + /* eslint-disable max-len */ + + return operator === "=" ? result === check : + operator === "!=" ? result !== check : + operator === "^=" ? check && result.indexOf( check ) === 0 : + operator === "*=" ? check && result.indexOf( check ) > -1 : + operator === "$=" ? check && result.slice( -check.length ) === check : + operator === "~=" ? ( " " + result.replace( rwhitespace, " " ) + " " ).indexOf( check ) > -1 : + operator === "|=" ? result === check || result.slice( 0, check.length + 1 ) === check + "-" : + false; + /* eslint-enable max-len */ + + }; + }, + + "CHILD": function( type, what, _argument, first, last ) { + var simple = type.slice( 0, 3 ) !== "nth", + forward = type.slice( -4 ) !== "last", + ofType = what === "of-type"; + + return first === 1 && last === 0 ? + + // Shortcut for :nth-*(n) + function( elem ) { + return !!elem.parentNode; + } : + + function( elem, _context, xml ) { + var cache, uniqueCache, outerCache, node, nodeIndex, start, + dir = simple !== forward ? "nextSibling" : "previousSibling", + parent = elem.parentNode, + name = ofType && elem.nodeName.toLowerCase(), + useCache = !xml && !ofType, + diff = false; + + if ( parent ) { + + // :(first|last|only)-(child|of-type) + if ( simple ) { + while ( dir ) { + node = elem; + while ( ( node = node[ dir ] ) ) { + if ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) { + + return false; + } + } + + // Reverse direction for :only-* (if we haven't yet done so) + start = dir = type === "only" && !start && "nextSibling"; + } + return true; + } + + start = [ forward ? parent.firstChild : parent.lastChild ]; + + // non-xml :nth-child(...) stores cache data on `parent` + if ( forward && useCache ) { + + // Seek `elem` from a previously-cached index + + // ...in a gzip-friendly way + node = parent; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex && cache[ 2 ]; + node = nodeIndex && parent.childNodes[ nodeIndex ]; + + while ( ( node = ++nodeIndex && node && node[ dir ] || + + // Fallback to seeking `elem` from the start + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + // When found, cache indexes on `parent` and break + if ( node.nodeType === 1 && ++diff && node === elem ) { + uniqueCache[ type ] = [ dirruns, nodeIndex, diff ]; + break; + } + } + + } else { + + // Use previously-cached element index if available + if ( useCache ) { + + // ...in a gzip-friendly way + node = elem; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex; + } + + // xml :nth-child(...) + // or :nth-last-child(...) or :nth(-last)?-of-type(...) + if ( diff === false ) { + + // Use the same loop as above to seek `elem` from the start + while ( ( node = ++nodeIndex && node && node[ dir ] || + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + if ( ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) && + ++diff ) { + + // Cache the index of each encountered element + if ( useCache ) { + outerCache = node[ expando ] || + ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + uniqueCache[ type ] = [ dirruns, diff ]; + } + + if ( node === elem ) { + break; + } + } + } + } + } + + // Incorporate the offset, then check against cycle size + diff -= last; + return diff === first || ( diff % first === 0 && diff / first >= 0 ); + } + }; + }, + + "PSEUDO": function( pseudo, argument ) { + + // pseudo-class names are case-insensitive + // http://www.w3.org/TR/selectors/#pseudo-classes + // Prioritize by case sensitivity in case custom pseudos are added with uppercase letters + // Remember that setFilters inherits from pseudos + var args, + fn = Expr.pseudos[ pseudo ] || Expr.setFilters[ pseudo.toLowerCase() ] || + Sizzle.error( "unsupported pseudo: " + pseudo ); + + // The user may use createPseudo to indicate that + // arguments are needed to create the filter function + // just as Sizzle does + if ( fn[ expando ] ) { + return fn( argument ); + } + + // But maintain support for old signatures + if ( fn.length > 1 ) { + args = [ pseudo, pseudo, "", argument ]; + return Expr.setFilters.hasOwnProperty( pseudo.toLowerCase() ) ? + markFunction( function( seed, matches ) { + var idx, + matched = fn( seed, argument ), + i = matched.length; + while ( i-- ) { + idx = indexOf( seed, matched[ i ] ); + seed[ idx ] = !( matches[ idx ] = matched[ i ] ); + } + } ) : + function( elem ) { + return fn( elem, 0, args ); + }; + } + + return fn; + } + }, + + pseudos: { + + // Potentially complex pseudos + "not": markFunction( function( selector ) { + + // Trim the selector passed to compile + // to avoid treating leading and trailing + // spaces as combinators + var input = [], + results = [], + matcher = compile( selector.replace( rtrim, "$1" ) ); + + return matcher[ expando ] ? + markFunction( function( seed, matches, _context, xml ) { + var elem, + unmatched = matcher( seed, null, xml, [] ), + i = seed.length; + + // Match elements unmatched by `matcher` + while ( i-- ) { + if ( ( elem = unmatched[ i ] ) ) { + seed[ i ] = !( matches[ i ] = elem ); + } + } + } ) : + function( elem, _context, xml ) { + input[ 0 ] = elem; + matcher( input, null, xml, results ); + + // Don't keep the element (issue #299) + input[ 0 ] = null; + return !results.pop(); + }; + } ), + + "has": markFunction( function( selector ) { + return function( elem ) { + return Sizzle( selector, elem ).length > 0; + }; + } ), + + "contains": markFunction( function( text ) { + text = text.replace( runescape, funescape ); + return function( elem ) { + return ( elem.textContent || getText( elem ) ).indexOf( text ) > -1; + }; + } ), + + // "Whether an element is represented by a :lang() selector + // is based solely on the element's language value + // being equal to the identifier C, + // or beginning with the identifier C immediately followed by "-". + // The matching of C against the element's language value is performed case-insensitively. + // The identifier C does not have to be a valid language name." + // http://www.w3.org/TR/selectors/#lang-pseudo + "lang": markFunction( function( lang ) { + + // lang value must be a valid identifier + if ( !ridentifier.test( lang || "" ) ) { + Sizzle.error( "unsupported lang: " + lang ); + } + lang = lang.replace( runescape, funescape ).toLowerCase(); + return function( elem ) { + var elemLang; + do { + if ( ( elemLang = documentIsHTML ? + elem.lang : + elem.getAttribute( "xml:lang" ) || elem.getAttribute( "lang" ) ) ) { + + elemLang = elemLang.toLowerCase(); + return elemLang === lang || elemLang.indexOf( lang + "-" ) === 0; + } + } while ( ( elem = elem.parentNode ) && elem.nodeType === 1 ); + return false; + }; + } ), + + // Miscellaneous + "target": function( elem ) { + var hash = window.location && window.location.hash; + return hash && hash.slice( 1 ) === elem.id; + }, + + "root": function( elem ) { + return elem === docElem; + }, + + "focus": function( elem ) { + return elem === document.activeElement && + ( !document.hasFocus || document.hasFocus() ) && + !!( elem.type || elem.href || ~elem.tabIndex ); + }, + + // Boolean properties + "enabled": createDisabledPseudo( false ), + "disabled": createDisabledPseudo( true ), + + "checked": function( elem ) { + + // In CSS3, :checked should return both checked and selected elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + var nodeName = elem.nodeName.toLowerCase(); + return ( nodeName === "input" && !!elem.checked ) || + ( nodeName === "option" && !!elem.selected ); + }, + + "selected": function( elem ) { + + // Accessing this property makes selected-by-default + // options in Safari work properly + if ( elem.parentNode ) { + // eslint-disable-next-line no-unused-expressions + elem.parentNode.selectedIndex; + } + + return elem.selected === true; + }, + + // Contents + "empty": function( elem ) { + + // http://www.w3.org/TR/selectors/#empty-pseudo + // :empty is negated by element (1) or content nodes (text: 3; cdata: 4; entity ref: 5), + // but not by others (comment: 8; processing instruction: 7; etc.) + // nodeType < 6 works because attributes (2) do not appear as children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + if ( elem.nodeType < 6 ) { + return false; + } + } + return true; + }, + + "parent": function( elem ) { + return !Expr.pseudos[ "empty" ]( elem ); + }, + + // Element/input types + "header": function( elem ) { + return rheader.test( elem.nodeName ); + }, + + "input": function( elem ) { + return rinputs.test( elem.nodeName ); + }, + + "button": function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === "button" || name === "button"; + }, + + "text": function( elem ) { + var attr; + return elem.nodeName.toLowerCase() === "input" && + elem.type === "text" && + + // Support: IE<8 + // New HTML5 attribute values (e.g., "search") appear with elem.type === "text" + ( ( attr = elem.getAttribute( "type" ) ) == null || + attr.toLowerCase() === "text" ); + }, + + // Position-in-collection + "first": createPositionalPseudo( function() { + return [ 0 ]; + } ), + + "last": createPositionalPseudo( function( _matchIndexes, length ) { + return [ length - 1 ]; + } ), + + "eq": createPositionalPseudo( function( _matchIndexes, length, argument ) { + return [ argument < 0 ? argument + length : argument ]; + } ), + + "even": createPositionalPseudo( function( matchIndexes, length ) { + var i = 0; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "odd": createPositionalPseudo( function( matchIndexes, length ) { + var i = 1; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "lt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? + argument + length : + argument > length ? + length : + argument; + for ( ; --i >= 0; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "gt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? argument + length : argument; + for ( ; ++i < length; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ) + } +}; + +Expr.pseudos[ "nth" ] = Expr.pseudos[ "eq" ]; + +// Add button/input type pseudos +for ( i in { radio: true, checkbox: true, file: true, password: true, image: true } ) { + Expr.pseudos[ i ] = createInputPseudo( i ); +} +for ( i in { submit: true, reset: true } ) { + Expr.pseudos[ i ] = createButtonPseudo( i ); +} + +// Easy API for creating new setFilters +function setFilters() {} +setFilters.prototype = Expr.filters = Expr.pseudos; +Expr.setFilters = new setFilters(); + +tokenize = Sizzle.tokenize = function( selector, parseOnly ) { + var matched, match, tokens, type, + soFar, groups, preFilters, + cached = tokenCache[ selector + " " ]; + + if ( cached ) { + return parseOnly ? 0 : cached.slice( 0 ); + } + + soFar = selector; + groups = []; + preFilters = Expr.preFilter; + + while ( soFar ) { + + // Comma and first run + if ( !matched || ( match = rcomma.exec( soFar ) ) ) { + if ( match ) { + + // Don't consume trailing commas as valid + soFar = soFar.slice( match[ 0 ].length ) || soFar; + } + groups.push( ( tokens = [] ) ); + } + + matched = false; + + // Combinators + if ( ( match = rcombinators.exec( soFar ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + + // Cast descendant combinators to space + type: match[ 0 ].replace( rtrim, " " ) + } ); + soFar = soFar.slice( matched.length ); + } + + // Filters + for ( type in Expr.filter ) { + if ( ( match = matchExpr[ type ].exec( soFar ) ) && ( !preFilters[ type ] || + ( match = preFilters[ type ]( match ) ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + type: type, + matches: match + } ); + soFar = soFar.slice( matched.length ); + } + } + + if ( !matched ) { + break; + } + } + + // Return the length of the invalid excess + // if we're just parsing + // Otherwise, throw an error or return tokens + return parseOnly ? + soFar.length : + soFar ? + Sizzle.error( selector ) : + + // Cache the tokens + tokenCache( selector, groups ).slice( 0 ); +}; + +function toSelector( tokens ) { + var i = 0, + len = tokens.length, + selector = ""; + for ( ; i < len; i++ ) { + selector += tokens[ i ].value; + } + return selector; +} + +function addCombinator( matcher, combinator, base ) { + var dir = combinator.dir, + skip = combinator.next, + key = skip || dir, + checkNonElements = base && key === "parentNode", + doneName = done++; + + return combinator.first ? + + // Check against closest ancestor/preceding element + function( elem, context, xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + return matcher( elem, context, xml ); + } + } + return false; + } : + + // Check against all ancestor/preceding elements + function( elem, context, xml ) { + var oldCache, uniqueCache, outerCache, + newCache = [ dirruns, doneName ]; + + // We can't set arbitrary data on XML nodes, so they don't benefit from combinator caching + if ( xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + if ( matcher( elem, context, xml ) ) { + return true; + } + } + } + } else { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + outerCache = elem[ expando ] || ( elem[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ elem.uniqueID ] || + ( outerCache[ elem.uniqueID ] = {} ); + + if ( skip && skip === elem.nodeName.toLowerCase() ) { + elem = elem[ dir ] || elem; + } else if ( ( oldCache = uniqueCache[ key ] ) && + oldCache[ 0 ] === dirruns && oldCache[ 1 ] === doneName ) { + + // Assign to newCache so results back-propagate to previous elements + return ( newCache[ 2 ] = oldCache[ 2 ] ); + } else { + + // Reuse newcache so results back-propagate to previous elements + uniqueCache[ key ] = newCache; + + // A match means we're done; a fail means we have to keep checking + if ( ( newCache[ 2 ] = matcher( elem, context, xml ) ) ) { + return true; + } + } + } + } + } + return false; + }; +} + +function elementMatcher( matchers ) { + return matchers.length > 1 ? + function( elem, context, xml ) { + var i = matchers.length; + while ( i-- ) { + if ( !matchers[ i ]( elem, context, xml ) ) { + return false; + } + } + return true; + } : + matchers[ 0 ]; +} + +function multipleContexts( selector, contexts, results ) { + var i = 0, + len = contexts.length; + for ( ; i < len; i++ ) { + Sizzle( selector, contexts[ i ], results ); + } + return results; +} + +function condense( unmatched, map, filter, context, xml ) { + var elem, + newUnmatched = [], + i = 0, + len = unmatched.length, + mapped = map != null; + + for ( ; i < len; i++ ) { + if ( ( elem = unmatched[ i ] ) ) { + if ( !filter || filter( elem, context, xml ) ) { + newUnmatched.push( elem ); + if ( mapped ) { + map.push( i ); + } + } + } + } + + return newUnmatched; +} + +function setMatcher( preFilter, selector, matcher, postFilter, postFinder, postSelector ) { + if ( postFilter && !postFilter[ expando ] ) { + postFilter = setMatcher( postFilter ); + } + if ( postFinder && !postFinder[ expando ] ) { + postFinder = setMatcher( postFinder, postSelector ); + } + return markFunction( function( seed, results, context, xml ) { + var temp, i, elem, + preMap = [], + postMap = [], + preexisting = results.length, + + // Get initial elements from seed or context + elems = seed || multipleContexts( + selector || "*", + context.nodeType ? [ context ] : context, + [] + ), + + // Prefilter to get matcher input, preserving a map for seed-results synchronization + matcherIn = preFilter && ( seed || !selector ) ? + condense( elems, preMap, preFilter, context, xml ) : + elems, + + matcherOut = matcher ? + + // If we have a postFinder, or filtered seed, or non-seed postFilter or preexisting results, + postFinder || ( seed ? preFilter : preexisting || postFilter ) ? + + // ...intermediate processing is necessary + [] : + + // ...otherwise use results directly + results : + matcherIn; + + // Find primary matches + if ( matcher ) { + matcher( matcherIn, matcherOut, context, xml ); + } + + // Apply postFilter + if ( postFilter ) { + temp = condense( matcherOut, postMap ); + postFilter( temp, [], context, xml ); + + // Un-match failing elements by moving them back to matcherIn + i = temp.length; + while ( i-- ) { + if ( ( elem = temp[ i ] ) ) { + matcherOut[ postMap[ i ] ] = !( matcherIn[ postMap[ i ] ] = elem ); + } + } + } + + if ( seed ) { + if ( postFinder || preFilter ) { + if ( postFinder ) { + + // Get the final matcherOut by condensing this intermediate into postFinder contexts + temp = []; + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) ) { + + // Restore matcherIn since elem is not yet a final match + temp.push( ( matcherIn[ i ] = elem ) ); + } + } + postFinder( null, ( matcherOut = [] ), temp, xml ); + } + + // Move matched elements from seed to results to keep them synchronized + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) && + ( temp = postFinder ? indexOf( seed, elem ) : preMap[ i ] ) > -1 ) { + + seed[ temp ] = !( results[ temp ] = elem ); + } + } + } + + // Add elements to results, through postFinder if defined + } else { + matcherOut = condense( + matcherOut === results ? + matcherOut.splice( preexisting, matcherOut.length ) : + matcherOut + ); + if ( postFinder ) { + postFinder( null, results, matcherOut, xml ); + } else { + push.apply( results, matcherOut ); + } + } + } ); +} + +function matcherFromTokens( tokens ) { + var checkContext, matcher, j, + len = tokens.length, + leadingRelative = Expr.relative[ tokens[ 0 ].type ], + implicitRelative = leadingRelative || Expr.relative[ " " ], + i = leadingRelative ? 1 : 0, + + // The foundational matcher ensures that elements are reachable from top-level context(s) + matchContext = addCombinator( function( elem ) { + return elem === checkContext; + }, implicitRelative, true ), + matchAnyContext = addCombinator( function( elem ) { + return indexOf( checkContext, elem ) > -1; + }, implicitRelative, true ), + matchers = [ function( elem, context, xml ) { + var ret = ( !leadingRelative && ( xml || context !== outermostContext ) ) || ( + ( checkContext = context ).nodeType ? + matchContext( elem, context, xml ) : + matchAnyContext( elem, context, xml ) ); + + // Avoid hanging onto element (issue #299) + checkContext = null; + return ret; + } ]; + + for ( ; i < len; i++ ) { + if ( ( matcher = Expr.relative[ tokens[ i ].type ] ) ) { + matchers = [ addCombinator( elementMatcher( matchers ), matcher ) ]; + } else { + matcher = Expr.filter[ tokens[ i ].type ].apply( null, tokens[ i ].matches ); + + // Return special upon seeing a positional matcher + if ( matcher[ expando ] ) { + + // Find the next relative operator (if any) for proper handling + j = ++i; + for ( ; j < len; j++ ) { + if ( Expr.relative[ tokens[ j ].type ] ) { + break; + } + } + return setMatcher( + i > 1 && elementMatcher( matchers ), + i > 1 && toSelector( + + // If the preceding token was a descendant combinator, insert an implicit any-element `*` + tokens + .slice( 0, i - 1 ) + .concat( { value: tokens[ i - 2 ].type === " " ? "*" : "" } ) + ).replace( rtrim, "$1" ), + matcher, + i < j && matcherFromTokens( tokens.slice( i, j ) ), + j < len && matcherFromTokens( ( tokens = tokens.slice( j ) ) ), + j < len && toSelector( tokens ) + ); + } + matchers.push( matcher ); + } + } + + return elementMatcher( matchers ); +} + +function matcherFromGroupMatchers( elementMatchers, setMatchers ) { + var bySet = setMatchers.length > 0, + byElement = elementMatchers.length > 0, + superMatcher = function( seed, context, xml, results, outermost ) { + var elem, j, matcher, + matchedCount = 0, + i = "0", + unmatched = seed && [], + setMatched = [], + contextBackup = outermostContext, + + // We must always have either seed elements or outermost context + elems = seed || byElement && Expr.find[ "TAG" ]( "*", outermost ), + + // Use integer dirruns iff this is the outermost matcher + dirrunsUnique = ( dirruns += contextBackup == null ? 1 : Math.random() || 0.1 ), + len = elems.length; + + if ( outermost ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + outermostContext = context == document || context || outermost; + } + + // Add elements passing elementMatchers directly to results + // Support: IE<9, Safari + // Tolerate NodeList properties (IE: "length"; Safari: ) matching elements by id + for ( ; i !== len && ( elem = elems[ i ] ) != null; i++ ) { + if ( byElement && elem ) { + j = 0; + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( !context && elem.ownerDocument != document ) { + setDocument( elem ); + xml = !documentIsHTML; + } + while ( ( matcher = elementMatchers[ j++ ] ) ) { + if ( matcher( elem, context || document, xml ) ) { + results.push( elem ); + break; + } + } + if ( outermost ) { + dirruns = dirrunsUnique; + } + } + + // Track unmatched elements for set filters + if ( bySet ) { + + // They will have gone through all possible matchers + if ( ( elem = !matcher && elem ) ) { + matchedCount--; + } + + // Lengthen the array for every element, matched or not + if ( seed ) { + unmatched.push( elem ); + } + } + } + + // `i` is now the count of elements visited above, and adding it to `matchedCount` + // makes the latter nonnegative. + matchedCount += i; + + // Apply set filters to unmatched elements + // NOTE: This can be skipped if there are no unmatched elements (i.e., `matchedCount` + // equals `i`), unless we didn't visit _any_ elements in the above loop because we have + // no element matchers and no seed. + // Incrementing an initially-string "0" `i` allows `i` to remain a string only in that + // case, which will result in a "00" `matchedCount` that differs from `i` but is also + // numerically zero. + if ( bySet && i !== matchedCount ) { + j = 0; + while ( ( matcher = setMatchers[ j++ ] ) ) { + matcher( unmatched, setMatched, context, xml ); + } + + if ( seed ) { + + // Reintegrate element matches to eliminate the need for sorting + if ( matchedCount > 0 ) { + while ( i-- ) { + if ( !( unmatched[ i ] || setMatched[ i ] ) ) { + setMatched[ i ] = pop.call( results ); + } + } + } + + // Discard index placeholder values to get only actual matches + setMatched = condense( setMatched ); + } + + // Add matches to results + push.apply( results, setMatched ); + + // Seedless set matches succeeding multiple successful matchers stipulate sorting + if ( outermost && !seed && setMatched.length > 0 && + ( matchedCount + setMatchers.length ) > 1 ) { + + Sizzle.uniqueSort( results ); + } + } + + // Override manipulation of globals by nested matchers + if ( outermost ) { + dirruns = dirrunsUnique; + outermostContext = contextBackup; + } + + return unmatched; + }; + + return bySet ? + markFunction( superMatcher ) : + superMatcher; +} + +compile = Sizzle.compile = function( selector, match /* Internal Use Only */ ) { + var i, + setMatchers = [], + elementMatchers = [], + cached = compilerCache[ selector + " " ]; + + if ( !cached ) { + + // Generate a function of recursive functions that can be used to check each element + if ( !match ) { + match = tokenize( selector ); + } + i = match.length; + while ( i-- ) { + cached = matcherFromTokens( match[ i ] ); + if ( cached[ expando ] ) { + setMatchers.push( cached ); + } else { + elementMatchers.push( cached ); + } + } + + // Cache the compiled function + cached = compilerCache( + selector, + matcherFromGroupMatchers( elementMatchers, setMatchers ) + ); + + // Save selector and tokenization + cached.selector = selector; + } + return cached; +}; + +/** + * A low-level selection function that works with Sizzle's compiled + * selector functions + * @param {String|Function} selector A selector or a pre-compiled + * selector function built with Sizzle.compile + * @param {Element} context + * @param {Array} [results] + * @param {Array} [seed] A set of elements to match against + */ +select = Sizzle.select = function( selector, context, results, seed ) { + var i, tokens, token, type, find, + compiled = typeof selector === "function" && selector, + match = !seed && tokenize( ( selector = compiled.selector || selector ) ); + + results = results || []; + + // Try to minimize operations if there is only one selector in the list and no seed + // (the latter of which guarantees us context) + if ( match.length === 1 ) { + + // Reduce context if the leading compound selector is an ID + tokens = match[ 0 ] = match[ 0 ].slice( 0 ); + if ( tokens.length > 2 && ( token = tokens[ 0 ] ).type === "ID" && + context.nodeType === 9 && documentIsHTML && Expr.relative[ tokens[ 1 ].type ] ) { + + context = ( Expr.find[ "ID" ]( token.matches[ 0 ] + .replace( runescape, funescape ), context ) || [] )[ 0 ]; + if ( !context ) { + return results; + + // Precompiled matchers will still verify ancestry, so step up a level + } else if ( compiled ) { + context = context.parentNode; + } + + selector = selector.slice( tokens.shift().value.length ); + } + + // Fetch a seed set for right-to-left matching + i = matchExpr[ "needsContext" ].test( selector ) ? 0 : tokens.length; + while ( i-- ) { + token = tokens[ i ]; + + // Abort if we hit a combinator + if ( Expr.relative[ ( type = token.type ) ] ) { + break; + } + if ( ( find = Expr.find[ type ] ) ) { + + // Search, expanding context for leading sibling combinators + if ( ( seed = find( + token.matches[ 0 ].replace( runescape, funescape ), + rsibling.test( tokens[ 0 ].type ) && testContext( context.parentNode ) || + context + ) ) ) { + + // If seed is empty or no tokens remain, we can return early + tokens.splice( i, 1 ); + selector = seed.length && toSelector( tokens ); + if ( !selector ) { + push.apply( results, seed ); + return results; + } + + break; + } + } + } + } + + // Compile and execute a filtering function if one is not provided + // Provide `match` to avoid retokenization if we modified the selector above + ( compiled || compile( selector, match ) )( + seed, + context, + !documentIsHTML, + results, + !context || rsibling.test( selector ) && testContext( context.parentNode ) || context + ); + return results; +}; + +// One-time assignments + +// Sort stability +support.sortStable = expando.split( "" ).sort( sortOrder ).join( "" ) === expando; + +// Support: Chrome 14-35+ +// Always assume duplicates if they aren't passed to the comparison function +support.detectDuplicates = !!hasDuplicate; + +// Initialize against the default document +setDocument(); + +// Support: Webkit<537.32 - Safari 6.0.3/Chrome 25 (fixed in Chrome 27) +// Detached nodes confoundingly follow *each other* +support.sortDetached = assert( function( el ) { + + // Should return 1, but returns 4 (following) + return el.compareDocumentPosition( document.createElement( "fieldset" ) ) & 1; +} ); + +// Support: IE<8 +// Prevent attribute/property "interpolation" +// https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx +if ( !assert( function( el ) { + el.innerHTML = ""; + return el.firstChild.getAttribute( "href" ) === "#"; +} ) ) { + addHandle( "type|href|height|width", function( elem, name, isXML ) { + if ( !isXML ) { + return elem.getAttribute( name, name.toLowerCase() === "type" ? 1 : 2 ); + } + } ); +} + +// Support: IE<9 +// Use defaultValue in place of getAttribute("value") +if ( !support.attributes || !assert( function( el ) { + el.innerHTML = ""; + el.firstChild.setAttribute( "value", "" ); + return el.firstChild.getAttribute( "value" ) === ""; +} ) ) { + addHandle( "value", function( elem, _name, isXML ) { + if ( !isXML && elem.nodeName.toLowerCase() === "input" ) { + return elem.defaultValue; + } + } ); +} + +// Support: IE<9 +// Use getAttributeNode to fetch booleans when getAttribute lies +if ( !assert( function( el ) { + return el.getAttribute( "disabled" ) == null; +} ) ) { + addHandle( booleans, function( elem, name, isXML ) { + var val; + if ( !isXML ) { + return elem[ name ] === true ? name.toLowerCase() : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; + } + } ); +} + +return Sizzle; + +} )( window ); + + + +jQuery.find = Sizzle; +jQuery.expr = Sizzle.selectors; + +// Deprecated +jQuery.expr[ ":" ] = jQuery.expr.pseudos; +jQuery.uniqueSort = jQuery.unique = Sizzle.uniqueSort; +jQuery.text = Sizzle.getText; +jQuery.isXMLDoc = Sizzle.isXML; +jQuery.contains = Sizzle.contains; +jQuery.escapeSelector = Sizzle.escape; + + + + +var dir = function( elem, dir, until ) { + var matched = [], + truncate = until !== undefined; + + while ( ( elem = elem[ dir ] ) && elem.nodeType !== 9 ) { + if ( elem.nodeType === 1 ) { + if ( truncate && jQuery( elem ).is( until ) ) { + break; + } + matched.push( elem ); + } + } + return matched; +}; + + +var siblings = function( n, elem ) { + var matched = []; + + for ( ; n; n = n.nextSibling ) { + if ( n.nodeType === 1 && n !== elem ) { + matched.push( n ); + } + } + + return matched; +}; + + +var rneedsContext = jQuery.expr.match.needsContext; + + + +function nodeName( elem, name ) { + + return elem.nodeName && elem.nodeName.toLowerCase() === name.toLowerCase(); + +} +var rsingleTag = ( /^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i ); + + + +// Implement the identical functionality for filter and not +function winnow( elements, qualifier, not ) { + if ( isFunction( qualifier ) ) { + return jQuery.grep( elements, function( elem, i ) { + return !!qualifier.call( elem, i, elem ) !== not; + } ); + } + + // Single element + if ( qualifier.nodeType ) { + return jQuery.grep( elements, function( elem ) { + return ( elem === qualifier ) !== not; + } ); + } + + // Arraylike of elements (jQuery, arguments, Array) + if ( typeof qualifier !== "string" ) { + return jQuery.grep( elements, function( elem ) { + return ( indexOf.call( qualifier, elem ) > -1 ) !== not; + } ); + } + + // Filtered directly for both simple and complex selectors + return jQuery.filter( qualifier, elements, not ); +} + +jQuery.filter = function( expr, elems, not ) { + var elem = elems[ 0 ]; + + if ( not ) { + expr = ":not(" + expr + ")"; + } + + if ( elems.length === 1 && elem.nodeType === 1 ) { + return jQuery.find.matchesSelector( elem, expr ) ? [ elem ] : []; + } + + return jQuery.find.matches( expr, jQuery.grep( elems, function( elem ) { + return elem.nodeType === 1; + } ) ); +}; + +jQuery.fn.extend( { + find: function( selector ) { + var i, ret, + len = this.length, + self = this; + + if ( typeof selector !== "string" ) { + return this.pushStack( jQuery( selector ).filter( function() { + for ( i = 0; i < len; i++ ) { + if ( jQuery.contains( self[ i ], this ) ) { + return true; + } + } + } ) ); + } + + ret = this.pushStack( [] ); + + for ( i = 0; i < len; i++ ) { + jQuery.find( selector, self[ i ], ret ); + } + + return len > 1 ? jQuery.uniqueSort( ret ) : ret; + }, + filter: function( selector ) { + return this.pushStack( winnow( this, selector || [], false ) ); + }, + not: function( selector ) { + return this.pushStack( winnow( this, selector || [], true ) ); + }, + is: function( selector ) { + return !!winnow( + this, + + // If this is a positional/relative selector, check membership in the returned set + // so $("p:first").is("p:last") won't return true for a doc with two "p". + typeof selector === "string" && rneedsContext.test( selector ) ? + jQuery( selector ) : + selector || [], + false + ).length; + } +} ); + + +// Initialize a jQuery object + + +// A central reference to the root jQuery(document) +var rootjQuery, + + // A simple way to check for HTML strings + // Prioritize #id over to avoid XSS via location.hash (#9521) + // Strict HTML recognition (#11290: must start with <) + // Shortcut simple #id case for speed + rquickExpr = /^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/, + + init = jQuery.fn.init = function( selector, context, root ) { + var match, elem; + + // HANDLE: $(""), $(null), $(undefined), $(false) + if ( !selector ) { + return this; + } + + // Method init() accepts an alternate rootjQuery + // so migrate can support jQuery.sub (gh-2101) + root = root || rootjQuery; + + // Handle HTML strings + if ( typeof selector === "string" ) { + if ( selector[ 0 ] === "<" && + selector[ selector.length - 1 ] === ">" && + selector.length >= 3 ) { + + // Assume that strings that start and end with <> are HTML and skip the regex check + match = [ null, selector, null ]; + + } else { + match = rquickExpr.exec( selector ); + } + + // Match html or make sure no context is specified for #id + if ( match && ( match[ 1 ] || !context ) ) { + + // HANDLE: $(html) -> $(array) + if ( match[ 1 ] ) { + context = context instanceof jQuery ? context[ 0 ] : context; + + // Option to run scripts is true for back-compat + // Intentionally let the error be thrown if parseHTML is not present + jQuery.merge( this, jQuery.parseHTML( + match[ 1 ], + context && context.nodeType ? context.ownerDocument || context : document, + true + ) ); + + // HANDLE: $(html, props) + if ( rsingleTag.test( match[ 1 ] ) && jQuery.isPlainObject( context ) ) { + for ( match in context ) { + + // Properties of context are called as methods if possible + if ( isFunction( this[ match ] ) ) { + this[ match ]( context[ match ] ); + + // ...and otherwise set as attributes + } else { + this.attr( match, context[ match ] ); + } + } + } + + return this; + + // HANDLE: $(#id) + } else { + elem = document.getElementById( match[ 2 ] ); + + if ( elem ) { + + // Inject the element directly into the jQuery object + this[ 0 ] = elem; + this.length = 1; + } + return this; + } + + // HANDLE: $(expr, $(...)) + } else if ( !context || context.jquery ) { + return ( context || root ).find( selector ); + + // HANDLE: $(expr, context) + // (which is just equivalent to: $(context).find(expr) + } else { + return this.constructor( context ).find( selector ); + } + + // HANDLE: $(DOMElement) + } else if ( selector.nodeType ) { + this[ 0 ] = selector; + this.length = 1; + return this; + + // HANDLE: $(function) + // Shortcut for document ready + } else if ( isFunction( selector ) ) { + return root.ready !== undefined ? + root.ready( selector ) : + + // Execute immediately if ready is not present + selector( jQuery ); + } + + return jQuery.makeArray( selector, this ); + }; + +// Give the init function the jQuery prototype for later instantiation +init.prototype = jQuery.fn; + +// Initialize central reference +rootjQuery = jQuery( document ); + + +var rparentsprev = /^(?:parents|prev(?:Until|All))/, + + // Methods guaranteed to produce a unique set when starting from a unique set + guaranteedUnique = { + children: true, + contents: true, + next: true, + prev: true + }; + +jQuery.fn.extend( { + has: function( target ) { + var targets = jQuery( target, this ), + l = targets.length; + + return this.filter( function() { + var i = 0; + for ( ; i < l; i++ ) { + if ( jQuery.contains( this, targets[ i ] ) ) { + return true; + } + } + } ); + }, + + closest: function( selectors, context ) { + var cur, + i = 0, + l = this.length, + matched = [], + targets = typeof selectors !== "string" && jQuery( selectors ); + + // Positional selectors never match, since there's no _selection_ context + if ( !rneedsContext.test( selectors ) ) { + for ( ; i < l; i++ ) { + for ( cur = this[ i ]; cur && cur !== context; cur = cur.parentNode ) { + + // Always skip document fragments + if ( cur.nodeType < 11 && ( targets ? + targets.index( cur ) > -1 : + + // Don't pass non-elements to Sizzle + cur.nodeType === 1 && + jQuery.find.matchesSelector( cur, selectors ) ) ) { + + matched.push( cur ); + break; + } + } + } + } + + return this.pushStack( matched.length > 1 ? jQuery.uniqueSort( matched ) : matched ); + }, + + // Determine the position of an element within the set + index: function( elem ) { + + // No argument, return index in parent + if ( !elem ) { + return ( this[ 0 ] && this[ 0 ].parentNode ) ? this.first().prevAll().length : -1; + } + + // Index in selector + if ( typeof elem === "string" ) { + return indexOf.call( jQuery( elem ), this[ 0 ] ); + } + + // Locate the position of the desired element + return indexOf.call( this, + + // If it receives a jQuery object, the first element is used + elem.jquery ? elem[ 0 ] : elem + ); + }, + + add: function( selector, context ) { + return this.pushStack( + jQuery.uniqueSort( + jQuery.merge( this.get(), jQuery( selector, context ) ) + ) + ); + }, + + addBack: function( selector ) { + return this.add( selector == null ? + this.prevObject : this.prevObject.filter( selector ) + ); + } +} ); + +function sibling( cur, dir ) { + while ( ( cur = cur[ dir ] ) && cur.nodeType !== 1 ) {} + return cur; +} + +jQuery.each( { + parent: function( elem ) { + var parent = elem.parentNode; + return parent && parent.nodeType !== 11 ? parent : null; + }, + parents: function( elem ) { + return dir( elem, "parentNode" ); + }, + parentsUntil: function( elem, _i, until ) { + return dir( elem, "parentNode", until ); + }, + next: function( elem ) { + return sibling( elem, "nextSibling" ); + }, + prev: function( elem ) { + return sibling( elem, "previousSibling" ); + }, + nextAll: function( elem ) { + return dir( elem, "nextSibling" ); + }, + prevAll: function( elem ) { + return dir( elem, "previousSibling" ); + }, + nextUntil: function( elem, _i, until ) { + return dir( elem, "nextSibling", until ); + }, + prevUntil: function( elem, _i, until ) { + return dir( elem, "previousSibling", until ); + }, + siblings: function( elem ) { + return siblings( ( elem.parentNode || {} ).firstChild, elem ); + }, + children: function( elem ) { + return siblings( elem.firstChild ); + }, + contents: function( elem ) { + if ( elem.contentDocument != null && + + // Support: IE 11+ + // elements with no `data` attribute has an object + // `contentDocument` with a `null` prototype. + getProto( elem.contentDocument ) ) { + + return elem.contentDocument; + } + + // Support: IE 9 - 11 only, iOS 7 only, Android Browser <=4.3 only + // Treat the template element as a regular one in browsers that + // don't support it. + if ( nodeName( elem, "template" ) ) { + elem = elem.content || elem; + } + + return jQuery.merge( [], elem.childNodes ); + } +}, function( name, fn ) { + jQuery.fn[ name ] = function( until, selector ) { + var matched = jQuery.map( this, fn, until ); + + if ( name.slice( -5 ) !== "Until" ) { + selector = until; + } + + if ( selector && typeof selector === "string" ) { + matched = jQuery.filter( selector, matched ); + } + + if ( this.length > 1 ) { + + // Remove duplicates + if ( !guaranteedUnique[ name ] ) { + jQuery.uniqueSort( matched ); + } + + // Reverse order for parents* and prev-derivatives + if ( rparentsprev.test( name ) ) { + matched.reverse(); + } + } + + return this.pushStack( matched ); + }; +} ); +var rnothtmlwhite = ( /[^\x20\t\r\n\f]+/g ); + + + +// Convert String-formatted options into Object-formatted ones +function createOptions( options ) { + var object = {}; + jQuery.each( options.match( rnothtmlwhite ) || [], function( _, flag ) { + object[ flag ] = true; + } ); + return object; +} + +/* + * Create a callback list using the following parameters: + * + * options: an optional list of space-separated options that will change how + * the callback list behaves or a more traditional option object + * + * By default a callback list will act like an event callback list and can be + * "fired" multiple times. + * + * Possible options: + * + * once: will ensure the callback list can only be fired once (like a Deferred) + * + * memory: will keep track of previous values and will call any callback added + * after the list has been fired right away with the latest "memorized" + * values (like a Deferred) + * + * unique: will ensure a callback can only be added once (no duplicate in the list) + * + * stopOnFalse: interrupt callings when a callback returns false + * + */ +jQuery.Callbacks = function( options ) { + + // Convert options from String-formatted to Object-formatted if needed + // (we check in cache first) + options = typeof options === "string" ? + createOptions( options ) : + jQuery.extend( {}, options ); + + var // Flag to know if list is currently firing + firing, + + // Last fire value for non-forgettable lists + memory, + + // Flag to know if list was already fired + fired, + + // Flag to prevent firing + locked, + + // Actual callback list + list = [], + + // Queue of execution data for repeatable lists + queue = [], + + // Index of currently firing callback (modified by add/remove as needed) + firingIndex = -1, + + // Fire callbacks + fire = function() { + + // Enforce single-firing + locked = locked || options.once; + + // Execute callbacks for all pending executions, + // respecting firingIndex overrides and runtime changes + fired = firing = true; + for ( ; queue.length; firingIndex = -1 ) { + memory = queue.shift(); + while ( ++firingIndex < list.length ) { + + // Run callback and check for early termination + if ( list[ firingIndex ].apply( memory[ 0 ], memory[ 1 ] ) === false && + options.stopOnFalse ) { + + // Jump to end and forget the data so .add doesn't re-fire + firingIndex = list.length; + memory = false; + } + } + } + + // Forget the data if we're done with it + if ( !options.memory ) { + memory = false; + } + + firing = false; + + // Clean up if we're done firing for good + if ( locked ) { + + // Keep an empty list if we have data for future add calls + if ( memory ) { + list = []; + + // Otherwise, this object is spent + } else { + list = ""; + } + } + }, + + // Actual Callbacks object + self = { + + // Add a callback or a collection of callbacks to the list + add: function() { + if ( list ) { + + // If we have memory from a past run, we should fire after adding + if ( memory && !firing ) { + firingIndex = list.length - 1; + queue.push( memory ); + } + + ( function add( args ) { + jQuery.each( args, function( _, arg ) { + if ( isFunction( arg ) ) { + if ( !options.unique || !self.has( arg ) ) { + list.push( arg ); + } + } else if ( arg && arg.length && toType( arg ) !== "string" ) { + + // Inspect recursively + add( arg ); + } + } ); + } )( arguments ); + + if ( memory && !firing ) { + fire(); + } + } + return this; + }, + + // Remove a callback from the list + remove: function() { + jQuery.each( arguments, function( _, arg ) { + var index; + while ( ( index = jQuery.inArray( arg, list, index ) ) > -1 ) { + list.splice( index, 1 ); + + // Handle firing indexes + if ( index <= firingIndex ) { + firingIndex--; + } + } + } ); + return this; + }, + + // Check if a given callback is in the list. + // If no argument is given, return whether or not list has callbacks attached. + has: function( fn ) { + return fn ? + jQuery.inArray( fn, list ) > -1 : + list.length > 0; + }, + + // Remove all callbacks from the list + empty: function() { + if ( list ) { + list = []; + } + return this; + }, + + // Disable .fire and .add + // Abort any current/pending executions + // Clear all callbacks and values + disable: function() { + locked = queue = []; + list = memory = ""; + return this; + }, + disabled: function() { + return !list; + }, + + // Disable .fire + // Also disable .add unless we have memory (since it would have no effect) + // Abort any pending executions + lock: function() { + locked = queue = []; + if ( !memory && !firing ) { + list = memory = ""; + } + return this; + }, + locked: function() { + return !!locked; + }, + + // Call all callbacks with the given context and arguments + fireWith: function( context, args ) { + if ( !locked ) { + args = args || []; + args = [ context, args.slice ? args.slice() : args ]; + queue.push( args ); + if ( !firing ) { + fire(); + } + } + return this; + }, + + // Call all the callbacks with the given arguments + fire: function() { + self.fireWith( this, arguments ); + return this; + }, + + // To know if the callbacks have already been called at least once + fired: function() { + return !!fired; + } + }; + + return self; +}; + + +function Identity( v ) { + return v; +} +function Thrower( ex ) { + throw ex; +} + +function adoptValue( value, resolve, reject, noValue ) { + var method; + + try { + + // Check for promise aspect first to privilege synchronous behavior + if ( value && isFunction( ( method = value.promise ) ) ) { + method.call( value ).done( resolve ).fail( reject ); + + // Other thenables + } else if ( value && isFunction( ( method = value.then ) ) ) { + method.call( value, resolve, reject ); + + // Other non-thenables + } else { + + // Control `resolve` arguments by letting Array#slice cast boolean `noValue` to integer: + // * false: [ value ].slice( 0 ) => resolve( value ) + // * true: [ value ].slice( 1 ) => resolve() + resolve.apply( undefined, [ value ].slice( noValue ) ); + } + + // For Promises/A+, convert exceptions into rejections + // Since jQuery.when doesn't unwrap thenables, we can skip the extra checks appearing in + // Deferred#then to conditionally suppress rejection. + } catch ( value ) { + + // Support: Android 4.0 only + // Strict mode functions invoked without .call/.apply get global-object context + reject.apply( undefined, [ value ] ); + } +} + +jQuery.extend( { + + Deferred: function( func ) { + var tuples = [ + + // action, add listener, callbacks, + // ... .then handlers, argument index, [final state] + [ "notify", "progress", jQuery.Callbacks( "memory" ), + jQuery.Callbacks( "memory" ), 2 ], + [ "resolve", "done", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 0, "resolved" ], + [ "reject", "fail", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 1, "rejected" ] + ], + state = "pending", + promise = { + state: function() { + return state; + }, + always: function() { + deferred.done( arguments ).fail( arguments ); + return this; + }, + "catch": function( fn ) { + return promise.then( null, fn ); + }, + + // Keep pipe for back-compat + pipe: function( /* fnDone, fnFail, fnProgress */ ) { + var fns = arguments; + + return jQuery.Deferred( function( newDefer ) { + jQuery.each( tuples, function( _i, tuple ) { + + // Map tuples (progress, done, fail) to arguments (done, fail, progress) + var fn = isFunction( fns[ tuple[ 4 ] ] ) && fns[ tuple[ 4 ] ]; + + // deferred.progress(function() { bind to newDefer or newDefer.notify }) + // deferred.done(function() { bind to newDefer or newDefer.resolve }) + // deferred.fail(function() { bind to newDefer or newDefer.reject }) + deferred[ tuple[ 1 ] ]( function() { + var returned = fn && fn.apply( this, arguments ); + if ( returned && isFunction( returned.promise ) ) { + returned.promise() + .progress( newDefer.notify ) + .done( newDefer.resolve ) + .fail( newDefer.reject ); + } else { + newDefer[ tuple[ 0 ] + "With" ]( + this, + fn ? [ returned ] : arguments + ); + } + } ); + } ); + fns = null; + } ).promise(); + }, + then: function( onFulfilled, onRejected, onProgress ) { + var maxDepth = 0; + function resolve( depth, deferred, handler, special ) { + return function() { + var that = this, + args = arguments, + mightThrow = function() { + var returned, then; + + // Support: Promises/A+ section 2.3.3.3.3 + // https://promisesaplus.com/#point-59 + // Ignore double-resolution attempts + if ( depth < maxDepth ) { + return; + } + + returned = handler.apply( that, args ); + + // Support: Promises/A+ section 2.3.1 + // https://promisesaplus.com/#point-48 + if ( returned === deferred.promise() ) { + throw new TypeError( "Thenable self-resolution" ); + } + + // Support: Promises/A+ sections 2.3.3.1, 3.5 + // https://promisesaplus.com/#point-54 + // https://promisesaplus.com/#point-75 + // Retrieve `then` only once + then = returned && + + // Support: Promises/A+ section 2.3.4 + // https://promisesaplus.com/#point-64 + // Only check objects and functions for thenability + ( typeof returned === "object" || + typeof returned === "function" ) && + returned.then; + + // Handle a returned thenable + if ( isFunction( then ) ) { + + // Special processors (notify) just wait for resolution + if ( special ) { + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ) + ); + + // Normal processors (resolve) also hook into progress + } else { + + // ...and disregard older resolution values + maxDepth++; + + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ), + resolve( maxDepth, deferred, Identity, + deferred.notifyWith ) + ); + } + + // Handle all other returned values + } else { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Identity ) { + that = undefined; + args = [ returned ]; + } + + // Process the value(s) + // Default process is resolve + ( special || deferred.resolveWith )( that, args ); + } + }, + + // Only normal processors (resolve) catch and reject exceptions + process = special ? + mightThrow : + function() { + try { + mightThrow(); + } catch ( e ) { + + if ( jQuery.Deferred.exceptionHook ) { + jQuery.Deferred.exceptionHook( e, + process.stackTrace ); + } + + // Support: Promises/A+ section 2.3.3.3.4.1 + // https://promisesaplus.com/#point-61 + // Ignore post-resolution exceptions + if ( depth + 1 >= maxDepth ) { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Thrower ) { + that = undefined; + args = [ e ]; + } + + deferred.rejectWith( that, args ); + } + } + }; + + // Support: Promises/A+ section 2.3.3.3.1 + // https://promisesaplus.com/#point-57 + // Re-resolve promises immediately to dodge false rejection from + // subsequent errors + if ( depth ) { + process(); + } else { + + // Call an optional hook to record the stack, in case of exception + // since it's otherwise lost when execution goes async + if ( jQuery.Deferred.getStackHook ) { + process.stackTrace = jQuery.Deferred.getStackHook(); + } + window.setTimeout( process ); + } + }; + } + + return jQuery.Deferred( function( newDefer ) { + + // progress_handlers.add( ... ) + tuples[ 0 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onProgress ) ? + onProgress : + Identity, + newDefer.notifyWith + ) + ); + + // fulfilled_handlers.add( ... ) + tuples[ 1 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onFulfilled ) ? + onFulfilled : + Identity + ) + ); + + // rejected_handlers.add( ... ) + tuples[ 2 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onRejected ) ? + onRejected : + Thrower + ) + ); + } ).promise(); + }, + + // Get a promise for this deferred + // If obj is provided, the promise aspect is added to the object + promise: function( obj ) { + return obj != null ? jQuery.extend( obj, promise ) : promise; + } + }, + deferred = {}; + + // Add list-specific methods + jQuery.each( tuples, function( i, tuple ) { + var list = tuple[ 2 ], + stateString = tuple[ 5 ]; + + // promise.progress = list.add + // promise.done = list.add + // promise.fail = list.add + promise[ tuple[ 1 ] ] = list.add; + + // Handle state + if ( stateString ) { + list.add( + function() { + + // state = "resolved" (i.e., fulfilled) + // state = "rejected" + state = stateString; + }, + + // rejected_callbacks.disable + // fulfilled_callbacks.disable + tuples[ 3 - i ][ 2 ].disable, + + // rejected_handlers.disable + // fulfilled_handlers.disable + tuples[ 3 - i ][ 3 ].disable, + + // progress_callbacks.lock + tuples[ 0 ][ 2 ].lock, + + // progress_handlers.lock + tuples[ 0 ][ 3 ].lock + ); + } + + // progress_handlers.fire + // fulfilled_handlers.fire + // rejected_handlers.fire + list.add( tuple[ 3 ].fire ); + + // deferred.notify = function() { deferred.notifyWith(...) } + // deferred.resolve = function() { deferred.resolveWith(...) } + // deferred.reject = function() { deferred.rejectWith(...) } + deferred[ tuple[ 0 ] ] = function() { + deferred[ tuple[ 0 ] + "With" ]( this === deferred ? undefined : this, arguments ); + return this; + }; + + // deferred.notifyWith = list.fireWith + // deferred.resolveWith = list.fireWith + // deferred.rejectWith = list.fireWith + deferred[ tuple[ 0 ] + "With" ] = list.fireWith; + } ); + + // Make the deferred a promise + promise.promise( deferred ); + + // Call given func if any + if ( func ) { + func.call( deferred, deferred ); + } + + // All done! + return deferred; + }, + + // Deferred helper + when: function( singleValue ) { + var + + // count of uncompleted subordinates + remaining = arguments.length, + + // count of unprocessed arguments + i = remaining, + + // subordinate fulfillment data + resolveContexts = Array( i ), + resolveValues = slice.call( arguments ), + + // the primary Deferred + primary = jQuery.Deferred(), + + // subordinate callback factory + updateFunc = function( i ) { + return function( value ) { + resolveContexts[ i ] = this; + resolveValues[ i ] = arguments.length > 1 ? slice.call( arguments ) : value; + if ( !( --remaining ) ) { + primary.resolveWith( resolveContexts, resolveValues ); + } + }; + }; + + // Single- and empty arguments are adopted like Promise.resolve + if ( remaining <= 1 ) { + adoptValue( singleValue, primary.done( updateFunc( i ) ).resolve, primary.reject, + !remaining ); + + // Use .then() to unwrap secondary thenables (cf. gh-3000) + if ( primary.state() === "pending" || + isFunction( resolveValues[ i ] && resolveValues[ i ].then ) ) { + + return primary.then(); + } + } + + // Multiple arguments are aggregated like Promise.all array elements + while ( i-- ) { + adoptValue( resolveValues[ i ], updateFunc( i ), primary.reject ); + } + + return primary.promise(); + } +} ); + + +// These usually indicate a programmer mistake during development, +// warn about them ASAP rather than swallowing them by default. +var rerrorNames = /^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/; + +jQuery.Deferred.exceptionHook = function( error, stack ) { + + // Support: IE 8 - 9 only + // Console exists when dev tools are open, which can happen at any time + if ( window.console && window.console.warn && error && rerrorNames.test( error.name ) ) { + window.console.warn( "jQuery.Deferred exception: " + error.message, error.stack, stack ); + } +}; + + + + +jQuery.readyException = function( error ) { + window.setTimeout( function() { + throw error; + } ); +}; + + + + +// The deferred used on DOM ready +var readyList = jQuery.Deferred(); + +jQuery.fn.ready = function( fn ) { + + readyList + .then( fn ) + + // Wrap jQuery.readyException in a function so that the lookup + // happens at the time of error handling instead of callback + // registration. + .catch( function( error ) { + jQuery.readyException( error ); + } ); + + return this; +}; + +jQuery.extend( { + + // Is the DOM ready to be used? Set to true once it occurs. + isReady: false, + + // A counter to track how many items to wait for before + // the ready event fires. See #6781 + readyWait: 1, + + // Handle when the DOM is ready + ready: function( wait ) { + + // Abort if there are pending holds or we're already ready + if ( wait === true ? --jQuery.readyWait : jQuery.isReady ) { + return; + } + + // Remember that the DOM is ready + jQuery.isReady = true; + + // If a normal DOM Ready event fired, decrement, and wait if need be + if ( wait !== true && --jQuery.readyWait > 0 ) { + return; + } + + // If there are functions bound, to execute + readyList.resolveWith( document, [ jQuery ] ); + } +} ); + +jQuery.ready.then = readyList.then; + +// The ready event handler and self cleanup method +function completed() { + document.removeEventListener( "DOMContentLoaded", completed ); + window.removeEventListener( "load", completed ); + jQuery.ready(); +} + +// Catch cases where $(document).ready() is called +// after the browser event has already occurred. +// Support: IE <=9 - 10 only +// Older IE sometimes signals "interactive" too soon +if ( document.readyState === "complete" || + ( document.readyState !== "loading" && !document.documentElement.doScroll ) ) { + + // Handle it asynchronously to allow scripts the opportunity to delay ready + window.setTimeout( jQuery.ready ); + +} else { + + // Use the handy event callback + document.addEventListener( "DOMContentLoaded", completed ); + + // A fallback to window.onload, that will always work + window.addEventListener( "load", completed ); +} + + + + +// Multifunctional method to get and set values of a collection +// The value/s can optionally be executed if it's a function +var access = function( elems, fn, key, value, chainable, emptyGet, raw ) { + var i = 0, + len = elems.length, + bulk = key == null; + + // Sets many values + if ( toType( key ) === "object" ) { + chainable = true; + for ( i in key ) { + access( elems, fn, i, key[ i ], true, emptyGet, raw ); + } + + // Sets one value + } else if ( value !== undefined ) { + chainable = true; + + if ( !isFunction( value ) ) { + raw = true; + } + + if ( bulk ) { + + // Bulk operations run against the entire set + if ( raw ) { + fn.call( elems, value ); + fn = null; + + // ...except when executing function values + } else { + bulk = fn; + fn = function( elem, _key, value ) { + return bulk.call( jQuery( elem ), value ); + }; + } + } + + if ( fn ) { + for ( ; i < len; i++ ) { + fn( + elems[ i ], key, raw ? + value : + value.call( elems[ i ], i, fn( elems[ i ], key ) ) + ); + } + } + } + + if ( chainable ) { + return elems; + } + + // Gets + if ( bulk ) { + return fn.call( elems ); + } + + return len ? fn( elems[ 0 ], key ) : emptyGet; +}; + + +// Matches dashed string for camelizing +var rmsPrefix = /^-ms-/, + rdashAlpha = /-([a-z])/g; + +// Used by camelCase as callback to replace() +function fcamelCase( _all, letter ) { + return letter.toUpperCase(); +} + +// Convert dashed to camelCase; used by the css and data modules +// Support: IE <=9 - 11, Edge 12 - 15 +// Microsoft forgot to hump their vendor prefix (#9572) +function camelCase( string ) { + return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase ); +} +var acceptData = function( owner ) { + + // Accepts only: + // - Node + // - Node.ELEMENT_NODE + // - Node.DOCUMENT_NODE + // - Object + // - Any + return owner.nodeType === 1 || owner.nodeType === 9 || !( +owner.nodeType ); +}; + + + + +function Data() { + this.expando = jQuery.expando + Data.uid++; +} + +Data.uid = 1; + +Data.prototype = { + + cache: function( owner ) { + + // Check if the owner object already has a cache + var value = owner[ this.expando ]; + + // If not, create one + if ( !value ) { + value = {}; + + // We can accept data for non-element nodes in modern browsers, + // but we should not, see #8335. + // Always return an empty object. + if ( acceptData( owner ) ) { + + // If it is a node unlikely to be stringify-ed or looped over + // use plain assignment + if ( owner.nodeType ) { + owner[ this.expando ] = value; + + // Otherwise secure it in a non-enumerable property + // configurable must be true to allow the property to be + // deleted when data is removed + } else { + Object.defineProperty( owner, this.expando, { + value: value, + configurable: true + } ); + } + } + } + + return value; + }, + set: function( owner, data, value ) { + var prop, + cache = this.cache( owner ); + + // Handle: [ owner, key, value ] args + // Always use camelCase key (gh-2257) + if ( typeof data === "string" ) { + cache[ camelCase( data ) ] = value; + + // Handle: [ owner, { properties } ] args + } else { + + // Copy the properties one-by-one to the cache object + for ( prop in data ) { + cache[ camelCase( prop ) ] = data[ prop ]; + } + } + return cache; + }, + get: function( owner, key ) { + return key === undefined ? + this.cache( owner ) : + + // Always use camelCase key (gh-2257) + owner[ this.expando ] && owner[ this.expando ][ camelCase( key ) ]; + }, + access: function( owner, key, value ) { + + // In cases where either: + // + // 1. No key was specified + // 2. A string key was specified, but no value provided + // + // Take the "read" path and allow the get method to determine + // which value to return, respectively either: + // + // 1. The entire cache object + // 2. The data stored at the key + // + if ( key === undefined || + ( ( key && typeof key === "string" ) && value === undefined ) ) { + + return this.get( owner, key ); + } + + // When the key is not a string, or both a key and value + // are specified, set or extend (existing objects) with either: + // + // 1. An object of properties + // 2. A key and value + // + this.set( owner, key, value ); + + // Since the "set" path can have two possible entry points + // return the expected data based on which path was taken[*] + return value !== undefined ? value : key; + }, + remove: function( owner, key ) { + var i, + cache = owner[ this.expando ]; + + if ( cache === undefined ) { + return; + } + + if ( key !== undefined ) { + + // Support array or space separated string of keys + if ( Array.isArray( key ) ) { + + // If key is an array of keys... + // We always set camelCase keys, so remove that. + key = key.map( camelCase ); + } else { + key = camelCase( key ); + + // If a key with the spaces exists, use it. + // Otherwise, create an array by matching non-whitespace + key = key in cache ? + [ key ] : + ( key.match( rnothtmlwhite ) || [] ); + } + + i = key.length; + + while ( i-- ) { + delete cache[ key[ i ] ]; + } + } + + // Remove the expando if there's no more data + if ( key === undefined || jQuery.isEmptyObject( cache ) ) { + + // Support: Chrome <=35 - 45 + // Webkit & Blink performance suffers when deleting properties + // from DOM nodes, so set to undefined instead + // https://bugs.chromium.org/p/chromium/issues/detail?id=378607 (bug restricted) + if ( owner.nodeType ) { + owner[ this.expando ] = undefined; + } else { + delete owner[ this.expando ]; + } + } + }, + hasData: function( owner ) { + var cache = owner[ this.expando ]; + return cache !== undefined && !jQuery.isEmptyObject( cache ); + } +}; +var dataPriv = new Data(); + +var dataUser = new Data(); + + + +// Implementation Summary +// +// 1. Enforce API surface and semantic compatibility with 1.9.x branch +// 2. Improve the module's maintainability by reducing the storage +// paths to a single mechanism. +// 3. Use the same single mechanism to support "private" and "user" data. +// 4. _Never_ expose "private" data to user code (TODO: Drop _data, _removeData) +// 5. Avoid exposing implementation details on user objects (eg. expando properties) +// 6. Provide a clear path for implementation upgrade to WeakMap in 2014 + +var rbrace = /^(?:\{[\w\W]*\}|\[[\w\W]*\])$/, + rmultiDash = /[A-Z]/g; + +function getData( data ) { + if ( data === "true" ) { + return true; + } + + if ( data === "false" ) { + return false; + } + + if ( data === "null" ) { + return null; + } + + // Only convert to a number if it doesn't change the string + if ( data === +data + "" ) { + return +data; + } + + if ( rbrace.test( data ) ) { + return JSON.parse( data ); + } + + return data; +} + +function dataAttr( elem, key, data ) { + var name; + + // If nothing was found internally, try to fetch any + // data from the HTML5 data-* attribute + if ( data === undefined && elem.nodeType === 1 ) { + name = "data-" + key.replace( rmultiDash, "-$&" ).toLowerCase(); + data = elem.getAttribute( name ); + + if ( typeof data === "string" ) { + try { + data = getData( data ); + } catch ( e ) {} + + // Make sure we set the data so it isn't changed later + dataUser.set( elem, key, data ); + } else { + data = undefined; + } + } + return data; +} + +jQuery.extend( { + hasData: function( elem ) { + return dataUser.hasData( elem ) || dataPriv.hasData( elem ); + }, + + data: function( elem, name, data ) { + return dataUser.access( elem, name, data ); + }, + + removeData: function( elem, name ) { + dataUser.remove( elem, name ); + }, + + // TODO: Now that all calls to _data and _removeData have been replaced + // with direct calls to dataPriv methods, these can be deprecated. + _data: function( elem, name, data ) { + return dataPriv.access( elem, name, data ); + }, + + _removeData: function( elem, name ) { + dataPriv.remove( elem, name ); + } +} ); + +jQuery.fn.extend( { + data: function( key, value ) { + var i, name, data, + elem = this[ 0 ], + attrs = elem && elem.attributes; + + // Gets all values + if ( key === undefined ) { + if ( this.length ) { + data = dataUser.get( elem ); + + if ( elem.nodeType === 1 && !dataPriv.get( elem, "hasDataAttrs" ) ) { + i = attrs.length; + while ( i-- ) { + + // Support: IE 11 only + // The attrs elements can be null (#14894) + if ( attrs[ i ] ) { + name = attrs[ i ].name; + if ( name.indexOf( "data-" ) === 0 ) { + name = camelCase( name.slice( 5 ) ); + dataAttr( elem, name, data[ name ] ); + } + } + } + dataPriv.set( elem, "hasDataAttrs", true ); + } + } + + return data; + } + + // Sets multiple values + if ( typeof key === "object" ) { + return this.each( function() { + dataUser.set( this, key ); + } ); + } + + return access( this, function( value ) { + var data; + + // The calling jQuery object (element matches) is not empty + // (and therefore has an element appears at this[ 0 ]) and the + // `value` parameter was not undefined. An empty jQuery object + // will result in `undefined` for elem = this[ 0 ] which will + // throw an exception if an attempt to read a data cache is made. + if ( elem && value === undefined ) { + + // Attempt to get data from the cache + // The key will always be camelCased in Data + data = dataUser.get( elem, key ); + if ( data !== undefined ) { + return data; + } + + // Attempt to "discover" the data in + // HTML5 custom data-* attrs + data = dataAttr( elem, key ); + if ( data !== undefined ) { + return data; + } + + // We tried really hard, but the data doesn't exist. + return; + } + + // Set the data... + this.each( function() { + + // We always store the camelCased key + dataUser.set( this, key, value ); + } ); + }, null, value, arguments.length > 1, null, true ); + }, + + removeData: function( key ) { + return this.each( function() { + dataUser.remove( this, key ); + } ); + } +} ); + + +jQuery.extend( { + queue: function( elem, type, data ) { + var queue; + + if ( elem ) { + type = ( type || "fx" ) + "queue"; + queue = dataPriv.get( elem, type ); + + // Speed up dequeue by getting out quickly if this is just a lookup + if ( data ) { + if ( !queue || Array.isArray( data ) ) { + queue = dataPriv.access( elem, type, jQuery.makeArray( data ) ); + } else { + queue.push( data ); + } + } + return queue || []; + } + }, + + dequeue: function( elem, type ) { + type = type || "fx"; + + var queue = jQuery.queue( elem, type ), + startLength = queue.length, + fn = queue.shift(), + hooks = jQuery._queueHooks( elem, type ), + next = function() { + jQuery.dequeue( elem, type ); + }; + + // If the fx queue is dequeued, always remove the progress sentinel + if ( fn === "inprogress" ) { + fn = queue.shift(); + startLength--; + } + + if ( fn ) { + + // Add a progress sentinel to prevent the fx queue from being + // automatically dequeued + if ( type === "fx" ) { + queue.unshift( "inprogress" ); + } + + // Clear up the last queue stop function + delete hooks.stop; + fn.call( elem, next, hooks ); + } + + if ( !startLength && hooks ) { + hooks.empty.fire(); + } + }, + + // Not public - generate a queueHooks object, or return the current one + _queueHooks: function( elem, type ) { + var key = type + "queueHooks"; + return dataPriv.get( elem, key ) || dataPriv.access( elem, key, { + empty: jQuery.Callbacks( "once memory" ).add( function() { + dataPriv.remove( elem, [ type + "queue", key ] ); + } ) + } ); + } +} ); + +jQuery.fn.extend( { + queue: function( type, data ) { + var setter = 2; + + if ( typeof type !== "string" ) { + data = type; + type = "fx"; + setter--; + } + + if ( arguments.length < setter ) { + return jQuery.queue( this[ 0 ], type ); + } + + return data === undefined ? + this : + this.each( function() { + var queue = jQuery.queue( this, type, data ); + + // Ensure a hooks for this queue + jQuery._queueHooks( this, type ); + + if ( type === "fx" && queue[ 0 ] !== "inprogress" ) { + jQuery.dequeue( this, type ); + } + } ); + }, + dequeue: function( type ) { + return this.each( function() { + jQuery.dequeue( this, type ); + } ); + }, + clearQueue: function( type ) { + return this.queue( type || "fx", [] ); + }, + + // Get a promise resolved when queues of a certain type + // are emptied (fx is the type by default) + promise: function( type, obj ) { + var tmp, + count = 1, + defer = jQuery.Deferred(), + elements = this, + i = this.length, + resolve = function() { + if ( !( --count ) ) { + defer.resolveWith( elements, [ elements ] ); + } + }; + + if ( typeof type !== "string" ) { + obj = type; + type = undefined; + } + type = type || "fx"; + + while ( i-- ) { + tmp = dataPriv.get( elements[ i ], type + "queueHooks" ); + if ( tmp && tmp.empty ) { + count++; + tmp.empty.add( resolve ); + } + } + resolve(); + return defer.promise( obj ); + } +} ); +var pnum = ( /[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/ ).source; + +var rcssNum = new RegExp( "^(?:([+-])=|)(" + pnum + ")([a-z%]*)$", "i" ); + + +var cssExpand = [ "Top", "Right", "Bottom", "Left" ]; + +var documentElement = document.documentElement; + + + + var isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ); + }, + composed = { composed: true }; + + // Support: IE 9 - 11+, Edge 12 - 18+, iOS 10.0 - 10.2 only + // Check attachment across shadow DOM boundaries when possible (gh-3504) + // Support: iOS 10.0-10.2 only + // Early iOS 10 versions support `attachShadow` but not `getRootNode`, + // leading to errors. We need to check for `getRootNode`. + if ( documentElement.getRootNode ) { + isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ) || + elem.getRootNode( composed ) === elem.ownerDocument; + }; + } +var isHiddenWithinTree = function( elem, el ) { + + // isHiddenWithinTree might be called from jQuery#filter function; + // in that case, element will be second argument + elem = el || elem; + + // Inline style trumps all + return elem.style.display === "none" || + elem.style.display === "" && + + // Otherwise, check computed style + // Support: Firefox <=43 - 45 + // Disconnected elements can have computed display: none, so first confirm that elem is + // in the document. + isAttached( elem ) && + + jQuery.css( elem, "display" ) === "none"; + }; + + + +function adjustCSS( elem, prop, valueParts, tween ) { + var adjusted, scale, + maxIterations = 20, + currentValue = tween ? + function() { + return tween.cur(); + } : + function() { + return jQuery.css( elem, prop, "" ); + }, + initial = currentValue(), + unit = valueParts && valueParts[ 3 ] || ( jQuery.cssNumber[ prop ] ? "" : "px" ), + + // Starting value computation is required for potential unit mismatches + initialInUnit = elem.nodeType && + ( jQuery.cssNumber[ prop ] || unit !== "px" && +initial ) && + rcssNum.exec( jQuery.css( elem, prop ) ); + + if ( initialInUnit && initialInUnit[ 3 ] !== unit ) { + + // Support: Firefox <=54 + // Halve the iteration target value to prevent interference from CSS upper bounds (gh-2144) + initial = initial / 2; + + // Trust units reported by jQuery.css + unit = unit || initialInUnit[ 3 ]; + + // Iteratively approximate from a nonzero starting point + initialInUnit = +initial || 1; + + while ( maxIterations-- ) { + + // Evaluate and update our best guess (doubling guesses that zero out). + // Finish if the scale equals or crosses 1 (making the old*new product non-positive). + jQuery.style( elem, prop, initialInUnit + unit ); + if ( ( 1 - scale ) * ( 1 - ( scale = currentValue() / initial || 0.5 ) ) <= 0 ) { + maxIterations = 0; + } + initialInUnit = initialInUnit / scale; + + } + + initialInUnit = initialInUnit * 2; + jQuery.style( elem, prop, initialInUnit + unit ); + + // Make sure we update the tween properties later on + valueParts = valueParts || []; + } + + if ( valueParts ) { + initialInUnit = +initialInUnit || +initial || 0; + + // Apply relative offset (+=/-=) if specified + adjusted = valueParts[ 1 ] ? + initialInUnit + ( valueParts[ 1 ] + 1 ) * valueParts[ 2 ] : + +valueParts[ 2 ]; + if ( tween ) { + tween.unit = unit; + tween.start = initialInUnit; + tween.end = adjusted; + } + } + return adjusted; +} + + +var defaultDisplayMap = {}; + +function getDefaultDisplay( elem ) { + var temp, + doc = elem.ownerDocument, + nodeName = elem.nodeName, + display = defaultDisplayMap[ nodeName ]; + + if ( display ) { + return display; + } + + temp = doc.body.appendChild( doc.createElement( nodeName ) ); + display = jQuery.css( temp, "display" ); + + temp.parentNode.removeChild( temp ); + + if ( display === "none" ) { + display = "block"; + } + defaultDisplayMap[ nodeName ] = display; + + return display; +} + +function showHide( elements, show ) { + var display, elem, + values = [], + index = 0, + length = elements.length; + + // Determine new display value for elements that need to change + for ( ; index < length; index++ ) { + elem = elements[ index ]; + if ( !elem.style ) { + continue; + } + + display = elem.style.display; + if ( show ) { + + // Since we force visibility upon cascade-hidden elements, an immediate (and slow) + // check is required in this first loop unless we have a nonempty display value (either + // inline or about-to-be-restored) + if ( display === "none" ) { + values[ index ] = dataPriv.get( elem, "display" ) || null; + if ( !values[ index ] ) { + elem.style.display = ""; + } + } + if ( elem.style.display === "" && isHiddenWithinTree( elem ) ) { + values[ index ] = getDefaultDisplay( elem ); + } + } else { + if ( display !== "none" ) { + values[ index ] = "none"; + + // Remember what we're overwriting + dataPriv.set( elem, "display", display ); + } + } + } + + // Set the display of the elements in a second loop to avoid constant reflow + for ( index = 0; index < length; index++ ) { + if ( values[ index ] != null ) { + elements[ index ].style.display = values[ index ]; + } + } + + return elements; +} + +jQuery.fn.extend( { + show: function() { + return showHide( this, true ); + }, + hide: function() { + return showHide( this ); + }, + toggle: function( state ) { + if ( typeof state === "boolean" ) { + return state ? this.show() : this.hide(); + } + + return this.each( function() { + if ( isHiddenWithinTree( this ) ) { + jQuery( this ).show(); + } else { + jQuery( this ).hide(); + } + } ); + } +} ); +var rcheckableType = ( /^(?:checkbox|radio)$/i ); + +var rtagName = ( /<([a-z][^\/\0>\x20\t\r\n\f]*)/i ); + +var rscriptType = ( /^$|^module$|\/(?:java|ecma)script/i ); + + + +( function() { + var fragment = document.createDocumentFragment(), + div = fragment.appendChild( document.createElement( "div" ) ), + input = document.createElement( "input" ); + + // Support: Android 4.0 - 4.3 only + // Check state lost if the name is set (#11217) + // Support: Windows Web Apps (WWA) + // `name` and `type` must use .setAttribute for WWA (#14901) + input.setAttribute( "type", "radio" ); + input.setAttribute( "checked", "checked" ); + input.setAttribute( "name", "t" ); + + div.appendChild( input ); + + // Support: Android <=4.1 only + // Older WebKit doesn't clone checked state correctly in fragments + support.checkClone = div.cloneNode( true ).cloneNode( true ).lastChild.checked; + + // Support: IE <=11 only + // Make sure textarea (and checkbox) defaultValue is properly cloned + div.innerHTML = ""; + support.noCloneChecked = !!div.cloneNode( true ).lastChild.defaultValue; + + // Support: IE <=9 only + // IE <=9 replaces "; + support.option = !!div.lastChild; +} )(); + + +// We have to close these tags to support XHTML (#13200) +var wrapMap = { + + // XHTML parsers do not magically insert elements in the + // same way that tag soup parsers do. So we cannot shorten + // this by omitting or other required elements. + thead: [ 1, "", "
" ], + col: [ 2, "", "
" ], + tr: [ 2, "", "
" ], + td: [ 3, "", "
" ], + + _default: [ 0, "", "" ] +}; + +wrapMap.tbody = wrapMap.tfoot = wrapMap.colgroup = wrapMap.caption = wrapMap.thead; +wrapMap.th = wrapMap.td; + +// Support: IE <=9 only +if ( !support.option ) { + wrapMap.optgroup = wrapMap.option = [ 1, "" ]; +} + + +function getAll( context, tag ) { + + // Support: IE <=9 - 11 only + // Use typeof to avoid zero-argument method invocation on host objects (#15151) + var ret; + + if ( typeof context.getElementsByTagName !== "undefined" ) { + ret = context.getElementsByTagName( tag || "*" ); + + } else if ( typeof context.querySelectorAll !== "undefined" ) { + ret = context.querySelectorAll( tag || "*" ); + + } else { + ret = []; + } + + if ( tag === undefined || tag && nodeName( context, tag ) ) { + return jQuery.merge( [ context ], ret ); + } + + return ret; +} + + +// Mark scripts as having already been evaluated +function setGlobalEval( elems, refElements ) { + var i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + dataPriv.set( + elems[ i ], + "globalEval", + !refElements || dataPriv.get( refElements[ i ], "globalEval" ) + ); + } +} + + +var rhtml = /<|&#?\w+;/; + +function buildFragment( elems, context, scripts, selection, ignored ) { + var elem, tmp, tag, wrap, attached, j, + fragment = context.createDocumentFragment(), + nodes = [], + i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + elem = elems[ i ]; + + if ( elem || elem === 0 ) { + + // Add nodes directly + if ( toType( elem ) === "object" ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, elem.nodeType ? [ elem ] : elem ); + + // Convert non-html into a text node + } else if ( !rhtml.test( elem ) ) { + nodes.push( context.createTextNode( elem ) ); + + // Convert html into DOM nodes + } else { + tmp = tmp || fragment.appendChild( context.createElement( "div" ) ); + + // Deserialize a standard representation + tag = ( rtagName.exec( elem ) || [ "", "" ] )[ 1 ].toLowerCase(); + wrap = wrapMap[ tag ] || wrapMap._default; + tmp.innerHTML = wrap[ 1 ] + jQuery.htmlPrefilter( elem ) + wrap[ 2 ]; + + // Descend through wrappers to the right content + j = wrap[ 0 ]; + while ( j-- ) { + tmp = tmp.lastChild; + } + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, tmp.childNodes ); + + // Remember the top-level container + tmp = fragment.firstChild; + + // Ensure the created nodes are orphaned (#12392) + tmp.textContent = ""; + } + } + } + + // Remove wrapper from fragment + fragment.textContent = ""; + + i = 0; + while ( ( elem = nodes[ i++ ] ) ) { + + // Skip elements already in the context collection (trac-4087) + if ( selection && jQuery.inArray( elem, selection ) > -1 ) { + if ( ignored ) { + ignored.push( elem ); + } + continue; + } + + attached = isAttached( elem ); + + // Append to fragment + tmp = getAll( fragment.appendChild( elem ), "script" ); + + // Preserve script evaluation history + if ( attached ) { + setGlobalEval( tmp ); + } + + // Capture executables + if ( scripts ) { + j = 0; + while ( ( elem = tmp[ j++ ] ) ) { + if ( rscriptType.test( elem.type || "" ) ) { + scripts.push( elem ); + } + } + } + } + + return fragment; +} + + +var rtypenamespace = /^([^.]*)(?:\.(.+)|)/; + +function returnTrue() { + return true; +} + +function returnFalse() { + return false; +} + +// Support: IE <=9 - 11+ +// focus() and blur() are asynchronous, except when they are no-op. +// So expect focus to be synchronous when the element is already active, +// and blur to be synchronous when the element is not already active. +// (focus and blur are always synchronous in other supported browsers, +// this just defines when we can count on it). +function expectSync( elem, type ) { + return ( elem === safeActiveElement() ) === ( type === "focus" ); +} + +// Support: IE <=9 only +// Accessing document.activeElement can throw unexpectedly +// https://bugs.jquery.com/ticket/13393 +function safeActiveElement() { + try { + return document.activeElement; + } catch ( err ) { } +} + +function on( elem, types, selector, data, fn, one ) { + var origFn, type; + + // Types can be a map of types/handlers + if ( typeof types === "object" ) { + + // ( types-Object, selector, data ) + if ( typeof selector !== "string" ) { + + // ( types-Object, data ) + data = data || selector; + selector = undefined; + } + for ( type in types ) { + on( elem, type, selector, data, types[ type ], one ); + } + return elem; + } + + if ( data == null && fn == null ) { + + // ( types, fn ) + fn = selector; + data = selector = undefined; + } else if ( fn == null ) { + if ( typeof selector === "string" ) { + + // ( types, selector, fn ) + fn = data; + data = undefined; + } else { + + // ( types, data, fn ) + fn = data; + data = selector; + selector = undefined; + } + } + if ( fn === false ) { + fn = returnFalse; + } else if ( !fn ) { + return elem; + } + + if ( one === 1 ) { + origFn = fn; + fn = function( event ) { + + // Can use an empty set, since event contains the info + jQuery().off( event ); + return origFn.apply( this, arguments ); + }; + + // Use same guid so caller can remove using origFn + fn.guid = origFn.guid || ( origFn.guid = jQuery.guid++ ); + } + return elem.each( function() { + jQuery.event.add( this, types, fn, data, selector ); + } ); +} + +/* + * Helper functions for managing events -- not part of the public interface. + * Props to Dean Edwards' addEvent library for many of the ideas. + */ +jQuery.event = { + + global: {}, + + add: function( elem, types, handler, data, selector ) { + + var handleObjIn, eventHandle, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.get( elem ); + + // Only attach events to objects that accept data + if ( !acceptData( elem ) ) { + return; + } + + // Caller can pass in an object of custom data in lieu of the handler + if ( handler.handler ) { + handleObjIn = handler; + handler = handleObjIn.handler; + selector = handleObjIn.selector; + } + + // Ensure that invalid selectors throw exceptions at attach time + // Evaluate against documentElement in case elem is a non-element node (e.g., document) + if ( selector ) { + jQuery.find.matchesSelector( documentElement, selector ); + } + + // Make sure that the handler has a unique ID, used to find/remove it later + if ( !handler.guid ) { + handler.guid = jQuery.guid++; + } + + // Init the element's event structure and main handler, if this is the first + if ( !( events = elemData.events ) ) { + events = elemData.events = Object.create( null ); + } + if ( !( eventHandle = elemData.handle ) ) { + eventHandle = elemData.handle = function( e ) { + + // Discard the second event of a jQuery.event.trigger() and + // when an event is called after a page has unloaded + return typeof jQuery !== "undefined" && jQuery.event.triggered !== e.type ? + jQuery.event.dispatch.apply( elem, arguments ) : undefined; + }; + } + + // Handle multiple events separated by a space + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // There *must* be a type, no attaching namespace-only handlers + if ( !type ) { + continue; + } + + // If event changes its type, use the special event handlers for the changed type + special = jQuery.event.special[ type ] || {}; + + // If selector defined, determine special event api type, otherwise given type + type = ( selector ? special.delegateType : special.bindType ) || type; + + // Update special based on newly reset type + special = jQuery.event.special[ type ] || {}; + + // handleObj is passed to all event handlers + handleObj = jQuery.extend( { + type: type, + origType: origType, + data: data, + handler: handler, + guid: handler.guid, + selector: selector, + needsContext: selector && jQuery.expr.match.needsContext.test( selector ), + namespace: namespaces.join( "." ) + }, handleObjIn ); + + // Init the event handler queue if we're the first + if ( !( handlers = events[ type ] ) ) { + handlers = events[ type ] = []; + handlers.delegateCount = 0; + + // Only use addEventListener if the special events handler returns false + if ( !special.setup || + special.setup.call( elem, data, namespaces, eventHandle ) === false ) { + + if ( elem.addEventListener ) { + elem.addEventListener( type, eventHandle ); + } + } + } + + if ( special.add ) { + special.add.call( elem, handleObj ); + + if ( !handleObj.handler.guid ) { + handleObj.handler.guid = handler.guid; + } + } + + // Add to the element's handler list, delegates in front + if ( selector ) { + handlers.splice( handlers.delegateCount++, 0, handleObj ); + } else { + handlers.push( handleObj ); + } + + // Keep track of which events have ever been used, for event optimization + jQuery.event.global[ type ] = true; + } + + }, + + // Detach an event or set of events from an element + remove: function( elem, types, handler, selector, mappedTypes ) { + + var j, origCount, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.hasData( elem ) && dataPriv.get( elem ); + + if ( !elemData || !( events = elemData.events ) ) { + return; + } + + // Once for each type.namespace in types; type may be omitted + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // Unbind all events (on this namespace, if provided) for the element + if ( !type ) { + for ( type in events ) { + jQuery.event.remove( elem, type + types[ t ], handler, selector, true ); + } + continue; + } + + special = jQuery.event.special[ type ] || {}; + type = ( selector ? special.delegateType : special.bindType ) || type; + handlers = events[ type ] || []; + tmp = tmp[ 2 ] && + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ); + + // Remove matching events + origCount = j = handlers.length; + while ( j-- ) { + handleObj = handlers[ j ]; + + if ( ( mappedTypes || origType === handleObj.origType ) && + ( !handler || handler.guid === handleObj.guid ) && + ( !tmp || tmp.test( handleObj.namespace ) ) && + ( !selector || selector === handleObj.selector || + selector === "**" && handleObj.selector ) ) { + handlers.splice( j, 1 ); + + if ( handleObj.selector ) { + handlers.delegateCount--; + } + if ( special.remove ) { + special.remove.call( elem, handleObj ); + } + } + } + + // Remove generic event handler if we removed something and no more handlers exist + // (avoids potential for endless recursion during removal of special event handlers) + if ( origCount && !handlers.length ) { + if ( !special.teardown || + special.teardown.call( elem, namespaces, elemData.handle ) === false ) { + + jQuery.removeEvent( elem, type, elemData.handle ); + } + + delete events[ type ]; + } + } + + // Remove data and the expando if it's no longer used + if ( jQuery.isEmptyObject( events ) ) { + dataPriv.remove( elem, "handle events" ); + } + }, + + dispatch: function( nativeEvent ) { + + var i, j, ret, matched, handleObj, handlerQueue, + args = new Array( arguments.length ), + + // Make a writable jQuery.Event from the native event object + event = jQuery.event.fix( nativeEvent ), + + handlers = ( + dataPriv.get( this, "events" ) || Object.create( null ) + )[ event.type ] || [], + special = jQuery.event.special[ event.type ] || {}; + + // Use the fix-ed jQuery.Event rather than the (read-only) native event + args[ 0 ] = event; + + for ( i = 1; i < arguments.length; i++ ) { + args[ i ] = arguments[ i ]; + } + + event.delegateTarget = this; + + // Call the preDispatch hook for the mapped type, and let it bail if desired + if ( special.preDispatch && special.preDispatch.call( this, event ) === false ) { + return; + } + + // Determine handlers + handlerQueue = jQuery.event.handlers.call( this, event, handlers ); + + // Run delegates first; they may want to stop propagation beneath us + i = 0; + while ( ( matched = handlerQueue[ i++ ] ) && !event.isPropagationStopped() ) { + event.currentTarget = matched.elem; + + j = 0; + while ( ( handleObj = matched.handlers[ j++ ] ) && + !event.isImmediatePropagationStopped() ) { + + // If the event is namespaced, then each handler is only invoked if it is + // specially universal or its namespaces are a superset of the event's. + if ( !event.rnamespace || handleObj.namespace === false || + event.rnamespace.test( handleObj.namespace ) ) { + + event.handleObj = handleObj; + event.data = handleObj.data; + + ret = ( ( jQuery.event.special[ handleObj.origType ] || {} ).handle || + handleObj.handler ).apply( matched.elem, args ); + + if ( ret !== undefined ) { + if ( ( event.result = ret ) === false ) { + event.preventDefault(); + event.stopPropagation(); + } + } + } + } + } + + // Call the postDispatch hook for the mapped type + if ( special.postDispatch ) { + special.postDispatch.call( this, event ); + } + + return event.result; + }, + + handlers: function( event, handlers ) { + var i, handleObj, sel, matchedHandlers, matchedSelectors, + handlerQueue = [], + delegateCount = handlers.delegateCount, + cur = event.target; + + // Find delegate handlers + if ( delegateCount && + + // Support: IE <=9 + // Black-hole SVG instance trees (trac-13180) + cur.nodeType && + + // Support: Firefox <=42 + // Suppress spec-violating clicks indicating a non-primary pointer button (trac-3861) + // https://www.w3.org/TR/DOM-Level-3-Events/#event-type-click + // Support: IE 11 only + // ...but not arrow key "clicks" of radio inputs, which can have `button` -1 (gh-2343) + !( event.type === "click" && event.button >= 1 ) ) { + + for ( ; cur !== this; cur = cur.parentNode || this ) { + + // Don't check non-elements (#13208) + // Don't process clicks on disabled elements (#6911, #8165, #11382, #11764) + if ( cur.nodeType === 1 && !( event.type === "click" && cur.disabled === true ) ) { + matchedHandlers = []; + matchedSelectors = {}; + for ( i = 0; i < delegateCount; i++ ) { + handleObj = handlers[ i ]; + + // Don't conflict with Object.prototype properties (#13203) + sel = handleObj.selector + " "; + + if ( matchedSelectors[ sel ] === undefined ) { + matchedSelectors[ sel ] = handleObj.needsContext ? + jQuery( sel, this ).index( cur ) > -1 : + jQuery.find( sel, this, null, [ cur ] ).length; + } + if ( matchedSelectors[ sel ] ) { + matchedHandlers.push( handleObj ); + } + } + if ( matchedHandlers.length ) { + handlerQueue.push( { elem: cur, handlers: matchedHandlers } ); + } + } + } + } + + // Add the remaining (directly-bound) handlers + cur = this; + if ( delegateCount < handlers.length ) { + handlerQueue.push( { elem: cur, handlers: handlers.slice( delegateCount ) } ); + } + + return handlerQueue; + }, + + addProp: function( name, hook ) { + Object.defineProperty( jQuery.Event.prototype, name, { + enumerable: true, + configurable: true, + + get: isFunction( hook ) ? + function() { + if ( this.originalEvent ) { + return hook( this.originalEvent ); + } + } : + function() { + if ( this.originalEvent ) { + return this.originalEvent[ name ]; + } + }, + + set: function( value ) { + Object.defineProperty( this, name, { + enumerable: true, + configurable: true, + writable: true, + value: value + } ); + } + } ); + }, + + fix: function( originalEvent ) { + return originalEvent[ jQuery.expando ] ? + originalEvent : + new jQuery.Event( originalEvent ); + }, + + special: { + load: { + + // Prevent triggered image.load events from bubbling to window.load + noBubble: true + }, + click: { + + // Utilize native event to ensure correct state for checkable inputs + setup: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Claim the first handler + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + // dataPriv.set( el, "click", ... ) + leverageNative( el, "click", returnTrue ); + } + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Force setup before triggering a click + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + leverageNative( el, "click" ); + } + + // Return non-false to allow normal event-path propagation + return true; + }, + + // For cross-browser consistency, suppress native .click() on links + // Also prevent it if we're currently inside a leveraged native-event stack + _default: function( event ) { + var target = event.target; + return rcheckableType.test( target.type ) && + target.click && nodeName( target, "input" ) && + dataPriv.get( target, "click" ) || + nodeName( target, "a" ); + } + }, + + beforeunload: { + postDispatch: function( event ) { + + // Support: Firefox 20+ + // Firefox doesn't alert if the returnValue field is not set. + if ( event.result !== undefined && event.originalEvent ) { + event.originalEvent.returnValue = event.result; + } + } + } + } +}; + +// Ensure the presence of an event listener that handles manually-triggered +// synthetic events by interrupting progress until reinvoked in response to +// *native* events that it fires directly, ensuring that state changes have +// already occurred before other listeners are invoked. +function leverageNative( el, type, expectSync ) { + + // Missing expectSync indicates a trigger call, which must force setup through jQuery.event.add + if ( !expectSync ) { + if ( dataPriv.get( el, type ) === undefined ) { + jQuery.event.add( el, type, returnTrue ); + } + return; + } + + // Register the controller as a special universal handler for all event namespaces + dataPriv.set( el, type, false ); + jQuery.event.add( el, type, { + namespace: false, + handler: function( event ) { + var notAsync, result, + saved = dataPriv.get( this, type ); + + if ( ( event.isTrigger & 1 ) && this[ type ] ) { + + // Interrupt processing of the outer synthetic .trigger()ed event + // Saved data should be false in such cases, but might be a leftover capture object + // from an async native handler (gh-4350) + if ( !saved.length ) { + + // Store arguments for use when handling the inner native event + // There will always be at least one argument (an event object), so this array + // will not be confused with a leftover capture object. + saved = slice.call( arguments ); + dataPriv.set( this, type, saved ); + + // Trigger the native event and capture its result + // Support: IE <=9 - 11+ + // focus() and blur() are asynchronous + notAsync = expectSync( this, type ); + this[ type ](); + result = dataPriv.get( this, type ); + if ( saved !== result || notAsync ) { + dataPriv.set( this, type, false ); + } else { + result = {}; + } + if ( saved !== result ) { + + // Cancel the outer synthetic event + event.stopImmediatePropagation(); + event.preventDefault(); + + // Support: Chrome 86+ + // In Chrome, if an element having a focusout handler is blurred by + // clicking outside of it, it invokes the handler synchronously. If + // that handler calls `.remove()` on the element, the data is cleared, + // leaving `result` undefined. We need to guard against this. + return result && result.value; + } + + // If this is an inner synthetic event for an event with a bubbling surrogate + // (focus or blur), assume that the surrogate already propagated from triggering the + // native event and prevent that from happening again here. + // This technically gets the ordering wrong w.r.t. to `.trigger()` (in which the + // bubbling surrogate propagates *after* the non-bubbling base), but that seems + // less bad than duplication. + } else if ( ( jQuery.event.special[ type ] || {} ).delegateType ) { + event.stopPropagation(); + } + + // If this is a native event triggered above, everything is now in order + // Fire an inner synthetic event with the original arguments + } else if ( saved.length ) { + + // ...and capture the result + dataPriv.set( this, type, { + value: jQuery.event.trigger( + + // Support: IE <=9 - 11+ + // Extend with the prototype to reset the above stopImmediatePropagation() + jQuery.extend( saved[ 0 ], jQuery.Event.prototype ), + saved.slice( 1 ), + this + ) + } ); + + // Abort handling of the native event + event.stopImmediatePropagation(); + } + } + } ); +} + +jQuery.removeEvent = function( elem, type, handle ) { + + // This "if" is needed for plain objects + if ( elem.removeEventListener ) { + elem.removeEventListener( type, handle ); + } +}; + +jQuery.Event = function( src, props ) { + + // Allow instantiation without the 'new' keyword + if ( !( this instanceof jQuery.Event ) ) { + return new jQuery.Event( src, props ); + } + + // Event object + if ( src && src.type ) { + this.originalEvent = src; + this.type = src.type; + + // Events bubbling up the document may have been marked as prevented + // by a handler lower down the tree; reflect the correct value. + this.isDefaultPrevented = src.defaultPrevented || + src.defaultPrevented === undefined && + + // Support: Android <=2.3 only + src.returnValue === false ? + returnTrue : + returnFalse; + + // Create target properties + // Support: Safari <=6 - 7 only + // Target should not be a text node (#504, #13143) + this.target = ( src.target && src.target.nodeType === 3 ) ? + src.target.parentNode : + src.target; + + this.currentTarget = src.currentTarget; + this.relatedTarget = src.relatedTarget; + + // Event type + } else { + this.type = src; + } + + // Put explicitly provided properties onto the event object + if ( props ) { + jQuery.extend( this, props ); + } + + // Create a timestamp if incoming event doesn't have one + this.timeStamp = src && src.timeStamp || Date.now(); + + // Mark it as fixed + this[ jQuery.expando ] = true; +}; + +// jQuery.Event is based on DOM3 Events as specified by the ECMAScript Language Binding +// https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html +jQuery.Event.prototype = { + constructor: jQuery.Event, + isDefaultPrevented: returnFalse, + isPropagationStopped: returnFalse, + isImmediatePropagationStopped: returnFalse, + isSimulated: false, + + preventDefault: function() { + var e = this.originalEvent; + + this.isDefaultPrevented = returnTrue; + + if ( e && !this.isSimulated ) { + e.preventDefault(); + } + }, + stopPropagation: function() { + var e = this.originalEvent; + + this.isPropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopPropagation(); + } + }, + stopImmediatePropagation: function() { + var e = this.originalEvent; + + this.isImmediatePropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopImmediatePropagation(); + } + + this.stopPropagation(); + } +}; + +// Includes all common event props including KeyEvent and MouseEvent specific props +jQuery.each( { + altKey: true, + bubbles: true, + cancelable: true, + changedTouches: true, + ctrlKey: true, + detail: true, + eventPhase: true, + metaKey: true, + pageX: true, + pageY: true, + shiftKey: true, + view: true, + "char": true, + code: true, + charCode: true, + key: true, + keyCode: true, + button: true, + buttons: true, + clientX: true, + clientY: true, + offsetX: true, + offsetY: true, + pointerId: true, + pointerType: true, + screenX: true, + screenY: true, + targetTouches: true, + toElement: true, + touches: true, + which: true +}, jQuery.event.addProp ); + +jQuery.each( { focus: "focusin", blur: "focusout" }, function( type, delegateType ) { + jQuery.event.special[ type ] = { + + // Utilize native event if possible so blur/focus sequence is correct + setup: function() { + + // Claim the first handler + // dataPriv.set( this, "focus", ... ) + // dataPriv.set( this, "blur", ... ) + leverageNative( this, type, expectSync ); + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function() { + + // Force setup before trigger + leverageNative( this, type ); + + // Return non-false to allow normal event-path propagation + return true; + }, + + // Suppress native focus or blur as it's already being fired + // in leverageNative. + _default: function() { + return true; + }, + + delegateType: delegateType + }; +} ); + +// Create mouseenter/leave events using mouseover/out and event-time checks +// so that event delegation works in jQuery. +// Do the same for pointerenter/pointerleave and pointerover/pointerout +// +// Support: Safari 7 only +// Safari sends mouseenter too often; see: +// https://bugs.chromium.org/p/chromium/issues/detail?id=470258 +// for the description of the bug (it existed in older Chrome versions as well). +jQuery.each( { + mouseenter: "mouseover", + mouseleave: "mouseout", + pointerenter: "pointerover", + pointerleave: "pointerout" +}, function( orig, fix ) { + jQuery.event.special[ orig ] = { + delegateType: fix, + bindType: fix, + + handle: function( event ) { + var ret, + target = this, + related = event.relatedTarget, + handleObj = event.handleObj; + + // For mouseenter/leave call the handler if related is outside the target. + // NB: No relatedTarget if the mouse left/entered the browser window + if ( !related || ( related !== target && !jQuery.contains( target, related ) ) ) { + event.type = handleObj.origType; + ret = handleObj.handler.apply( this, arguments ); + event.type = fix; + } + return ret; + } + }; +} ); + +jQuery.fn.extend( { + + on: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn ); + }, + one: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn, 1 ); + }, + off: function( types, selector, fn ) { + var handleObj, type; + if ( types && types.preventDefault && types.handleObj ) { + + // ( event ) dispatched jQuery.Event + handleObj = types.handleObj; + jQuery( types.delegateTarget ).off( + handleObj.namespace ? + handleObj.origType + "." + handleObj.namespace : + handleObj.origType, + handleObj.selector, + handleObj.handler + ); + return this; + } + if ( typeof types === "object" ) { + + // ( types-object [, selector] ) + for ( type in types ) { + this.off( type, selector, types[ type ] ); + } + return this; + } + if ( selector === false || typeof selector === "function" ) { + + // ( types [, fn] ) + fn = selector; + selector = undefined; + } + if ( fn === false ) { + fn = returnFalse; + } + return this.each( function() { + jQuery.event.remove( this, types, fn, selector ); + } ); + } +} ); + + +var + + // Support: IE <=10 - 11, Edge 12 - 13 only + // In IE/Edge using regex groups here causes severe slowdowns. + // See https://connect.microsoft.com/IE/feedback/details/1736512/ + rnoInnerhtml = /\s*$/g; + +// Prefer a tbody over its parent table for containing new rows +function manipulationTarget( elem, content ) { + if ( nodeName( elem, "table" ) && + nodeName( content.nodeType !== 11 ? content : content.firstChild, "tr" ) ) { + + return jQuery( elem ).children( "tbody" )[ 0 ] || elem; + } + + return elem; +} + +// Replace/restore the type attribute of script elements for safe DOM manipulation +function disableScript( elem ) { + elem.type = ( elem.getAttribute( "type" ) !== null ) + "/" + elem.type; + return elem; +} +function restoreScript( elem ) { + if ( ( elem.type || "" ).slice( 0, 5 ) === "true/" ) { + elem.type = elem.type.slice( 5 ); + } else { + elem.removeAttribute( "type" ); + } + + return elem; +} + +function cloneCopyEvent( src, dest ) { + var i, l, type, pdataOld, udataOld, udataCur, events; + + if ( dest.nodeType !== 1 ) { + return; + } + + // 1. Copy private data: events, handlers, etc. + if ( dataPriv.hasData( src ) ) { + pdataOld = dataPriv.get( src ); + events = pdataOld.events; + + if ( events ) { + dataPriv.remove( dest, "handle events" ); + + for ( type in events ) { + for ( i = 0, l = events[ type ].length; i < l; i++ ) { + jQuery.event.add( dest, type, events[ type ][ i ] ); + } + } + } + } + + // 2. Copy user data + if ( dataUser.hasData( src ) ) { + udataOld = dataUser.access( src ); + udataCur = jQuery.extend( {}, udataOld ); + + dataUser.set( dest, udataCur ); + } +} + +// Fix IE bugs, see support tests +function fixInput( src, dest ) { + var nodeName = dest.nodeName.toLowerCase(); + + // Fails to persist the checked state of a cloned checkbox or radio button. + if ( nodeName === "input" && rcheckableType.test( src.type ) ) { + dest.checked = src.checked; + + // Fails to return the selected option to the default selected state when cloning options + } else if ( nodeName === "input" || nodeName === "textarea" ) { + dest.defaultValue = src.defaultValue; + } +} + +function domManip( collection, args, callback, ignored ) { + + // Flatten any nested arrays + args = flat( args ); + + var fragment, first, scripts, hasScripts, node, doc, + i = 0, + l = collection.length, + iNoClone = l - 1, + value = args[ 0 ], + valueIsFunction = isFunction( value ); + + // We can't cloneNode fragments that contain checked, in WebKit + if ( valueIsFunction || + ( l > 1 && typeof value === "string" && + !support.checkClone && rchecked.test( value ) ) ) { + return collection.each( function( index ) { + var self = collection.eq( index ); + if ( valueIsFunction ) { + args[ 0 ] = value.call( this, index, self.html() ); + } + domManip( self, args, callback, ignored ); + } ); + } + + if ( l ) { + fragment = buildFragment( args, collection[ 0 ].ownerDocument, false, collection, ignored ); + first = fragment.firstChild; + + if ( fragment.childNodes.length === 1 ) { + fragment = first; + } + + // Require either new content or an interest in ignored elements to invoke the callback + if ( first || ignored ) { + scripts = jQuery.map( getAll( fragment, "script" ), disableScript ); + hasScripts = scripts.length; + + // Use the original fragment for the last item + // instead of the first because it can end up + // being emptied incorrectly in certain situations (#8070). + for ( ; i < l; i++ ) { + node = fragment; + + if ( i !== iNoClone ) { + node = jQuery.clone( node, true, true ); + + // Keep references to cloned scripts for later restoration + if ( hasScripts ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( scripts, getAll( node, "script" ) ); + } + } + + callback.call( collection[ i ], node, i ); + } + + if ( hasScripts ) { + doc = scripts[ scripts.length - 1 ].ownerDocument; + + // Reenable scripts + jQuery.map( scripts, restoreScript ); + + // Evaluate executable scripts on first document insertion + for ( i = 0; i < hasScripts; i++ ) { + node = scripts[ i ]; + if ( rscriptType.test( node.type || "" ) && + !dataPriv.access( node, "globalEval" ) && + jQuery.contains( doc, node ) ) { + + if ( node.src && ( node.type || "" ).toLowerCase() !== "module" ) { + + // Optional AJAX dependency, but won't run scripts if not present + if ( jQuery._evalUrl && !node.noModule ) { + jQuery._evalUrl( node.src, { + nonce: node.nonce || node.getAttribute( "nonce" ) + }, doc ); + } + } else { + DOMEval( node.textContent.replace( rcleanScript, "" ), node, doc ); + } + } + } + } + } + } + + return collection; +} + +function remove( elem, selector, keepData ) { + var node, + nodes = selector ? jQuery.filter( selector, elem ) : elem, + i = 0; + + for ( ; ( node = nodes[ i ] ) != null; i++ ) { + if ( !keepData && node.nodeType === 1 ) { + jQuery.cleanData( getAll( node ) ); + } + + if ( node.parentNode ) { + if ( keepData && isAttached( node ) ) { + setGlobalEval( getAll( node, "script" ) ); + } + node.parentNode.removeChild( node ); + } + } + + return elem; +} + +jQuery.extend( { + htmlPrefilter: function( html ) { + return html; + }, + + clone: function( elem, dataAndEvents, deepDataAndEvents ) { + var i, l, srcElements, destElements, + clone = elem.cloneNode( true ), + inPage = isAttached( elem ); + + // Fix IE cloning issues + if ( !support.noCloneChecked && ( elem.nodeType === 1 || elem.nodeType === 11 ) && + !jQuery.isXMLDoc( elem ) ) { + + // We eschew Sizzle here for performance reasons: https://jsperf.com/getall-vs-sizzle/2 + destElements = getAll( clone ); + srcElements = getAll( elem ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + fixInput( srcElements[ i ], destElements[ i ] ); + } + } + + // Copy the events from the original to the clone + if ( dataAndEvents ) { + if ( deepDataAndEvents ) { + srcElements = srcElements || getAll( elem ); + destElements = destElements || getAll( clone ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + cloneCopyEvent( srcElements[ i ], destElements[ i ] ); + } + } else { + cloneCopyEvent( elem, clone ); + } + } + + // Preserve script evaluation history + destElements = getAll( clone, "script" ); + if ( destElements.length > 0 ) { + setGlobalEval( destElements, !inPage && getAll( elem, "script" ) ); + } + + // Return the cloned set + return clone; + }, + + cleanData: function( elems ) { + var data, elem, type, + special = jQuery.event.special, + i = 0; + + for ( ; ( elem = elems[ i ] ) !== undefined; i++ ) { + if ( acceptData( elem ) ) { + if ( ( data = elem[ dataPriv.expando ] ) ) { + if ( data.events ) { + for ( type in data.events ) { + if ( special[ type ] ) { + jQuery.event.remove( elem, type ); + + // This is a shortcut to avoid jQuery.event.remove's overhead + } else { + jQuery.removeEvent( elem, type, data.handle ); + } + } + } + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataPriv.expando ] = undefined; + } + if ( elem[ dataUser.expando ] ) { + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataUser.expando ] = undefined; + } + } + } + } +} ); + +jQuery.fn.extend( { + detach: function( selector ) { + return remove( this, selector, true ); + }, + + remove: function( selector ) { + return remove( this, selector ); + }, + + text: function( value ) { + return access( this, function( value ) { + return value === undefined ? + jQuery.text( this ) : + this.empty().each( function() { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + this.textContent = value; + } + } ); + }, null, value, arguments.length ); + }, + + append: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.appendChild( elem ); + } + } ); + }, + + prepend: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.insertBefore( elem, target.firstChild ); + } + } ); + }, + + before: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this ); + } + } ); + }, + + after: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this.nextSibling ); + } + } ); + }, + + empty: function() { + var elem, + i = 0; + + for ( ; ( elem = this[ i ] ) != null; i++ ) { + if ( elem.nodeType === 1 ) { + + // Prevent memory leaks + jQuery.cleanData( getAll( elem, false ) ); + + // Remove any remaining nodes + elem.textContent = ""; + } + } + + return this; + }, + + clone: function( dataAndEvents, deepDataAndEvents ) { + dataAndEvents = dataAndEvents == null ? false : dataAndEvents; + deepDataAndEvents = deepDataAndEvents == null ? dataAndEvents : deepDataAndEvents; + + return this.map( function() { + return jQuery.clone( this, dataAndEvents, deepDataAndEvents ); + } ); + }, + + html: function( value ) { + return access( this, function( value ) { + var elem = this[ 0 ] || {}, + i = 0, + l = this.length; + + if ( value === undefined && elem.nodeType === 1 ) { + return elem.innerHTML; + } + + // See if we can take a shortcut and just use innerHTML + if ( typeof value === "string" && !rnoInnerhtml.test( value ) && + !wrapMap[ ( rtagName.exec( value ) || [ "", "" ] )[ 1 ].toLowerCase() ] ) { + + value = jQuery.htmlPrefilter( value ); + + try { + for ( ; i < l; i++ ) { + elem = this[ i ] || {}; + + // Remove element nodes and prevent memory leaks + if ( elem.nodeType === 1 ) { + jQuery.cleanData( getAll( elem, false ) ); + elem.innerHTML = value; + } + } + + elem = 0; + + // If using innerHTML throws an exception, use the fallback method + } catch ( e ) {} + } + + if ( elem ) { + this.empty().append( value ); + } + }, null, value, arguments.length ); + }, + + replaceWith: function() { + var ignored = []; + + // Make the changes, replacing each non-ignored context element with the new content + return domManip( this, arguments, function( elem ) { + var parent = this.parentNode; + + if ( jQuery.inArray( this, ignored ) < 0 ) { + jQuery.cleanData( getAll( this ) ); + if ( parent ) { + parent.replaceChild( elem, this ); + } + } + + // Force callback invocation + }, ignored ); + } +} ); + +jQuery.each( { + appendTo: "append", + prependTo: "prepend", + insertBefore: "before", + insertAfter: "after", + replaceAll: "replaceWith" +}, function( name, original ) { + jQuery.fn[ name ] = function( selector ) { + var elems, + ret = [], + insert = jQuery( selector ), + last = insert.length - 1, + i = 0; + + for ( ; i <= last; i++ ) { + elems = i === last ? this : this.clone( true ); + jQuery( insert[ i ] )[ original ]( elems ); + + // Support: Android <=4.0 only, PhantomJS 1 only + // .get() because push.apply(_, arraylike) throws on ancient WebKit + push.apply( ret, elems.get() ); + } + + return this.pushStack( ret ); + }; +} ); +var rnumnonpx = new RegExp( "^(" + pnum + ")(?!px)[a-z%]+$", "i" ); + +var getStyles = function( elem ) { + + // Support: IE <=11 only, Firefox <=30 (#15098, #14150) + // IE throws on elements created in popups + // FF meanwhile throws on frame elements through "defaultView.getComputedStyle" + var view = elem.ownerDocument.defaultView; + + if ( !view || !view.opener ) { + view = window; + } + + return view.getComputedStyle( elem ); + }; + +var swap = function( elem, options, callback ) { + var ret, name, + old = {}; + + // Remember the old values, and insert the new ones + for ( name in options ) { + old[ name ] = elem.style[ name ]; + elem.style[ name ] = options[ name ]; + } + + ret = callback.call( elem ); + + // Revert the old values + for ( name in options ) { + elem.style[ name ] = old[ name ]; + } + + return ret; +}; + + +var rboxStyle = new RegExp( cssExpand.join( "|" ), "i" ); + + + +( function() { + + // Executing both pixelPosition & boxSizingReliable tests require only one layout + // so they're executed at the same time to save the second computation. + function computeStyleTests() { + + // This is a singleton, we need to execute it only once + if ( !div ) { + return; + } + + container.style.cssText = "position:absolute;left:-11111px;width:60px;" + + "margin-top:1px;padding:0;border:0"; + div.style.cssText = + "position:relative;display:block;box-sizing:border-box;overflow:scroll;" + + "margin:auto;border:1px;padding:1px;" + + "width:60%;top:1%"; + documentElement.appendChild( container ).appendChild( div ); + + var divStyle = window.getComputedStyle( div ); + pixelPositionVal = divStyle.top !== "1%"; + + // Support: Android 4.0 - 4.3 only, Firefox <=3 - 44 + reliableMarginLeftVal = roundPixelMeasures( divStyle.marginLeft ) === 12; + + // Support: Android 4.0 - 4.3 only, Safari <=9.1 - 10.1, iOS <=7.0 - 9.3 + // Some styles come back with percentage values, even though they shouldn't + div.style.right = "60%"; + pixelBoxStylesVal = roundPixelMeasures( divStyle.right ) === 36; + + // Support: IE 9 - 11 only + // Detect misreporting of content dimensions for box-sizing:border-box elements + boxSizingReliableVal = roundPixelMeasures( divStyle.width ) === 36; + + // Support: IE 9 only + // Detect overflow:scroll screwiness (gh-3699) + // Support: Chrome <=64 + // Don't get tricked when zoom affects offsetWidth (gh-4029) + div.style.position = "absolute"; + scrollboxSizeVal = roundPixelMeasures( div.offsetWidth / 3 ) === 12; + + documentElement.removeChild( container ); + + // Nullify the div so it wouldn't be stored in the memory and + // it will also be a sign that checks already performed + div = null; + } + + function roundPixelMeasures( measure ) { + return Math.round( parseFloat( measure ) ); + } + + var pixelPositionVal, boxSizingReliableVal, scrollboxSizeVal, pixelBoxStylesVal, + reliableTrDimensionsVal, reliableMarginLeftVal, + container = document.createElement( "div" ), + div = document.createElement( "div" ); + + // Finish early in limited (non-browser) environments + if ( !div.style ) { + return; + } + + // Support: IE <=9 - 11 only + // Style of cloned element affects source element cloned (#8908) + div.style.backgroundClip = "content-box"; + div.cloneNode( true ).style.backgroundClip = ""; + support.clearCloneStyle = div.style.backgroundClip === "content-box"; + + jQuery.extend( support, { + boxSizingReliable: function() { + computeStyleTests(); + return boxSizingReliableVal; + }, + pixelBoxStyles: function() { + computeStyleTests(); + return pixelBoxStylesVal; + }, + pixelPosition: function() { + computeStyleTests(); + return pixelPositionVal; + }, + reliableMarginLeft: function() { + computeStyleTests(); + return reliableMarginLeftVal; + }, + scrollboxSize: function() { + computeStyleTests(); + return scrollboxSizeVal; + }, + + // Support: IE 9 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Behavior in IE 9 is more subtle than in newer versions & it passes + // some versions of this test; make sure not to make it pass there! + // + // Support: Firefox 70+ + // Only Firefox includes border widths + // in computed dimensions. (gh-4529) + reliableTrDimensions: function() { + var table, tr, trChild, trStyle; + if ( reliableTrDimensionsVal == null ) { + table = document.createElement( "table" ); + tr = document.createElement( "tr" ); + trChild = document.createElement( "div" ); + + table.style.cssText = "position:absolute;left:-11111px;border-collapse:separate"; + tr.style.cssText = "border:1px solid"; + + // Support: Chrome 86+ + // Height set through cssText does not get applied. + // Computed height then comes back as 0. + tr.style.height = "1px"; + trChild.style.height = "9px"; + + // Support: Android 8 Chrome 86+ + // In our bodyBackground.html iframe, + // display for all div elements is set to "inline", + // which causes a problem only in Android 8 Chrome 86. + // Ensuring the div is display: block + // gets around this issue. + trChild.style.display = "block"; + + documentElement + .appendChild( table ) + .appendChild( tr ) + .appendChild( trChild ); + + trStyle = window.getComputedStyle( tr ); + reliableTrDimensionsVal = ( parseInt( trStyle.height, 10 ) + + parseInt( trStyle.borderTopWidth, 10 ) + + parseInt( trStyle.borderBottomWidth, 10 ) ) === tr.offsetHeight; + + documentElement.removeChild( table ); + } + return reliableTrDimensionsVal; + } + } ); +} )(); + + +function curCSS( elem, name, computed ) { + var width, minWidth, maxWidth, ret, + + // Support: Firefox 51+ + // Retrieving style before computed somehow + // fixes an issue with getting wrong values + // on detached elements + style = elem.style; + + computed = computed || getStyles( elem ); + + // getPropertyValue is needed for: + // .css('filter') (IE 9 only, #12537) + // .css('--customProperty) (#3144) + if ( computed ) { + ret = computed.getPropertyValue( name ) || computed[ name ]; + + if ( ret === "" && !isAttached( elem ) ) { + ret = jQuery.style( elem, name ); + } + + // A tribute to the "awesome hack by Dean Edwards" + // Android Browser returns percentage for some values, + // but width seems to be reliably pixels. + // This is against the CSSOM draft spec: + // https://drafts.csswg.org/cssom/#resolved-values + if ( !support.pixelBoxStyles() && rnumnonpx.test( ret ) && rboxStyle.test( name ) ) { + + // Remember the original values + width = style.width; + minWidth = style.minWidth; + maxWidth = style.maxWidth; + + // Put in the new values to get a computed value out + style.minWidth = style.maxWidth = style.width = ret; + ret = computed.width; + + // Revert the changed values + style.width = width; + style.minWidth = minWidth; + style.maxWidth = maxWidth; + } + } + + return ret !== undefined ? + + // Support: IE <=9 - 11 only + // IE returns zIndex value as an integer. + ret + "" : + ret; +} + + +function addGetHookIf( conditionFn, hookFn ) { + + // Define the hook, we'll check on the first run if it's really needed. + return { + get: function() { + if ( conditionFn() ) { + + // Hook not needed (or it's not possible to use it due + // to missing dependency), remove it. + delete this.get; + return; + } + + // Hook needed; redefine it so that the support test is not executed again. + return ( this.get = hookFn ).apply( this, arguments ); + } + }; +} + + +var cssPrefixes = [ "Webkit", "Moz", "ms" ], + emptyStyle = document.createElement( "div" ).style, + vendorProps = {}; + +// Return a vendor-prefixed property or undefined +function vendorPropName( name ) { + + // Check for vendor prefixed names + var capName = name[ 0 ].toUpperCase() + name.slice( 1 ), + i = cssPrefixes.length; + + while ( i-- ) { + name = cssPrefixes[ i ] + capName; + if ( name in emptyStyle ) { + return name; + } + } +} + +// Return a potentially-mapped jQuery.cssProps or vendor prefixed property +function finalPropName( name ) { + var final = jQuery.cssProps[ name ] || vendorProps[ name ]; + + if ( final ) { + return final; + } + if ( name in emptyStyle ) { + return name; + } + return vendorProps[ name ] = vendorPropName( name ) || name; +} + + +var + + // Swappable if display is none or starts with table + // except "table", "table-cell", or "table-caption" + // See here for display values: https://developer.mozilla.org/en-US/docs/CSS/display + rdisplayswap = /^(none|table(?!-c[ea]).+)/, + rcustomProp = /^--/, + cssShow = { position: "absolute", visibility: "hidden", display: "block" }, + cssNormalTransform = { + letterSpacing: "0", + fontWeight: "400" + }; + +function setPositiveNumber( _elem, value, subtract ) { + + // Any relative (+/-) values have already been + // normalized at this point + var matches = rcssNum.exec( value ); + return matches ? + + // Guard against undefined "subtract", e.g., when used as in cssHooks + Math.max( 0, matches[ 2 ] - ( subtract || 0 ) ) + ( matches[ 3 ] || "px" ) : + value; +} + +function boxModelAdjustment( elem, dimension, box, isBorderBox, styles, computedVal ) { + var i = dimension === "width" ? 1 : 0, + extra = 0, + delta = 0; + + // Adjustment may not be necessary + if ( box === ( isBorderBox ? "border" : "content" ) ) { + return 0; + } + + for ( ; i < 4; i += 2 ) { + + // Both box models exclude margin + if ( box === "margin" ) { + delta += jQuery.css( elem, box + cssExpand[ i ], true, styles ); + } + + // If we get here with a content-box, we're seeking "padding" or "border" or "margin" + if ( !isBorderBox ) { + + // Add padding + delta += jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + + // For "border" or "margin", add border + if ( box !== "padding" ) { + delta += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + + // But still keep track of it otherwise + } else { + extra += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + + // If we get here with a border-box (content + padding + border), we're seeking "content" or + // "padding" or "margin" + } else { + + // For "content", subtract padding + if ( box === "content" ) { + delta -= jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + } + + // For "content" or "padding", subtract border + if ( box !== "margin" ) { + delta -= jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + } + } + + // Account for positive content-box scroll gutter when requested by providing computedVal + if ( !isBorderBox && computedVal >= 0 ) { + + // offsetWidth/offsetHeight is a rounded sum of content, padding, scroll gutter, and border + // Assuming integer scroll gutter, subtract the rest and round down + delta += Math.max( 0, Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + computedVal - + delta - + extra - + 0.5 + + // If offsetWidth/offsetHeight is unknown, then we can't determine content-box scroll gutter + // Use an explicit zero to avoid NaN (gh-3964) + ) ) || 0; + } + + return delta; +} + +function getWidthOrHeight( elem, dimension, extra ) { + + // Start with computed style + var styles = getStyles( elem ), + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-4322). + // Fake content-box until we know it's needed to know the true value. + boxSizingNeeded = !support.boxSizingReliable() || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + valueIsBorderBox = isBorderBox, + + val = curCSS( elem, dimension, styles ), + offsetProp = "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ); + + // Support: Firefox <=54 + // Return a confounding non-pixel value or feign ignorance, as appropriate. + if ( rnumnonpx.test( val ) ) { + if ( !extra ) { + return val; + } + val = "auto"; + } + + + // Support: IE 9 - 11 only + // Use offsetWidth/offsetHeight for when box sizing is unreliable. + // In those cases, the computed value can be trusted to be border-box. + if ( ( !support.boxSizingReliable() && isBorderBox || + + // Support: IE 10 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Interestingly, in some cases IE 9 doesn't suffer from this issue. + !support.reliableTrDimensions() && nodeName( elem, "tr" ) || + + // Fall back to offsetWidth/offsetHeight when value is "auto" + // This happens for inline elements with no explicit setting (gh-3571) + val === "auto" || + + // Support: Android <=4.1 - 4.3 only + // Also use offsetWidth/offsetHeight for misreported inline dimensions (gh-3602) + !parseFloat( val ) && jQuery.css( elem, "display", false, styles ) === "inline" ) && + + // Make sure the element is visible & connected + elem.getClientRects().length ) { + + isBorderBox = jQuery.css( elem, "boxSizing", false, styles ) === "border-box"; + + // Where available, offsetWidth/offsetHeight approximate border box dimensions. + // Where not available (e.g., SVG), assume unreliable box-sizing and interpret the + // retrieved value as a content box dimension. + valueIsBorderBox = offsetProp in elem; + if ( valueIsBorderBox ) { + val = elem[ offsetProp ]; + } + } + + // Normalize "" and auto + val = parseFloat( val ) || 0; + + // Adjust for the element's box model + return ( val + + boxModelAdjustment( + elem, + dimension, + extra || ( isBorderBox ? "border" : "content" ), + valueIsBorderBox, + styles, + + // Provide the current computed size to request scroll gutter calculation (gh-3589) + val + ) + ) + "px"; +} + +jQuery.extend( { + + // Add in style property hooks for overriding the default + // behavior of getting and setting a style property + cssHooks: { + opacity: { + get: function( elem, computed ) { + if ( computed ) { + + // We should always get a number back from opacity + var ret = curCSS( elem, "opacity" ); + return ret === "" ? "1" : ret; + } + } + } + }, + + // Don't automatically add "px" to these possibly-unitless properties + cssNumber: { + "animationIterationCount": true, + "columnCount": true, + "fillOpacity": true, + "flexGrow": true, + "flexShrink": true, + "fontWeight": true, + "gridArea": true, + "gridColumn": true, + "gridColumnEnd": true, + "gridColumnStart": true, + "gridRow": true, + "gridRowEnd": true, + "gridRowStart": true, + "lineHeight": true, + "opacity": true, + "order": true, + "orphans": true, + "widows": true, + "zIndex": true, + "zoom": true + }, + + // Add in properties whose names you wish to fix before + // setting or getting the value + cssProps: {}, + + // Get and set the style property on a DOM Node + style: function( elem, name, value, extra ) { + + // Don't set styles on text and comment nodes + if ( !elem || elem.nodeType === 3 || elem.nodeType === 8 || !elem.style ) { + return; + } + + // Make sure that we're working with the right name + var ret, type, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ), + style = elem.style; + + // Make sure that we're working with the right name. We don't + // want to query the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Gets hook for the prefixed version, then unprefixed version + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // Check if we're setting a value + if ( value !== undefined ) { + type = typeof value; + + // Convert "+=" or "-=" to relative numbers (#7345) + if ( type === "string" && ( ret = rcssNum.exec( value ) ) && ret[ 1 ] ) { + value = adjustCSS( elem, name, ret ); + + // Fixes bug #9237 + type = "number"; + } + + // Make sure that null and NaN values aren't set (#7116) + if ( value == null || value !== value ) { + return; + } + + // If a number was passed in, add the unit (except for certain CSS properties) + // The isCustomProp check can be removed in jQuery 4.0 when we only auto-append + // "px" to a few hardcoded values. + if ( type === "number" && !isCustomProp ) { + value += ret && ret[ 3 ] || ( jQuery.cssNumber[ origName ] ? "" : "px" ); + } + + // background-* props affect original clone's values + if ( !support.clearCloneStyle && value === "" && name.indexOf( "background" ) === 0 ) { + style[ name ] = "inherit"; + } + + // If a hook was provided, use that value, otherwise just set the specified value + if ( !hooks || !( "set" in hooks ) || + ( value = hooks.set( elem, value, extra ) ) !== undefined ) { + + if ( isCustomProp ) { + style.setProperty( name, value ); + } else { + style[ name ] = value; + } + } + + } else { + + // If a hook was provided get the non-computed value from there + if ( hooks && "get" in hooks && + ( ret = hooks.get( elem, false, extra ) ) !== undefined ) { + + return ret; + } + + // Otherwise just get the value from the style object + return style[ name ]; + } + }, + + css: function( elem, name, extra, styles ) { + var val, num, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ); + + // Make sure that we're working with the right name. We don't + // want to modify the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Try prefixed name followed by the unprefixed name + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // If a hook was provided get the computed value from there + if ( hooks && "get" in hooks ) { + val = hooks.get( elem, true, extra ); + } + + // Otherwise, if a way to get the computed value exists, use that + if ( val === undefined ) { + val = curCSS( elem, name, styles ); + } + + // Convert "normal" to computed value + if ( val === "normal" && name in cssNormalTransform ) { + val = cssNormalTransform[ name ]; + } + + // Make numeric if forced or a qualifier was provided and val looks numeric + if ( extra === "" || extra ) { + num = parseFloat( val ); + return extra === true || isFinite( num ) ? num || 0 : val; + } + + return val; + } +} ); + +jQuery.each( [ "height", "width" ], function( _i, dimension ) { + jQuery.cssHooks[ dimension ] = { + get: function( elem, computed, extra ) { + if ( computed ) { + + // Certain elements can have dimension info if we invisibly show them + // but it must have a current display style that would benefit + return rdisplayswap.test( jQuery.css( elem, "display" ) ) && + + // Support: Safari 8+ + // Table columns in Safari have non-zero offsetWidth & zero + // getBoundingClientRect().width unless display is changed. + // Support: IE <=11 only + // Running getBoundingClientRect on a disconnected node + // in IE throws an error. + ( !elem.getClientRects().length || !elem.getBoundingClientRect().width ) ? + swap( elem, cssShow, function() { + return getWidthOrHeight( elem, dimension, extra ); + } ) : + getWidthOrHeight( elem, dimension, extra ); + } + }, + + set: function( elem, value, extra ) { + var matches, + styles = getStyles( elem ), + + // Only read styles.position if the test has a chance to fail + // to avoid forcing a reflow. + scrollboxSizeBuggy = !support.scrollboxSize() && + styles.position === "absolute", + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-3991) + boxSizingNeeded = scrollboxSizeBuggy || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + subtract = extra ? + boxModelAdjustment( + elem, + dimension, + extra, + isBorderBox, + styles + ) : + 0; + + // Account for unreliable border-box dimensions by comparing offset* to computed and + // faking a content-box to get border and padding (gh-3699) + if ( isBorderBox && scrollboxSizeBuggy ) { + subtract -= Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + parseFloat( styles[ dimension ] ) - + boxModelAdjustment( elem, dimension, "border", false, styles ) - + 0.5 + ); + } + + // Convert to pixels if value adjustment is needed + if ( subtract && ( matches = rcssNum.exec( value ) ) && + ( matches[ 3 ] || "px" ) !== "px" ) { + + elem.style[ dimension ] = value; + value = jQuery.css( elem, dimension ); + } + + return setPositiveNumber( elem, value, subtract ); + } + }; +} ); + +jQuery.cssHooks.marginLeft = addGetHookIf( support.reliableMarginLeft, + function( elem, computed ) { + if ( computed ) { + return ( parseFloat( curCSS( elem, "marginLeft" ) ) || + elem.getBoundingClientRect().left - + swap( elem, { marginLeft: 0 }, function() { + return elem.getBoundingClientRect().left; + } ) + ) + "px"; + } + } +); + +// These hooks are used by animate to expand properties +jQuery.each( { + margin: "", + padding: "", + border: "Width" +}, function( prefix, suffix ) { + jQuery.cssHooks[ prefix + suffix ] = { + expand: function( value ) { + var i = 0, + expanded = {}, + + // Assumes a single number if not a string + parts = typeof value === "string" ? value.split( " " ) : [ value ]; + + for ( ; i < 4; i++ ) { + expanded[ prefix + cssExpand[ i ] + suffix ] = + parts[ i ] || parts[ i - 2 ] || parts[ 0 ]; + } + + return expanded; + } + }; + + if ( prefix !== "margin" ) { + jQuery.cssHooks[ prefix + suffix ].set = setPositiveNumber; + } +} ); + +jQuery.fn.extend( { + css: function( name, value ) { + return access( this, function( elem, name, value ) { + var styles, len, + map = {}, + i = 0; + + if ( Array.isArray( name ) ) { + styles = getStyles( elem ); + len = name.length; + + for ( ; i < len; i++ ) { + map[ name[ i ] ] = jQuery.css( elem, name[ i ], false, styles ); + } + + return map; + } + + return value !== undefined ? + jQuery.style( elem, name, value ) : + jQuery.css( elem, name ); + }, name, value, arguments.length > 1 ); + } +} ); + + +function Tween( elem, options, prop, end, easing ) { + return new Tween.prototype.init( elem, options, prop, end, easing ); +} +jQuery.Tween = Tween; + +Tween.prototype = { + constructor: Tween, + init: function( elem, options, prop, end, easing, unit ) { + this.elem = elem; + this.prop = prop; + this.easing = easing || jQuery.easing._default; + this.options = options; + this.start = this.now = this.cur(); + this.end = end; + this.unit = unit || ( jQuery.cssNumber[ prop ] ? "" : "px" ); + }, + cur: function() { + var hooks = Tween.propHooks[ this.prop ]; + + return hooks && hooks.get ? + hooks.get( this ) : + Tween.propHooks._default.get( this ); + }, + run: function( percent ) { + var eased, + hooks = Tween.propHooks[ this.prop ]; + + if ( this.options.duration ) { + this.pos = eased = jQuery.easing[ this.easing ]( + percent, this.options.duration * percent, 0, 1, this.options.duration + ); + } else { + this.pos = eased = percent; + } + this.now = ( this.end - this.start ) * eased + this.start; + + if ( this.options.step ) { + this.options.step.call( this.elem, this.now, this ); + } + + if ( hooks && hooks.set ) { + hooks.set( this ); + } else { + Tween.propHooks._default.set( this ); + } + return this; + } +}; + +Tween.prototype.init.prototype = Tween.prototype; + +Tween.propHooks = { + _default: { + get: function( tween ) { + var result; + + // Use a property on the element directly when it is not a DOM element, + // or when there is no matching style property that exists. + if ( tween.elem.nodeType !== 1 || + tween.elem[ tween.prop ] != null && tween.elem.style[ tween.prop ] == null ) { + return tween.elem[ tween.prop ]; + } + + // Passing an empty string as a 3rd parameter to .css will automatically + // attempt a parseFloat and fallback to a string if the parse fails. + // Simple values such as "10px" are parsed to Float; + // complex values such as "rotate(1rad)" are returned as-is. + result = jQuery.css( tween.elem, tween.prop, "" ); + + // Empty strings, null, undefined and "auto" are converted to 0. + return !result || result === "auto" ? 0 : result; + }, + set: function( tween ) { + + // Use step hook for back compat. + // Use cssHook if its there. + // Use .style if available and use plain properties where available. + if ( jQuery.fx.step[ tween.prop ] ) { + jQuery.fx.step[ tween.prop ]( tween ); + } else if ( tween.elem.nodeType === 1 && ( + jQuery.cssHooks[ tween.prop ] || + tween.elem.style[ finalPropName( tween.prop ) ] != null ) ) { + jQuery.style( tween.elem, tween.prop, tween.now + tween.unit ); + } else { + tween.elem[ tween.prop ] = tween.now; + } + } + } +}; + +// Support: IE <=9 only +// Panic based approach to setting things on disconnected nodes +Tween.propHooks.scrollTop = Tween.propHooks.scrollLeft = { + set: function( tween ) { + if ( tween.elem.nodeType && tween.elem.parentNode ) { + tween.elem[ tween.prop ] = tween.now; + } + } +}; + +jQuery.easing = { + linear: function( p ) { + return p; + }, + swing: function( p ) { + return 0.5 - Math.cos( p * Math.PI ) / 2; + }, + _default: "swing" +}; + +jQuery.fx = Tween.prototype.init; + +// Back compat <1.8 extension point +jQuery.fx.step = {}; + + + + +var + fxNow, inProgress, + rfxtypes = /^(?:toggle|show|hide)$/, + rrun = /queueHooks$/; + +function schedule() { + if ( inProgress ) { + if ( document.hidden === false && window.requestAnimationFrame ) { + window.requestAnimationFrame( schedule ); + } else { + window.setTimeout( schedule, jQuery.fx.interval ); + } + + jQuery.fx.tick(); + } +} + +// Animations created synchronously will run synchronously +function createFxNow() { + window.setTimeout( function() { + fxNow = undefined; + } ); + return ( fxNow = Date.now() ); +} + +// Generate parameters to create a standard animation +function genFx( type, includeWidth ) { + var which, + i = 0, + attrs = { height: type }; + + // If we include width, step value is 1 to do all cssExpand values, + // otherwise step value is 2 to skip over Left and Right + includeWidth = includeWidth ? 1 : 0; + for ( ; i < 4; i += 2 - includeWidth ) { + which = cssExpand[ i ]; + attrs[ "margin" + which ] = attrs[ "padding" + which ] = type; + } + + if ( includeWidth ) { + attrs.opacity = attrs.width = type; + } + + return attrs; +} + +function createTween( value, prop, animation ) { + var tween, + collection = ( Animation.tweeners[ prop ] || [] ).concat( Animation.tweeners[ "*" ] ), + index = 0, + length = collection.length; + for ( ; index < length; index++ ) { + if ( ( tween = collection[ index ].call( animation, prop, value ) ) ) { + + // We're done with this property + return tween; + } + } +} + +function defaultPrefilter( elem, props, opts ) { + var prop, value, toggle, hooks, oldfire, propTween, restoreDisplay, display, + isBox = "width" in props || "height" in props, + anim = this, + orig = {}, + style = elem.style, + hidden = elem.nodeType && isHiddenWithinTree( elem ), + dataShow = dataPriv.get( elem, "fxshow" ); + + // Queue-skipping animations hijack the fx hooks + if ( !opts.queue ) { + hooks = jQuery._queueHooks( elem, "fx" ); + if ( hooks.unqueued == null ) { + hooks.unqueued = 0; + oldfire = hooks.empty.fire; + hooks.empty.fire = function() { + if ( !hooks.unqueued ) { + oldfire(); + } + }; + } + hooks.unqueued++; + + anim.always( function() { + + // Ensure the complete handler is called before this completes + anim.always( function() { + hooks.unqueued--; + if ( !jQuery.queue( elem, "fx" ).length ) { + hooks.empty.fire(); + } + } ); + } ); + } + + // Detect show/hide animations + for ( prop in props ) { + value = props[ prop ]; + if ( rfxtypes.test( value ) ) { + delete props[ prop ]; + toggle = toggle || value === "toggle"; + if ( value === ( hidden ? "hide" : "show" ) ) { + + // Pretend to be hidden if this is a "show" and + // there is still data from a stopped show/hide + if ( value === "show" && dataShow && dataShow[ prop ] !== undefined ) { + hidden = true; + + // Ignore all other no-op show/hide data + } else { + continue; + } + } + orig[ prop ] = dataShow && dataShow[ prop ] || jQuery.style( elem, prop ); + } + } + + // Bail out if this is a no-op like .hide().hide() + propTween = !jQuery.isEmptyObject( props ); + if ( !propTween && jQuery.isEmptyObject( orig ) ) { + return; + } + + // Restrict "overflow" and "display" styles during box animations + if ( isBox && elem.nodeType === 1 ) { + + // Support: IE <=9 - 11, Edge 12 - 15 + // Record all 3 overflow attributes because IE does not infer the shorthand + // from identically-valued overflowX and overflowY and Edge just mirrors + // the overflowX value there. + opts.overflow = [ style.overflow, style.overflowX, style.overflowY ]; + + // Identify a display type, preferring old show/hide data over the CSS cascade + restoreDisplay = dataShow && dataShow.display; + if ( restoreDisplay == null ) { + restoreDisplay = dataPriv.get( elem, "display" ); + } + display = jQuery.css( elem, "display" ); + if ( display === "none" ) { + if ( restoreDisplay ) { + display = restoreDisplay; + } else { + + // Get nonempty value(s) by temporarily forcing visibility + showHide( [ elem ], true ); + restoreDisplay = elem.style.display || restoreDisplay; + display = jQuery.css( elem, "display" ); + showHide( [ elem ] ); + } + } + + // Animate inline elements as inline-block + if ( display === "inline" || display === "inline-block" && restoreDisplay != null ) { + if ( jQuery.css( elem, "float" ) === "none" ) { + + // Restore the original display value at the end of pure show/hide animations + if ( !propTween ) { + anim.done( function() { + style.display = restoreDisplay; + } ); + if ( restoreDisplay == null ) { + display = style.display; + restoreDisplay = display === "none" ? "" : display; + } + } + style.display = "inline-block"; + } + } + } + + if ( opts.overflow ) { + style.overflow = "hidden"; + anim.always( function() { + style.overflow = opts.overflow[ 0 ]; + style.overflowX = opts.overflow[ 1 ]; + style.overflowY = opts.overflow[ 2 ]; + } ); + } + + // Implement show/hide animations + propTween = false; + for ( prop in orig ) { + + // General show/hide setup for this element animation + if ( !propTween ) { + if ( dataShow ) { + if ( "hidden" in dataShow ) { + hidden = dataShow.hidden; + } + } else { + dataShow = dataPriv.access( elem, "fxshow", { display: restoreDisplay } ); + } + + // Store hidden/visible for toggle so `.stop().toggle()` "reverses" + if ( toggle ) { + dataShow.hidden = !hidden; + } + + // Show elements before animating them + if ( hidden ) { + showHide( [ elem ], true ); + } + + /* eslint-disable no-loop-func */ + + anim.done( function() { + + /* eslint-enable no-loop-func */ + + // The final step of a "hide" animation is actually hiding the element + if ( !hidden ) { + showHide( [ elem ] ); + } + dataPriv.remove( elem, "fxshow" ); + for ( prop in orig ) { + jQuery.style( elem, prop, orig[ prop ] ); + } + } ); + } + + // Per-property setup + propTween = createTween( hidden ? dataShow[ prop ] : 0, prop, anim ); + if ( !( prop in dataShow ) ) { + dataShow[ prop ] = propTween.start; + if ( hidden ) { + propTween.end = propTween.start; + propTween.start = 0; + } + } + } +} + +function propFilter( props, specialEasing ) { + var index, name, easing, value, hooks; + + // camelCase, specialEasing and expand cssHook pass + for ( index in props ) { + name = camelCase( index ); + easing = specialEasing[ name ]; + value = props[ index ]; + if ( Array.isArray( value ) ) { + easing = value[ 1 ]; + value = props[ index ] = value[ 0 ]; + } + + if ( index !== name ) { + props[ name ] = value; + delete props[ index ]; + } + + hooks = jQuery.cssHooks[ name ]; + if ( hooks && "expand" in hooks ) { + value = hooks.expand( value ); + delete props[ name ]; + + // Not quite $.extend, this won't overwrite existing keys. + // Reusing 'index' because we have the correct "name" + for ( index in value ) { + if ( !( index in props ) ) { + props[ index ] = value[ index ]; + specialEasing[ index ] = easing; + } + } + } else { + specialEasing[ name ] = easing; + } + } +} + +function Animation( elem, properties, options ) { + var result, + stopped, + index = 0, + length = Animation.prefilters.length, + deferred = jQuery.Deferred().always( function() { + + // Don't match elem in the :animated selector + delete tick.elem; + } ), + tick = function() { + if ( stopped ) { + return false; + } + var currentTime = fxNow || createFxNow(), + remaining = Math.max( 0, animation.startTime + animation.duration - currentTime ), + + // Support: Android 2.3 only + // Archaic crash bug won't allow us to use `1 - ( 0.5 || 0 )` (#12497) + temp = remaining / animation.duration || 0, + percent = 1 - temp, + index = 0, + length = animation.tweens.length; + + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( percent ); + } + + deferred.notifyWith( elem, [ animation, percent, remaining ] ); + + // If there's more to do, yield + if ( percent < 1 && length ) { + return remaining; + } + + // If this was an empty animation, synthesize a final progress notification + if ( !length ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + } + + // Resolve the animation and report its conclusion + deferred.resolveWith( elem, [ animation ] ); + return false; + }, + animation = deferred.promise( { + elem: elem, + props: jQuery.extend( {}, properties ), + opts: jQuery.extend( true, { + specialEasing: {}, + easing: jQuery.easing._default + }, options ), + originalProperties: properties, + originalOptions: options, + startTime: fxNow || createFxNow(), + duration: options.duration, + tweens: [], + createTween: function( prop, end ) { + var tween = jQuery.Tween( elem, animation.opts, prop, end, + animation.opts.specialEasing[ prop ] || animation.opts.easing ); + animation.tweens.push( tween ); + return tween; + }, + stop: function( gotoEnd ) { + var index = 0, + + // If we are going to the end, we want to run all the tweens + // otherwise we skip this part + length = gotoEnd ? animation.tweens.length : 0; + if ( stopped ) { + return this; + } + stopped = true; + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( 1 ); + } + + // Resolve when we played the last frame; otherwise, reject + if ( gotoEnd ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + deferred.resolveWith( elem, [ animation, gotoEnd ] ); + } else { + deferred.rejectWith( elem, [ animation, gotoEnd ] ); + } + return this; + } + } ), + props = animation.props; + + propFilter( props, animation.opts.specialEasing ); + + for ( ; index < length; index++ ) { + result = Animation.prefilters[ index ].call( animation, elem, props, animation.opts ); + if ( result ) { + if ( isFunction( result.stop ) ) { + jQuery._queueHooks( animation.elem, animation.opts.queue ).stop = + result.stop.bind( result ); + } + return result; + } + } + + jQuery.map( props, createTween, animation ); + + if ( isFunction( animation.opts.start ) ) { + animation.opts.start.call( elem, animation ); + } + + // Attach callbacks from options + animation + .progress( animation.opts.progress ) + .done( animation.opts.done, animation.opts.complete ) + .fail( animation.opts.fail ) + .always( animation.opts.always ); + + jQuery.fx.timer( + jQuery.extend( tick, { + elem: elem, + anim: animation, + queue: animation.opts.queue + } ) + ); + + return animation; +} + +jQuery.Animation = jQuery.extend( Animation, { + + tweeners: { + "*": [ function( prop, value ) { + var tween = this.createTween( prop, value ); + adjustCSS( tween.elem, prop, rcssNum.exec( value ), tween ); + return tween; + } ] + }, + + tweener: function( props, callback ) { + if ( isFunction( props ) ) { + callback = props; + props = [ "*" ]; + } else { + props = props.match( rnothtmlwhite ); + } + + var prop, + index = 0, + length = props.length; + + for ( ; index < length; index++ ) { + prop = props[ index ]; + Animation.tweeners[ prop ] = Animation.tweeners[ prop ] || []; + Animation.tweeners[ prop ].unshift( callback ); + } + }, + + prefilters: [ defaultPrefilter ], + + prefilter: function( callback, prepend ) { + if ( prepend ) { + Animation.prefilters.unshift( callback ); + } else { + Animation.prefilters.push( callback ); + } + } +} ); + +jQuery.speed = function( speed, easing, fn ) { + var opt = speed && typeof speed === "object" ? jQuery.extend( {}, speed ) : { + complete: fn || !fn && easing || + isFunction( speed ) && speed, + duration: speed, + easing: fn && easing || easing && !isFunction( easing ) && easing + }; + + // Go to the end state if fx are off + if ( jQuery.fx.off ) { + opt.duration = 0; + + } else { + if ( typeof opt.duration !== "number" ) { + if ( opt.duration in jQuery.fx.speeds ) { + opt.duration = jQuery.fx.speeds[ opt.duration ]; + + } else { + opt.duration = jQuery.fx.speeds._default; + } + } + } + + // Normalize opt.queue - true/undefined/null -> "fx" + if ( opt.queue == null || opt.queue === true ) { + opt.queue = "fx"; + } + + // Queueing + opt.old = opt.complete; + + opt.complete = function() { + if ( isFunction( opt.old ) ) { + opt.old.call( this ); + } + + if ( opt.queue ) { + jQuery.dequeue( this, opt.queue ); + } + }; + + return opt; +}; + +jQuery.fn.extend( { + fadeTo: function( speed, to, easing, callback ) { + + // Show any hidden elements after setting opacity to 0 + return this.filter( isHiddenWithinTree ).css( "opacity", 0 ).show() + + // Animate to the value specified + .end().animate( { opacity: to }, speed, easing, callback ); + }, + animate: function( prop, speed, easing, callback ) { + var empty = jQuery.isEmptyObject( prop ), + optall = jQuery.speed( speed, easing, callback ), + doAnimation = function() { + + // Operate on a copy of prop so per-property easing won't be lost + var anim = Animation( this, jQuery.extend( {}, prop ), optall ); + + // Empty animations, or finishing resolves immediately + if ( empty || dataPriv.get( this, "finish" ) ) { + anim.stop( true ); + } + }; + + doAnimation.finish = doAnimation; + + return empty || optall.queue === false ? + this.each( doAnimation ) : + this.queue( optall.queue, doAnimation ); + }, + stop: function( type, clearQueue, gotoEnd ) { + var stopQueue = function( hooks ) { + var stop = hooks.stop; + delete hooks.stop; + stop( gotoEnd ); + }; + + if ( typeof type !== "string" ) { + gotoEnd = clearQueue; + clearQueue = type; + type = undefined; + } + if ( clearQueue ) { + this.queue( type || "fx", [] ); + } + + return this.each( function() { + var dequeue = true, + index = type != null && type + "queueHooks", + timers = jQuery.timers, + data = dataPriv.get( this ); + + if ( index ) { + if ( data[ index ] && data[ index ].stop ) { + stopQueue( data[ index ] ); + } + } else { + for ( index in data ) { + if ( data[ index ] && data[ index ].stop && rrun.test( index ) ) { + stopQueue( data[ index ] ); + } + } + } + + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && + ( type == null || timers[ index ].queue === type ) ) { + + timers[ index ].anim.stop( gotoEnd ); + dequeue = false; + timers.splice( index, 1 ); + } + } + + // Start the next in the queue if the last step wasn't forced. + // Timers currently will call their complete callbacks, which + // will dequeue but only if they were gotoEnd. + if ( dequeue || !gotoEnd ) { + jQuery.dequeue( this, type ); + } + } ); + }, + finish: function( type ) { + if ( type !== false ) { + type = type || "fx"; + } + return this.each( function() { + var index, + data = dataPriv.get( this ), + queue = data[ type + "queue" ], + hooks = data[ type + "queueHooks" ], + timers = jQuery.timers, + length = queue ? queue.length : 0; + + // Enable finishing flag on private data + data.finish = true; + + // Empty the queue first + jQuery.queue( this, type, [] ); + + if ( hooks && hooks.stop ) { + hooks.stop.call( this, true ); + } + + // Look for any active animations, and finish them + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && timers[ index ].queue === type ) { + timers[ index ].anim.stop( true ); + timers.splice( index, 1 ); + } + } + + // Look for any animations in the old queue and finish them + for ( index = 0; index < length; index++ ) { + if ( queue[ index ] && queue[ index ].finish ) { + queue[ index ].finish.call( this ); + } + } + + // Turn off finishing flag + delete data.finish; + } ); + } +} ); + +jQuery.each( [ "toggle", "show", "hide" ], function( _i, name ) { + var cssFn = jQuery.fn[ name ]; + jQuery.fn[ name ] = function( speed, easing, callback ) { + return speed == null || typeof speed === "boolean" ? + cssFn.apply( this, arguments ) : + this.animate( genFx( name, true ), speed, easing, callback ); + }; +} ); + +// Generate shortcuts for custom animations +jQuery.each( { + slideDown: genFx( "show" ), + slideUp: genFx( "hide" ), + slideToggle: genFx( "toggle" ), + fadeIn: { opacity: "show" }, + fadeOut: { opacity: "hide" }, + fadeToggle: { opacity: "toggle" } +}, function( name, props ) { + jQuery.fn[ name ] = function( speed, easing, callback ) { + return this.animate( props, speed, easing, callback ); + }; +} ); + +jQuery.timers = []; +jQuery.fx.tick = function() { + var timer, + i = 0, + timers = jQuery.timers; + + fxNow = Date.now(); + + for ( ; i < timers.length; i++ ) { + timer = timers[ i ]; + + // Run the timer and safely remove it when done (allowing for external removal) + if ( !timer() && timers[ i ] === timer ) { + timers.splice( i--, 1 ); + } + } + + if ( !timers.length ) { + jQuery.fx.stop(); + } + fxNow = undefined; +}; + +jQuery.fx.timer = function( timer ) { + jQuery.timers.push( timer ); + jQuery.fx.start(); +}; + +jQuery.fx.interval = 13; +jQuery.fx.start = function() { + if ( inProgress ) { + return; + } + + inProgress = true; + schedule(); +}; + +jQuery.fx.stop = function() { + inProgress = null; +}; + +jQuery.fx.speeds = { + slow: 600, + fast: 200, + + // Default speed + _default: 400 +}; + + +// Based off of the plugin by Clint Helfers, with permission. +// https://web.archive.org/web/20100324014747/http://blindsignals.com/index.php/2009/07/jquery-delay/ +jQuery.fn.delay = function( time, type ) { + time = jQuery.fx ? jQuery.fx.speeds[ time ] || time : time; + type = type || "fx"; + + return this.queue( type, function( next, hooks ) { + var timeout = window.setTimeout( next, time ); + hooks.stop = function() { + window.clearTimeout( timeout ); + }; + } ); +}; + + +( function() { + var input = document.createElement( "input" ), + select = document.createElement( "select" ), + opt = select.appendChild( document.createElement( "option" ) ); + + input.type = "checkbox"; + + // Support: Android <=4.3 only + // Default value for a checkbox should be "on" + support.checkOn = input.value !== ""; + + // Support: IE <=11 only + // Must access selectedIndex to make default options select + support.optSelected = opt.selected; + + // Support: IE <=11 only + // An input loses its value after becoming a radio + input = document.createElement( "input" ); + input.value = "t"; + input.type = "radio"; + support.radioValue = input.value === "t"; +} )(); + + +var boolHook, + attrHandle = jQuery.expr.attrHandle; + +jQuery.fn.extend( { + attr: function( name, value ) { + return access( this, jQuery.attr, name, value, arguments.length > 1 ); + }, + + removeAttr: function( name ) { + return this.each( function() { + jQuery.removeAttr( this, name ); + } ); + } +} ); + +jQuery.extend( { + attr: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set attributes on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + // Fallback to prop when attributes are not supported + if ( typeof elem.getAttribute === "undefined" ) { + return jQuery.prop( elem, name, value ); + } + + // Attribute hooks are determined by the lowercase version + // Grab necessary hook if one is defined + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + hooks = jQuery.attrHooks[ name.toLowerCase() ] || + ( jQuery.expr.match.bool.test( name ) ? boolHook : undefined ); + } + + if ( value !== undefined ) { + if ( value === null ) { + jQuery.removeAttr( elem, name ); + return; + } + + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + elem.setAttribute( name, value + "" ); + return value; + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + ret = jQuery.find.attr( elem, name ); + + // Non-existent attributes return null, we normalize to undefined + return ret == null ? undefined : ret; + }, + + attrHooks: { + type: { + set: function( elem, value ) { + if ( !support.radioValue && value === "radio" && + nodeName( elem, "input" ) ) { + var val = elem.value; + elem.setAttribute( "type", value ); + if ( val ) { + elem.value = val; + } + return value; + } + } + } + }, + + removeAttr: function( elem, value ) { + var name, + i = 0, + + // Attribute names can contain non-HTML whitespace characters + // https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 + attrNames = value && value.match( rnothtmlwhite ); + + if ( attrNames && elem.nodeType === 1 ) { + while ( ( name = attrNames[ i++ ] ) ) { + elem.removeAttribute( name ); + } + } + } +} ); + +// Hooks for boolean attributes +boolHook = { + set: function( elem, value, name ) { + if ( value === false ) { + + // Remove boolean attributes when set to false + jQuery.removeAttr( elem, name ); + } else { + elem.setAttribute( name, name ); + } + return name; + } +}; + +jQuery.each( jQuery.expr.match.bool.source.match( /\w+/g ), function( _i, name ) { + var getter = attrHandle[ name ] || jQuery.find.attr; + + attrHandle[ name ] = function( elem, name, isXML ) { + var ret, handle, + lowercaseName = name.toLowerCase(); + + if ( !isXML ) { + + // Avoid an infinite loop by temporarily removing this function from the getter + handle = attrHandle[ lowercaseName ]; + attrHandle[ lowercaseName ] = ret; + ret = getter( elem, name, isXML ) != null ? + lowercaseName : + null; + attrHandle[ lowercaseName ] = handle; + } + return ret; + }; +} ); + + + + +var rfocusable = /^(?:input|select|textarea|button)$/i, + rclickable = /^(?:a|area)$/i; + +jQuery.fn.extend( { + prop: function( name, value ) { + return access( this, jQuery.prop, name, value, arguments.length > 1 ); + }, + + removeProp: function( name ) { + return this.each( function() { + delete this[ jQuery.propFix[ name ] || name ]; + } ); + } +} ); + +jQuery.extend( { + prop: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set properties on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + + // Fix name and attach hooks + name = jQuery.propFix[ name ] || name; + hooks = jQuery.propHooks[ name ]; + } + + if ( value !== undefined ) { + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + return ( elem[ name ] = value ); + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + return elem[ name ]; + }, + + propHooks: { + tabIndex: { + get: function( elem ) { + + // Support: IE <=9 - 11 only + // elem.tabIndex doesn't always return the + // correct value when it hasn't been explicitly set + // https://web.archive.org/web/20141116233347/http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ + // Use proper attribute retrieval(#12072) + var tabindex = jQuery.find.attr( elem, "tabindex" ); + + if ( tabindex ) { + return parseInt( tabindex, 10 ); + } + + if ( + rfocusable.test( elem.nodeName ) || + rclickable.test( elem.nodeName ) && + elem.href + ) { + return 0; + } + + return -1; + } + } + }, + + propFix: { + "for": "htmlFor", + "class": "className" + } +} ); + +// Support: IE <=11 only +// Accessing the selectedIndex property +// forces the browser to respect setting selected +// on the option +// The getter ensures a default option is selected +// when in an optgroup +// eslint rule "no-unused-expressions" is disabled for this code +// since it considers such accessions noop +if ( !support.optSelected ) { + jQuery.propHooks.selected = { + get: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent && parent.parentNode ) { + parent.parentNode.selectedIndex; + } + return null; + }, + set: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent ) { + parent.selectedIndex; + + if ( parent.parentNode ) { + parent.parentNode.selectedIndex; + } + } + } + }; +} + +jQuery.each( [ + "tabIndex", + "readOnly", + "maxLength", + "cellSpacing", + "cellPadding", + "rowSpan", + "colSpan", + "useMap", + "frameBorder", + "contentEditable" +], function() { + jQuery.propFix[ this.toLowerCase() ] = this; +} ); + + + + + // Strip and collapse whitespace according to HTML spec + // https://infra.spec.whatwg.org/#strip-and-collapse-ascii-whitespace + function stripAndCollapse( value ) { + var tokens = value.match( rnothtmlwhite ) || []; + return tokens.join( " " ); + } + + +function getClass( elem ) { + return elem.getAttribute && elem.getAttribute( "class" ) || ""; +} + +function classesToArray( value ) { + if ( Array.isArray( value ) ) { + return value; + } + if ( typeof value === "string" ) { + return value.match( rnothtmlwhite ) || []; + } + return []; +} + +jQuery.fn.extend( { + addClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).addClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + if ( cur.indexOf( " " + clazz + " " ) < 0 ) { + cur += clazz + " "; + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + removeClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).removeClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + if ( !arguments.length ) { + return this.attr( "class", "" ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + + // This expression is here for better compressibility (see addClass) + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + + // Remove *all* instances + while ( cur.indexOf( " " + clazz + " " ) > -1 ) { + cur = cur.replace( " " + clazz + " ", " " ); + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + toggleClass: function( value, stateVal ) { + var type = typeof value, + isValidValue = type === "string" || Array.isArray( value ); + + if ( typeof stateVal === "boolean" && isValidValue ) { + return stateVal ? this.addClass( value ) : this.removeClass( value ); + } + + if ( isFunction( value ) ) { + return this.each( function( i ) { + jQuery( this ).toggleClass( + value.call( this, i, getClass( this ), stateVal ), + stateVal + ); + } ); + } + + return this.each( function() { + var className, i, self, classNames; + + if ( isValidValue ) { + + // Toggle individual class names + i = 0; + self = jQuery( this ); + classNames = classesToArray( value ); + + while ( ( className = classNames[ i++ ] ) ) { + + // Check each className given, space separated list + if ( self.hasClass( className ) ) { + self.removeClass( className ); + } else { + self.addClass( className ); + } + } + + // Toggle whole class name + } else if ( value === undefined || type === "boolean" ) { + className = getClass( this ); + if ( className ) { + + // Store className if set + dataPriv.set( this, "__className__", className ); + } + + // If the element has a class name or if we're passed `false`, + // then remove the whole classname (if there was one, the above saved it). + // Otherwise bring back whatever was previously saved (if anything), + // falling back to the empty string if nothing was stored. + if ( this.setAttribute ) { + this.setAttribute( "class", + className || value === false ? + "" : + dataPriv.get( this, "__className__" ) || "" + ); + } + } + } ); + }, + + hasClass: function( selector ) { + var className, elem, + i = 0; + + className = " " + selector + " "; + while ( ( elem = this[ i++ ] ) ) { + if ( elem.nodeType === 1 && + ( " " + stripAndCollapse( getClass( elem ) ) + " " ).indexOf( className ) > -1 ) { + return true; + } + } + + return false; + } +} ); + + + + +var rreturn = /\r/g; + +jQuery.fn.extend( { + val: function( value ) { + var hooks, ret, valueIsFunction, + elem = this[ 0 ]; + + if ( !arguments.length ) { + if ( elem ) { + hooks = jQuery.valHooks[ elem.type ] || + jQuery.valHooks[ elem.nodeName.toLowerCase() ]; + + if ( hooks && + "get" in hooks && + ( ret = hooks.get( elem, "value" ) ) !== undefined + ) { + return ret; + } + + ret = elem.value; + + // Handle most common string cases + if ( typeof ret === "string" ) { + return ret.replace( rreturn, "" ); + } + + // Handle cases where value is null/undef or number + return ret == null ? "" : ret; + } + + return; + } + + valueIsFunction = isFunction( value ); + + return this.each( function( i ) { + var val; + + if ( this.nodeType !== 1 ) { + return; + } + + if ( valueIsFunction ) { + val = value.call( this, i, jQuery( this ).val() ); + } else { + val = value; + } + + // Treat null/undefined as ""; convert numbers to string + if ( val == null ) { + val = ""; + + } else if ( typeof val === "number" ) { + val += ""; + + } else if ( Array.isArray( val ) ) { + val = jQuery.map( val, function( value ) { + return value == null ? "" : value + ""; + } ); + } + + hooks = jQuery.valHooks[ this.type ] || jQuery.valHooks[ this.nodeName.toLowerCase() ]; + + // If set returns undefined, fall back to normal setting + if ( !hooks || !( "set" in hooks ) || hooks.set( this, val, "value" ) === undefined ) { + this.value = val; + } + } ); + } +} ); + +jQuery.extend( { + valHooks: { + option: { + get: function( elem ) { + + var val = jQuery.find.attr( elem, "value" ); + return val != null ? + val : + + // Support: IE <=10 - 11 only + // option.text throws exceptions (#14686, #14858) + // Strip and collapse whitespace + // https://html.spec.whatwg.org/#strip-and-collapse-whitespace + stripAndCollapse( jQuery.text( elem ) ); + } + }, + select: { + get: function( elem ) { + var value, option, i, + options = elem.options, + index = elem.selectedIndex, + one = elem.type === "select-one", + values = one ? null : [], + max = one ? index + 1 : options.length; + + if ( index < 0 ) { + i = max; + + } else { + i = one ? index : 0; + } + + // Loop through all the selected options + for ( ; i < max; i++ ) { + option = options[ i ]; + + // Support: IE <=9 only + // IE8-9 doesn't update selected after form reset (#2551) + if ( ( option.selected || i === index ) && + + // Don't return options that are disabled or in a disabled optgroup + !option.disabled && + ( !option.parentNode.disabled || + !nodeName( option.parentNode, "optgroup" ) ) ) { + + // Get the specific value for the option + value = jQuery( option ).val(); + + // We don't need an array for one selects + if ( one ) { + return value; + } + + // Multi-Selects return an array + values.push( value ); + } + } + + return values; + }, + + set: function( elem, value ) { + var optionSet, option, + options = elem.options, + values = jQuery.makeArray( value ), + i = options.length; + + while ( i-- ) { + option = options[ i ]; + + /* eslint-disable no-cond-assign */ + + if ( option.selected = + jQuery.inArray( jQuery.valHooks.option.get( option ), values ) > -1 + ) { + optionSet = true; + } + + /* eslint-enable no-cond-assign */ + } + + // Force browsers to behave consistently when non-matching value is set + if ( !optionSet ) { + elem.selectedIndex = -1; + } + return values; + } + } + } +} ); + +// Radios and checkboxes getter/setter +jQuery.each( [ "radio", "checkbox" ], function() { + jQuery.valHooks[ this ] = { + set: function( elem, value ) { + if ( Array.isArray( value ) ) { + return ( elem.checked = jQuery.inArray( jQuery( elem ).val(), value ) > -1 ); + } + } + }; + if ( !support.checkOn ) { + jQuery.valHooks[ this ].get = function( elem ) { + return elem.getAttribute( "value" ) === null ? "on" : elem.value; + }; + } +} ); + + + + +// Return jQuery for attributes-only inclusion + + +support.focusin = "onfocusin" in window; + + +var rfocusMorph = /^(?:focusinfocus|focusoutblur)$/, + stopPropagationCallback = function( e ) { + e.stopPropagation(); + }; + +jQuery.extend( jQuery.event, { + + trigger: function( event, data, elem, onlyHandlers ) { + + var i, cur, tmp, bubbleType, ontype, handle, special, lastElement, + eventPath = [ elem || document ], + type = hasOwn.call( event, "type" ) ? event.type : event, + namespaces = hasOwn.call( event, "namespace" ) ? event.namespace.split( "." ) : []; + + cur = lastElement = tmp = elem = elem || document; + + // Don't do events on text and comment nodes + if ( elem.nodeType === 3 || elem.nodeType === 8 ) { + return; + } + + // focus/blur morphs to focusin/out; ensure we're not firing them right now + if ( rfocusMorph.test( type + jQuery.event.triggered ) ) { + return; + } + + if ( type.indexOf( "." ) > -1 ) { + + // Namespaced trigger; create a regexp to match event type in handle() + namespaces = type.split( "." ); + type = namespaces.shift(); + namespaces.sort(); + } + ontype = type.indexOf( ":" ) < 0 && "on" + type; + + // Caller can pass in a jQuery.Event object, Object, or just an event type string + event = event[ jQuery.expando ] ? + event : + new jQuery.Event( type, typeof event === "object" && event ); + + // Trigger bitmask: & 1 for native handlers; & 2 for jQuery (always true) + event.isTrigger = onlyHandlers ? 2 : 3; + event.namespace = namespaces.join( "." ); + event.rnamespace = event.namespace ? + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ) : + null; + + // Clean up the event in case it is being reused + event.result = undefined; + if ( !event.target ) { + event.target = elem; + } + + // Clone any incoming data and prepend the event, creating the handler arg list + data = data == null ? + [ event ] : + jQuery.makeArray( data, [ event ] ); + + // Allow special events to draw outside the lines + special = jQuery.event.special[ type ] || {}; + if ( !onlyHandlers && special.trigger && special.trigger.apply( elem, data ) === false ) { + return; + } + + // Determine event propagation path in advance, per W3C events spec (#9951) + // Bubble up to document, then to window; watch for a global ownerDocument var (#9724) + if ( !onlyHandlers && !special.noBubble && !isWindow( elem ) ) { + + bubbleType = special.delegateType || type; + if ( !rfocusMorph.test( bubbleType + type ) ) { + cur = cur.parentNode; + } + for ( ; cur; cur = cur.parentNode ) { + eventPath.push( cur ); + tmp = cur; + } + + // Only add window if we got to document (e.g., not plain obj or detached DOM) + if ( tmp === ( elem.ownerDocument || document ) ) { + eventPath.push( tmp.defaultView || tmp.parentWindow || window ); + } + } + + // Fire handlers on the event path + i = 0; + while ( ( cur = eventPath[ i++ ] ) && !event.isPropagationStopped() ) { + lastElement = cur; + event.type = i > 1 ? + bubbleType : + special.bindType || type; + + // jQuery handler + handle = ( dataPriv.get( cur, "events" ) || Object.create( null ) )[ event.type ] && + dataPriv.get( cur, "handle" ); + if ( handle ) { + handle.apply( cur, data ); + } + + // Native handler + handle = ontype && cur[ ontype ]; + if ( handle && handle.apply && acceptData( cur ) ) { + event.result = handle.apply( cur, data ); + if ( event.result === false ) { + event.preventDefault(); + } + } + } + event.type = type; + + // If nobody prevented the default action, do it now + if ( !onlyHandlers && !event.isDefaultPrevented() ) { + + if ( ( !special._default || + special._default.apply( eventPath.pop(), data ) === false ) && + acceptData( elem ) ) { + + // Call a native DOM method on the target with the same name as the event. + // Don't do default actions on window, that's where global variables be (#6170) + if ( ontype && isFunction( elem[ type ] ) && !isWindow( elem ) ) { + + // Don't re-trigger an onFOO event when we call its FOO() method + tmp = elem[ ontype ]; + + if ( tmp ) { + elem[ ontype ] = null; + } + + // Prevent re-triggering of the same event, since we already bubbled it above + jQuery.event.triggered = type; + + if ( event.isPropagationStopped() ) { + lastElement.addEventListener( type, stopPropagationCallback ); + } + + elem[ type ](); + + if ( event.isPropagationStopped() ) { + lastElement.removeEventListener( type, stopPropagationCallback ); + } + + jQuery.event.triggered = undefined; + + if ( tmp ) { + elem[ ontype ] = tmp; + } + } + } + } + + return event.result; + }, + + // Piggyback on a donor event to simulate a different one + // Used only for `focus(in | out)` events + simulate: function( type, elem, event ) { + var e = jQuery.extend( + new jQuery.Event(), + event, + { + type: type, + isSimulated: true + } + ); + + jQuery.event.trigger( e, null, elem ); + } + +} ); + +jQuery.fn.extend( { + + trigger: function( type, data ) { + return this.each( function() { + jQuery.event.trigger( type, data, this ); + } ); + }, + triggerHandler: function( type, data ) { + var elem = this[ 0 ]; + if ( elem ) { + return jQuery.event.trigger( type, data, elem, true ); + } + } +} ); + + +// Support: Firefox <=44 +// Firefox doesn't have focus(in | out) events +// Related ticket - https://bugzilla.mozilla.org/show_bug.cgi?id=687787 +// +// Support: Chrome <=48 - 49, Safari <=9.0 - 9.1 +// focus(in | out) events fire after focus & blur events, +// which is spec violation - http://www.w3.org/TR/DOM-Level-3-Events/#events-focusevent-event-order +// Related ticket - https://bugs.chromium.org/p/chromium/issues/detail?id=449857 +if ( !support.focusin ) { + jQuery.each( { focus: "focusin", blur: "focusout" }, function( orig, fix ) { + + // Attach a single capturing handler on the document while someone wants focusin/focusout + var handler = function( event ) { + jQuery.event.simulate( fix, event.target, jQuery.event.fix( event ) ); + }; + + jQuery.event.special[ fix ] = { + setup: function() { + + // Handle: regular nodes (via `this.ownerDocument`), window + // (via `this.document`) & document (via `this`). + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ); + + if ( !attaches ) { + doc.addEventListener( orig, handler, true ); + } + dataPriv.access( doc, fix, ( attaches || 0 ) + 1 ); + }, + teardown: function() { + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ) - 1; + + if ( !attaches ) { + doc.removeEventListener( orig, handler, true ); + dataPriv.remove( doc, fix ); + + } else { + dataPriv.access( doc, fix, attaches ); + } + } + }; + } ); +} +var location = window.location; + +var nonce = { guid: Date.now() }; + +var rquery = ( /\?/ ); + + + +// Cross-browser xml parsing +jQuery.parseXML = function( data ) { + var xml, parserErrorElem; + if ( !data || typeof data !== "string" ) { + return null; + } + + // Support: IE 9 - 11 only + // IE throws on parseFromString with invalid input. + try { + xml = ( new window.DOMParser() ).parseFromString( data, "text/xml" ); + } catch ( e ) {} + + parserErrorElem = xml && xml.getElementsByTagName( "parsererror" )[ 0 ]; + if ( !xml || parserErrorElem ) { + jQuery.error( "Invalid XML: " + ( + parserErrorElem ? + jQuery.map( parserErrorElem.childNodes, function( el ) { + return el.textContent; + } ).join( "\n" ) : + data + ) ); + } + return xml; +}; + + +var + rbracket = /\[\]$/, + rCRLF = /\r?\n/g, + rsubmitterTypes = /^(?:submit|button|image|reset|file)$/i, + rsubmittable = /^(?:input|select|textarea|keygen)/i; + +function buildParams( prefix, obj, traditional, add ) { + var name; + + if ( Array.isArray( obj ) ) { + + // Serialize array item. + jQuery.each( obj, function( i, v ) { + if ( traditional || rbracket.test( prefix ) ) { + + // Treat each array item as a scalar. + add( prefix, v ); + + } else { + + // Item is non-scalar (array or object), encode its numeric index. + buildParams( + prefix + "[" + ( typeof v === "object" && v != null ? i : "" ) + "]", + v, + traditional, + add + ); + } + } ); + + } else if ( !traditional && toType( obj ) === "object" ) { + + // Serialize object item. + for ( name in obj ) { + buildParams( prefix + "[" + name + "]", obj[ name ], traditional, add ); + } + + } else { + + // Serialize scalar item. + add( prefix, obj ); + } +} + +// Serialize an array of form elements or a set of +// key/values into a query string +jQuery.param = function( a, traditional ) { + var prefix, + s = [], + add = function( key, valueOrFunction ) { + + // If value is a function, invoke it and use its return value + var value = isFunction( valueOrFunction ) ? + valueOrFunction() : + valueOrFunction; + + s[ s.length ] = encodeURIComponent( key ) + "=" + + encodeURIComponent( value == null ? "" : value ); + }; + + if ( a == null ) { + return ""; + } + + // If an array was passed in, assume that it is an array of form elements. + if ( Array.isArray( a ) || ( a.jquery && !jQuery.isPlainObject( a ) ) ) { + + // Serialize the form elements + jQuery.each( a, function() { + add( this.name, this.value ); + } ); + + } else { + + // If traditional, encode the "old" way (the way 1.3.2 or older + // did it), otherwise encode params recursively. + for ( prefix in a ) { + buildParams( prefix, a[ prefix ], traditional, add ); + } + } + + // Return the resulting serialization + return s.join( "&" ); +}; + +jQuery.fn.extend( { + serialize: function() { + return jQuery.param( this.serializeArray() ); + }, + serializeArray: function() { + return this.map( function() { + + // Can add propHook for "elements" to filter or add form elements + var elements = jQuery.prop( this, "elements" ); + return elements ? jQuery.makeArray( elements ) : this; + } ).filter( function() { + var type = this.type; + + // Use .is( ":disabled" ) so that fieldset[disabled] works + return this.name && !jQuery( this ).is( ":disabled" ) && + rsubmittable.test( this.nodeName ) && !rsubmitterTypes.test( type ) && + ( this.checked || !rcheckableType.test( type ) ); + } ).map( function( _i, elem ) { + var val = jQuery( this ).val(); + + if ( val == null ) { + return null; + } + + if ( Array.isArray( val ) ) { + return jQuery.map( val, function( val ) { + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ); + } + + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ).get(); + } +} ); + + +var + r20 = /%20/g, + rhash = /#.*$/, + rantiCache = /([?&])_=[^&]*/, + rheaders = /^(.*?):[ \t]*([^\r\n]*)$/mg, + + // #7653, #8125, #8152: local protocol detection + rlocalProtocol = /^(?:about|app|app-storage|.+-extension|file|res|widget):$/, + rnoContent = /^(?:GET|HEAD)$/, + rprotocol = /^\/\//, + + /* Prefilters + * 1) They are useful to introduce custom dataTypes (see ajax/jsonp.js for an example) + * 2) These are called: + * - BEFORE asking for a transport + * - AFTER param serialization (s.data is a string if s.processData is true) + * 3) key is the dataType + * 4) the catchall symbol "*" can be used + * 5) execution will start with transport dataType and THEN continue down to "*" if needed + */ + prefilters = {}, + + /* Transports bindings + * 1) key is the dataType + * 2) the catchall symbol "*" can be used + * 3) selection will start with transport dataType and THEN go to "*" if needed + */ + transports = {}, + + // Avoid comment-prolog char sequence (#10098); must appease lint and evade compression + allTypes = "*/".concat( "*" ), + + // Anchor tag for parsing the document origin + originAnchor = document.createElement( "a" ); + +originAnchor.href = location.href; + +// Base "constructor" for jQuery.ajaxPrefilter and jQuery.ajaxTransport +function addToPrefiltersOrTransports( structure ) { + + // dataTypeExpression is optional and defaults to "*" + return function( dataTypeExpression, func ) { + + if ( typeof dataTypeExpression !== "string" ) { + func = dataTypeExpression; + dataTypeExpression = "*"; + } + + var dataType, + i = 0, + dataTypes = dataTypeExpression.toLowerCase().match( rnothtmlwhite ) || []; + + if ( isFunction( func ) ) { + + // For each dataType in the dataTypeExpression + while ( ( dataType = dataTypes[ i++ ] ) ) { + + // Prepend if requested + if ( dataType[ 0 ] === "+" ) { + dataType = dataType.slice( 1 ) || "*"; + ( structure[ dataType ] = structure[ dataType ] || [] ).unshift( func ); + + // Otherwise append + } else { + ( structure[ dataType ] = structure[ dataType ] || [] ).push( func ); + } + } + } + }; +} + +// Base inspection function for prefilters and transports +function inspectPrefiltersOrTransports( structure, options, originalOptions, jqXHR ) { + + var inspected = {}, + seekingTransport = ( structure === transports ); + + function inspect( dataType ) { + var selected; + inspected[ dataType ] = true; + jQuery.each( structure[ dataType ] || [], function( _, prefilterOrFactory ) { + var dataTypeOrTransport = prefilterOrFactory( options, originalOptions, jqXHR ); + if ( typeof dataTypeOrTransport === "string" && + !seekingTransport && !inspected[ dataTypeOrTransport ] ) { + + options.dataTypes.unshift( dataTypeOrTransport ); + inspect( dataTypeOrTransport ); + return false; + } else if ( seekingTransport ) { + return !( selected = dataTypeOrTransport ); + } + } ); + return selected; + } + + return inspect( options.dataTypes[ 0 ] ) || !inspected[ "*" ] && inspect( "*" ); +} + +// A special extend for ajax options +// that takes "flat" options (not to be deep extended) +// Fixes #9887 +function ajaxExtend( target, src ) { + var key, deep, + flatOptions = jQuery.ajaxSettings.flatOptions || {}; + + for ( key in src ) { + if ( src[ key ] !== undefined ) { + ( flatOptions[ key ] ? target : ( deep || ( deep = {} ) ) )[ key ] = src[ key ]; + } + } + if ( deep ) { + jQuery.extend( true, target, deep ); + } + + return target; +} + +/* Handles responses to an ajax request: + * - finds the right dataType (mediates between content-type and expected dataType) + * - returns the corresponding response + */ +function ajaxHandleResponses( s, jqXHR, responses ) { + + var ct, type, finalDataType, firstDataType, + contents = s.contents, + dataTypes = s.dataTypes; + + // Remove auto dataType and get content-type in the process + while ( dataTypes[ 0 ] === "*" ) { + dataTypes.shift(); + if ( ct === undefined ) { + ct = s.mimeType || jqXHR.getResponseHeader( "Content-Type" ); + } + } + + // Check if we're dealing with a known content-type + if ( ct ) { + for ( type in contents ) { + if ( contents[ type ] && contents[ type ].test( ct ) ) { + dataTypes.unshift( type ); + break; + } + } + } + + // Check to see if we have a response for the expected dataType + if ( dataTypes[ 0 ] in responses ) { + finalDataType = dataTypes[ 0 ]; + } else { + + // Try convertible dataTypes + for ( type in responses ) { + if ( !dataTypes[ 0 ] || s.converters[ type + " " + dataTypes[ 0 ] ] ) { + finalDataType = type; + break; + } + if ( !firstDataType ) { + firstDataType = type; + } + } + + // Or just use first one + finalDataType = finalDataType || firstDataType; + } + + // If we found a dataType + // We add the dataType to the list if needed + // and return the corresponding response + if ( finalDataType ) { + if ( finalDataType !== dataTypes[ 0 ] ) { + dataTypes.unshift( finalDataType ); + } + return responses[ finalDataType ]; + } +} + +/* Chain conversions given the request and the original response + * Also sets the responseXXX fields on the jqXHR instance + */ +function ajaxConvert( s, response, jqXHR, isSuccess ) { + var conv2, current, conv, tmp, prev, + converters = {}, + + // Work with a copy of dataTypes in case we need to modify it for conversion + dataTypes = s.dataTypes.slice(); + + // Create converters map with lowercased keys + if ( dataTypes[ 1 ] ) { + for ( conv in s.converters ) { + converters[ conv.toLowerCase() ] = s.converters[ conv ]; + } + } + + current = dataTypes.shift(); + + // Convert to each sequential dataType + while ( current ) { + + if ( s.responseFields[ current ] ) { + jqXHR[ s.responseFields[ current ] ] = response; + } + + // Apply the dataFilter if provided + if ( !prev && isSuccess && s.dataFilter ) { + response = s.dataFilter( response, s.dataType ); + } + + prev = current; + current = dataTypes.shift(); + + if ( current ) { + + // There's only work to do if current dataType is non-auto + if ( current === "*" ) { + + current = prev; + + // Convert response if prev dataType is non-auto and differs from current + } else if ( prev !== "*" && prev !== current ) { + + // Seek a direct converter + conv = converters[ prev + " " + current ] || converters[ "* " + current ]; + + // If none found, seek a pair + if ( !conv ) { + for ( conv2 in converters ) { + + // If conv2 outputs current + tmp = conv2.split( " " ); + if ( tmp[ 1 ] === current ) { + + // If prev can be converted to accepted input + conv = converters[ prev + " " + tmp[ 0 ] ] || + converters[ "* " + tmp[ 0 ] ]; + if ( conv ) { + + // Condense equivalence converters + if ( conv === true ) { + conv = converters[ conv2 ]; + + // Otherwise, insert the intermediate dataType + } else if ( converters[ conv2 ] !== true ) { + current = tmp[ 0 ]; + dataTypes.unshift( tmp[ 1 ] ); + } + break; + } + } + } + } + + // Apply converter (if not an equivalence) + if ( conv !== true ) { + + // Unless errors are allowed to bubble, catch and return them + if ( conv && s.throws ) { + response = conv( response ); + } else { + try { + response = conv( response ); + } catch ( e ) { + return { + state: "parsererror", + error: conv ? e : "No conversion from " + prev + " to " + current + }; + } + } + } + } + } + } + + return { state: "success", data: response }; +} + +jQuery.extend( { + + // Counter for holding the number of active queries + active: 0, + + // Last-Modified header cache for next request + lastModified: {}, + etag: {}, + + ajaxSettings: { + url: location.href, + type: "GET", + isLocal: rlocalProtocol.test( location.protocol ), + global: true, + processData: true, + async: true, + contentType: "application/x-www-form-urlencoded; charset=UTF-8", + + /* + timeout: 0, + data: null, + dataType: null, + username: null, + password: null, + cache: null, + throws: false, + traditional: false, + headers: {}, + */ + + accepts: { + "*": allTypes, + text: "text/plain", + html: "text/html", + xml: "application/xml, text/xml", + json: "application/json, text/javascript" + }, + + contents: { + xml: /\bxml\b/, + html: /\bhtml/, + json: /\bjson\b/ + }, + + responseFields: { + xml: "responseXML", + text: "responseText", + json: "responseJSON" + }, + + // Data converters + // Keys separate source (or catchall "*") and destination types with a single space + converters: { + + // Convert anything to text + "* text": String, + + // Text to html (true = no transformation) + "text html": true, + + // Evaluate text as a json expression + "text json": JSON.parse, + + // Parse text as xml + "text xml": jQuery.parseXML + }, + + // For options that shouldn't be deep extended: + // you can add your own custom options here if + // and when you create one that shouldn't be + // deep extended (see ajaxExtend) + flatOptions: { + url: true, + context: true + } + }, + + // Creates a full fledged settings object into target + // with both ajaxSettings and settings fields. + // If target is omitted, writes into ajaxSettings. + ajaxSetup: function( target, settings ) { + return settings ? + + // Building a settings object + ajaxExtend( ajaxExtend( target, jQuery.ajaxSettings ), settings ) : + + // Extending ajaxSettings + ajaxExtend( jQuery.ajaxSettings, target ); + }, + + ajaxPrefilter: addToPrefiltersOrTransports( prefilters ), + ajaxTransport: addToPrefiltersOrTransports( transports ), + + // Main method + ajax: function( url, options ) { + + // If url is an object, simulate pre-1.5 signature + if ( typeof url === "object" ) { + options = url; + url = undefined; + } + + // Force options to be an object + options = options || {}; + + var transport, + + // URL without anti-cache param + cacheURL, + + // Response headers + responseHeadersString, + responseHeaders, + + // timeout handle + timeoutTimer, + + // Url cleanup var + urlAnchor, + + // Request state (becomes false upon send and true upon completion) + completed, + + // To know if global events are to be dispatched + fireGlobals, + + // Loop variable + i, + + // uncached part of the url + uncached, + + // Create the final options object + s = jQuery.ajaxSetup( {}, options ), + + // Callbacks context + callbackContext = s.context || s, + + // Context for global events is callbackContext if it is a DOM node or jQuery collection + globalEventContext = s.context && + ( callbackContext.nodeType || callbackContext.jquery ) ? + jQuery( callbackContext ) : + jQuery.event, + + // Deferreds + deferred = jQuery.Deferred(), + completeDeferred = jQuery.Callbacks( "once memory" ), + + // Status-dependent callbacks + statusCode = s.statusCode || {}, + + // Headers (they are sent all at once) + requestHeaders = {}, + requestHeadersNames = {}, + + // Default abort message + strAbort = "canceled", + + // Fake xhr + jqXHR = { + readyState: 0, + + // Builds headers hashtable if needed + getResponseHeader: function( key ) { + var match; + if ( completed ) { + if ( !responseHeaders ) { + responseHeaders = {}; + while ( ( match = rheaders.exec( responseHeadersString ) ) ) { + responseHeaders[ match[ 1 ].toLowerCase() + " " ] = + ( responseHeaders[ match[ 1 ].toLowerCase() + " " ] || [] ) + .concat( match[ 2 ] ); + } + } + match = responseHeaders[ key.toLowerCase() + " " ]; + } + return match == null ? null : match.join( ", " ); + }, + + // Raw string + getAllResponseHeaders: function() { + return completed ? responseHeadersString : null; + }, + + // Caches the header + setRequestHeader: function( name, value ) { + if ( completed == null ) { + name = requestHeadersNames[ name.toLowerCase() ] = + requestHeadersNames[ name.toLowerCase() ] || name; + requestHeaders[ name ] = value; + } + return this; + }, + + // Overrides response content-type header + overrideMimeType: function( type ) { + if ( completed == null ) { + s.mimeType = type; + } + return this; + }, + + // Status-dependent callbacks + statusCode: function( map ) { + var code; + if ( map ) { + if ( completed ) { + + // Execute the appropriate callbacks + jqXHR.always( map[ jqXHR.status ] ); + } else { + + // Lazy-add the new callbacks in a way that preserves old ones + for ( code in map ) { + statusCode[ code ] = [ statusCode[ code ], map[ code ] ]; + } + } + } + return this; + }, + + // Cancel the request + abort: function( statusText ) { + var finalText = statusText || strAbort; + if ( transport ) { + transport.abort( finalText ); + } + done( 0, finalText ); + return this; + } + }; + + // Attach deferreds + deferred.promise( jqXHR ); + + // Add protocol if not provided (prefilters might expect it) + // Handle falsy url in the settings object (#10093: consistency with old signature) + // We also use the url parameter if available + s.url = ( ( url || s.url || location.href ) + "" ) + .replace( rprotocol, location.protocol + "//" ); + + // Alias method option to type as per ticket #12004 + s.type = options.method || options.type || s.method || s.type; + + // Extract dataTypes list + s.dataTypes = ( s.dataType || "*" ).toLowerCase().match( rnothtmlwhite ) || [ "" ]; + + // A cross-domain request is in order when the origin doesn't match the current origin. + if ( s.crossDomain == null ) { + urlAnchor = document.createElement( "a" ); + + // Support: IE <=8 - 11, Edge 12 - 15 + // IE throws exception on accessing the href property if url is malformed, + // e.g. http://example.com:80x/ + try { + urlAnchor.href = s.url; + + // Support: IE <=8 - 11 only + // Anchor's host property isn't correctly set when s.url is relative + urlAnchor.href = urlAnchor.href; + s.crossDomain = originAnchor.protocol + "//" + originAnchor.host !== + urlAnchor.protocol + "//" + urlAnchor.host; + } catch ( e ) { + + // If there is an error parsing the URL, assume it is crossDomain, + // it can be rejected by the transport if it is invalid + s.crossDomain = true; + } + } + + // Convert data if not already a string + if ( s.data && s.processData && typeof s.data !== "string" ) { + s.data = jQuery.param( s.data, s.traditional ); + } + + // Apply prefilters + inspectPrefiltersOrTransports( prefilters, s, options, jqXHR ); + + // If request was aborted inside a prefilter, stop there + if ( completed ) { + return jqXHR; + } + + // We can fire global events as of now if asked to + // Don't fire events if jQuery.event is undefined in an AMD-usage scenario (#15118) + fireGlobals = jQuery.event && s.global; + + // Watch for a new set of requests + if ( fireGlobals && jQuery.active++ === 0 ) { + jQuery.event.trigger( "ajaxStart" ); + } + + // Uppercase the type + s.type = s.type.toUpperCase(); + + // Determine if request has content + s.hasContent = !rnoContent.test( s.type ); + + // Save the URL in case we're toying with the If-Modified-Since + // and/or If-None-Match header later on + // Remove hash to simplify url manipulation + cacheURL = s.url.replace( rhash, "" ); + + // More options handling for requests with no content + if ( !s.hasContent ) { + + // Remember the hash so we can put it back + uncached = s.url.slice( cacheURL.length ); + + // If data is available and should be processed, append data to url + if ( s.data && ( s.processData || typeof s.data === "string" ) ) { + cacheURL += ( rquery.test( cacheURL ) ? "&" : "?" ) + s.data; + + // #9682: remove data so that it's not used in an eventual retry + delete s.data; + } + + // Add or update anti-cache param if needed + if ( s.cache === false ) { + cacheURL = cacheURL.replace( rantiCache, "$1" ); + uncached = ( rquery.test( cacheURL ) ? "&" : "?" ) + "_=" + ( nonce.guid++ ) + + uncached; + } + + // Put hash and anti-cache on the URL that will be requested (gh-1732) + s.url = cacheURL + uncached; + + // Change '%20' to '+' if this is encoded form body content (gh-2658) + } else if ( s.data && s.processData && + ( s.contentType || "" ).indexOf( "application/x-www-form-urlencoded" ) === 0 ) { + s.data = s.data.replace( r20, "+" ); + } + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + if ( jQuery.lastModified[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-Modified-Since", jQuery.lastModified[ cacheURL ] ); + } + if ( jQuery.etag[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-None-Match", jQuery.etag[ cacheURL ] ); + } + } + + // Set the correct header, if data is being sent + if ( s.data && s.hasContent && s.contentType !== false || options.contentType ) { + jqXHR.setRequestHeader( "Content-Type", s.contentType ); + } + + // Set the Accepts header for the server, depending on the dataType + jqXHR.setRequestHeader( + "Accept", + s.dataTypes[ 0 ] && s.accepts[ s.dataTypes[ 0 ] ] ? + s.accepts[ s.dataTypes[ 0 ] ] + + ( s.dataTypes[ 0 ] !== "*" ? ", " + allTypes + "; q=0.01" : "" ) : + s.accepts[ "*" ] + ); + + // Check for headers option + for ( i in s.headers ) { + jqXHR.setRequestHeader( i, s.headers[ i ] ); + } + + // Allow custom headers/mimetypes and early abort + if ( s.beforeSend && + ( s.beforeSend.call( callbackContext, jqXHR, s ) === false || completed ) ) { + + // Abort if not done already and return + return jqXHR.abort(); + } + + // Aborting is no longer a cancellation + strAbort = "abort"; + + // Install callbacks on deferreds + completeDeferred.add( s.complete ); + jqXHR.done( s.success ); + jqXHR.fail( s.error ); + + // Get transport + transport = inspectPrefiltersOrTransports( transports, s, options, jqXHR ); + + // If no transport, we auto-abort + if ( !transport ) { + done( -1, "No Transport" ); + } else { + jqXHR.readyState = 1; + + // Send global event + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxSend", [ jqXHR, s ] ); + } + + // If request was aborted inside ajaxSend, stop there + if ( completed ) { + return jqXHR; + } + + // Timeout + if ( s.async && s.timeout > 0 ) { + timeoutTimer = window.setTimeout( function() { + jqXHR.abort( "timeout" ); + }, s.timeout ); + } + + try { + completed = false; + transport.send( requestHeaders, done ); + } catch ( e ) { + + // Rethrow post-completion exceptions + if ( completed ) { + throw e; + } + + // Propagate others as results + done( -1, e ); + } + } + + // Callback for when everything is done + function done( status, nativeStatusText, responses, headers ) { + var isSuccess, success, error, response, modified, + statusText = nativeStatusText; + + // Ignore repeat invocations + if ( completed ) { + return; + } + + completed = true; + + // Clear timeout if it exists + if ( timeoutTimer ) { + window.clearTimeout( timeoutTimer ); + } + + // Dereference transport for early garbage collection + // (no matter how long the jqXHR object will be used) + transport = undefined; + + // Cache response headers + responseHeadersString = headers || ""; + + // Set readyState + jqXHR.readyState = status > 0 ? 4 : 0; + + // Determine if successful + isSuccess = status >= 200 && status < 300 || status === 304; + + // Get response data + if ( responses ) { + response = ajaxHandleResponses( s, jqXHR, responses ); + } + + // Use a noop converter for missing script but not if jsonp + if ( !isSuccess && + jQuery.inArray( "script", s.dataTypes ) > -1 && + jQuery.inArray( "json", s.dataTypes ) < 0 ) { + s.converters[ "text script" ] = function() {}; + } + + // Convert no matter what (that way responseXXX fields are always set) + response = ajaxConvert( s, response, jqXHR, isSuccess ); + + // If successful, handle type chaining + if ( isSuccess ) { + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + modified = jqXHR.getResponseHeader( "Last-Modified" ); + if ( modified ) { + jQuery.lastModified[ cacheURL ] = modified; + } + modified = jqXHR.getResponseHeader( "etag" ); + if ( modified ) { + jQuery.etag[ cacheURL ] = modified; + } + } + + // if no content + if ( status === 204 || s.type === "HEAD" ) { + statusText = "nocontent"; + + // if not modified + } else if ( status === 304 ) { + statusText = "notmodified"; + + // If we have data, let's convert it + } else { + statusText = response.state; + success = response.data; + error = response.error; + isSuccess = !error; + } + } else { + + // Extract error from statusText and normalize for non-aborts + error = statusText; + if ( status || !statusText ) { + statusText = "error"; + if ( status < 0 ) { + status = 0; + } + } + } + + // Set data for the fake xhr object + jqXHR.status = status; + jqXHR.statusText = ( nativeStatusText || statusText ) + ""; + + // Success/Error + if ( isSuccess ) { + deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] ); + } else { + deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] ); + } + + // Status-dependent callbacks + jqXHR.statusCode( statusCode ); + statusCode = undefined; + + if ( fireGlobals ) { + globalEventContext.trigger( isSuccess ? "ajaxSuccess" : "ajaxError", + [ jqXHR, s, isSuccess ? success : error ] ); + } + + // Complete + completeDeferred.fireWith( callbackContext, [ jqXHR, statusText ] ); + + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxComplete", [ jqXHR, s ] ); + + // Handle the global AJAX counter + if ( !( --jQuery.active ) ) { + jQuery.event.trigger( "ajaxStop" ); + } + } + } + + return jqXHR; + }, + + getJSON: function( url, data, callback ) { + return jQuery.get( url, data, callback, "json" ); + }, + + getScript: function( url, callback ) { + return jQuery.get( url, undefined, callback, "script" ); + } +} ); + +jQuery.each( [ "get", "post" ], function( _i, method ) { + jQuery[ method ] = function( url, data, callback, type ) { + + // Shift arguments if data argument was omitted + if ( isFunction( data ) ) { + type = type || callback; + callback = data; + data = undefined; + } + + // The url can be an options object (which then must have .url) + return jQuery.ajax( jQuery.extend( { + url: url, + type: method, + dataType: type, + data: data, + success: callback + }, jQuery.isPlainObject( url ) && url ) ); + }; +} ); + +jQuery.ajaxPrefilter( function( s ) { + var i; + for ( i in s.headers ) { + if ( i.toLowerCase() === "content-type" ) { + s.contentType = s.headers[ i ] || ""; + } + } +} ); + + +jQuery._evalUrl = function( url, options, doc ) { + return jQuery.ajax( { + url: url, + + // Make this explicit, since user can override this through ajaxSetup (#11264) + type: "GET", + dataType: "script", + cache: true, + async: false, + global: false, + + // Only evaluate the response if it is successful (gh-4126) + // dataFilter is not invoked for failure responses, so using it instead + // of the default converter is kludgy but it works. + converters: { + "text script": function() {} + }, + dataFilter: function( response ) { + jQuery.globalEval( response, options, doc ); + } + } ); +}; + + +jQuery.fn.extend( { + wrapAll: function( html ) { + var wrap; + + if ( this[ 0 ] ) { + if ( isFunction( html ) ) { + html = html.call( this[ 0 ] ); + } + + // The elements to wrap the target around + wrap = jQuery( html, this[ 0 ].ownerDocument ).eq( 0 ).clone( true ); + + if ( this[ 0 ].parentNode ) { + wrap.insertBefore( this[ 0 ] ); + } + + wrap.map( function() { + var elem = this; + + while ( elem.firstElementChild ) { + elem = elem.firstElementChild; + } + + return elem; + } ).append( this ); + } + + return this; + }, + + wrapInner: function( html ) { + if ( isFunction( html ) ) { + return this.each( function( i ) { + jQuery( this ).wrapInner( html.call( this, i ) ); + } ); + } + + return this.each( function() { + var self = jQuery( this ), + contents = self.contents(); + + if ( contents.length ) { + contents.wrapAll( html ); + + } else { + self.append( html ); + } + } ); + }, + + wrap: function( html ) { + var htmlIsFunction = isFunction( html ); + + return this.each( function( i ) { + jQuery( this ).wrapAll( htmlIsFunction ? html.call( this, i ) : html ); + } ); + }, + + unwrap: function( selector ) { + this.parent( selector ).not( "body" ).each( function() { + jQuery( this ).replaceWith( this.childNodes ); + } ); + return this; + } +} ); + + +jQuery.expr.pseudos.hidden = function( elem ) { + return !jQuery.expr.pseudos.visible( elem ); +}; +jQuery.expr.pseudos.visible = function( elem ) { + return !!( elem.offsetWidth || elem.offsetHeight || elem.getClientRects().length ); +}; + + + + +jQuery.ajaxSettings.xhr = function() { + try { + return new window.XMLHttpRequest(); + } catch ( e ) {} +}; + +var xhrSuccessStatus = { + + // File protocol always yields status code 0, assume 200 + 0: 200, + + // Support: IE <=9 only + // #1450: sometimes IE returns 1223 when it should be 204 + 1223: 204 + }, + xhrSupported = jQuery.ajaxSettings.xhr(); + +support.cors = !!xhrSupported && ( "withCredentials" in xhrSupported ); +support.ajax = xhrSupported = !!xhrSupported; + +jQuery.ajaxTransport( function( options ) { + var callback, errorCallback; + + // Cross domain only allowed if supported through XMLHttpRequest + if ( support.cors || xhrSupported && !options.crossDomain ) { + return { + send: function( headers, complete ) { + var i, + xhr = options.xhr(); + + xhr.open( + options.type, + options.url, + options.async, + options.username, + options.password + ); + + // Apply custom fields if provided + if ( options.xhrFields ) { + for ( i in options.xhrFields ) { + xhr[ i ] = options.xhrFields[ i ]; + } + } + + // Override mime type if needed + if ( options.mimeType && xhr.overrideMimeType ) { + xhr.overrideMimeType( options.mimeType ); + } + + // X-Requested-With header + // For cross-domain requests, seeing as conditions for a preflight are + // akin to a jigsaw puzzle, we simply never set it to be sure. + // (it can always be set on a per-request basis or even using ajaxSetup) + // For same-domain requests, won't change header if already provided. + if ( !options.crossDomain && !headers[ "X-Requested-With" ] ) { + headers[ "X-Requested-With" ] = "XMLHttpRequest"; + } + + // Set headers + for ( i in headers ) { + xhr.setRequestHeader( i, headers[ i ] ); + } + + // Callback + callback = function( type ) { + return function() { + if ( callback ) { + callback = errorCallback = xhr.onload = + xhr.onerror = xhr.onabort = xhr.ontimeout = + xhr.onreadystatechange = null; + + if ( type === "abort" ) { + xhr.abort(); + } else if ( type === "error" ) { + + // Support: IE <=9 only + // On a manual native abort, IE9 throws + // errors on any property access that is not readyState + if ( typeof xhr.status !== "number" ) { + complete( 0, "error" ); + } else { + complete( + + // File: protocol always yields status 0; see #8605, #14207 + xhr.status, + xhr.statusText + ); + } + } else { + complete( + xhrSuccessStatus[ xhr.status ] || xhr.status, + xhr.statusText, + + // Support: IE <=9 only + // IE9 has no XHR2 but throws on binary (trac-11426) + // For XHR2 non-text, let the caller handle it (gh-2498) + ( xhr.responseType || "text" ) !== "text" || + typeof xhr.responseText !== "string" ? + { binary: xhr.response } : + { text: xhr.responseText }, + xhr.getAllResponseHeaders() + ); + } + } + }; + }; + + // Listen to events + xhr.onload = callback(); + errorCallback = xhr.onerror = xhr.ontimeout = callback( "error" ); + + // Support: IE 9 only + // Use onreadystatechange to replace onabort + // to handle uncaught aborts + if ( xhr.onabort !== undefined ) { + xhr.onabort = errorCallback; + } else { + xhr.onreadystatechange = function() { + + // Check readyState before timeout as it changes + if ( xhr.readyState === 4 ) { + + // Allow onerror to be called first, + // but that will not handle a native abort + // Also, save errorCallback to a variable + // as xhr.onerror cannot be accessed + window.setTimeout( function() { + if ( callback ) { + errorCallback(); + } + } ); + } + }; + } + + // Create the abort callback + callback = callback( "abort" ); + + try { + + // Do send the request (this may raise an exception) + xhr.send( options.hasContent && options.data || null ); + } catch ( e ) { + + // #14683: Only rethrow if this hasn't been notified as an error yet + if ( callback ) { + throw e; + } + } + }, + + abort: function() { + if ( callback ) { + callback(); + } + } + }; + } +} ); + + + + +// Prevent auto-execution of scripts when no explicit dataType was provided (See gh-2432) +jQuery.ajaxPrefilter( function( s ) { + if ( s.crossDomain ) { + s.contents.script = false; + } +} ); + +// Install script dataType +jQuery.ajaxSetup( { + accepts: { + script: "text/javascript, application/javascript, " + + "application/ecmascript, application/x-ecmascript" + }, + contents: { + script: /\b(?:java|ecma)script\b/ + }, + converters: { + "text script": function( text ) { + jQuery.globalEval( text ); + return text; + } + } +} ); + +// Handle cache's special case and crossDomain +jQuery.ajaxPrefilter( "script", function( s ) { + if ( s.cache === undefined ) { + s.cache = false; + } + if ( s.crossDomain ) { + s.type = "GET"; + } +} ); + +// Bind script tag hack transport +jQuery.ajaxTransport( "script", function( s ) { + + // This transport only deals with cross domain or forced-by-attrs requests + if ( s.crossDomain || s.scriptAttrs ) { + var script, callback; + return { + send: function( _, complete ) { + script = jQuery( " + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Accounts

+

The accounts section is for the management of user accounts, access permissions, and tenants.

+
+

Users

+

Description of User account fields:

+
    +
  • Username - Unique username.

  • +
  • Full Name - Full Name of the user.

  • +
  • E-mail - The email address of the user. Also used for system notifications and for password retrieval.

  • +
  • E-mail CC - Send copies of email notifications to this address.

  • +
  • Phone Number - User’s phone number.

  • +
  • Company - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers.

  • +
  • Position - Position within the company.

  • +
  • User Role - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate.

  • +
  • Permission Group - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used)

  • +
  • +Tenant - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used)

  • +
+

Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin.

+User Management +

Password: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side.

+

Example: Listing of user accounts.

+List User Accounts +
+
+

Tenants

+

IP addresses and Switch Ports are network resources that can be assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs) as part of DevOps/NetOps pipeline.

+

A Tenant has just two fields, the unique name and custom description.

+

Example: Adding a tenant.

+Adding Tenants +
+
+

Permission Groups

+

Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections.

+

Example: Permission Group.

+Managing Permissions +
+
+

User Roles

+

Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username.

+User Roles +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/acls.html b/en/3.4/acls.html new file mode 100644 index 0000000000..108c9afeda --- /dev/null +++ b/en/3.4/acls.html @@ -0,0 +1,786 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Access Control Lists (ACL) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Access Control Lists (ACL)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Access Control Lists (ACL)

+

Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access.

+

Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches.

+

Screenshot: TCAM utilization can be seen under Net→Inventory

+_images/TCAM.png +

Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements.

+
+

ACL Default Policy

+

The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules.

+

Example: Changing “ACL Default Policy” for the site “siteDefault”.

+_images/siteDefault.png +
+
+

ACL Rules

+

ACL rules can be created, listed, edited, approved under Services→ACL.

+

Description of ACL fields. +General

+
    +
  • Name - Unique name for the ACL entry.

  • +
  • Protocol - IP protocol to match.

    +
      +
    • All - Any IP protocols.

    • +
    • IP - Specific IP protocol number.

    • +
    • TCP - TCP.

    • +
    • UDP - UDP.

    • +
    • ICMP ALL - Any IPv4 ICMP protocol.

    • +
    • ICMP Custom - Custom IPv4 ICMP code.

    • +
    • ICMPv6 ALL - Any IPv6 ICMP protocol.

    • +
    • ICMPv6 Custom - Custom IPv6 ICMP code.

    • +
    +
  • +
  • Active Until - Disable this rule at the defined date/time.

  • +
  • Action - Permit or Deny forwarding of matched packets.

  • +
  • Established/Reverse - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination.

  • +
+

Source/Destination - Source and destination addresses and ports to match.

+
    +
  • Source IPv4/IPv6 - IPv4/IPv6 address.

  • +
  • Ports Type

    +
      +
    • Port Range - Match on the port or a port range defined in this window.

    • +
    • Port Group - Match on a group of ports defined under Services→ ACL Port Group.

    • +
    +
  • +
  • From Port - Port range starting from.

  • +
  • To Port - Port range ending with.

  • +
  • Comment - Descriptive comment, commonly used for approval workflows.

  • +
  • Check button - Check if Another ACL on the system already permits the described network access.

  • +
+

Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established).

+_images/action_permit.png +

Example: “Check” shows that requested access is already provided by a broader ACL rule.

+_images/ACL_rule.png +
+
+

ACL Approval Workflow

+

When one tenant (one team) needs to get network access to resources under the responsibility of another tenant (another team), an ACL can be created but will activate only after approval of the tenant responsible for the destination address resources. See the below example.

+

Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant.

+_images/ACL_approval.png +

Screenshot: ACL stays in “waiting for approval” state until approved.

+_images/waiting_approval.png +

Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it.

+_images/approve_reject.png +

Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric.

+_images/ACL_active.png +
+
+

ACL Processing Order

+
    +
  1. User-defined Deny Rules

  2. +
  3. User-defined Permit Rules

  4. +
  5. Deny the rest

  6. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/controller-k3s-installation.html b/en/3.4/controller-k3s-installation.html new file mode 100644 index 0000000000..c8aff6cde5 --- /dev/null +++ b/en/3.4/controller-k3s-installation.html @@ -0,0 +1,885 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Netris Controller installation on a generic Linux host — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Controller installation on a generic Linux host

+
+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+
+

Note

+

K3s is expected to work on most modern Linux systems.

+

Some OSs have specific requirements:

+
    +
  • If you are using Raspbian Buster, follow these steps to switch to legacy iptables.

  • +
  • If you are using Alpine Linux, follow these steps for additional setup.

  • +
  • If you are using (Red Hat/CentOS) Enterprise Linux, follow these steps for additional setup.

  • +
+
+
+
+

Installation

+

The following command will install the Netris Controller on your Linux server:

+
curl -sfL https://get.netris.io | sh -
+
+
+

Once installed, you will be able to log in to Netris Controller using your host’s IP address.

+
+

Note

+

The installation script does the following:

+ +
+
+

Installation with the specific host name

+

In order to set the specific ingress host name to the Netris Controller, use the --ctl-hostname installation argument:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com
+
+
+

A self-signed SSL certificate will be generated from that host name.

+
+
+

Installation with the Let’s Encrypt SSL

+

The installation script supports Let’s Encrypt SSL generation out-of-box. To instruct the installation script to do that use --ctl-ssl-issuer argument.

+
+

Note

+
+
The argument --ctl-ssl-issuer is passing cert-manager.io/cluster-issuer value to the ingress resource of the Netris Controller. The installation script can create two types of ClusterIssuer resource: selfsigned or letsencrypt, where selfsigned is just Cert-Manager self-signed SSL and the letsencrypt is the ACME issuer with HTTP01 challenge validation.
+
If the --ctl-ssl-issuer argument is not set, the installation script will proceed with selfsigned ClusterIssuer type.
+
+
+

Run the following command to install Netris Controller and use letsencrypt ClusterIssuer for SSL generation:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

To successfully validate and complete Let’s Encrypt SSL generation, a valid A/CNAME record for the domain/subdomain name should exist prior, and that name must be accessible from the Internet.

+
+
+
+

Installation with the Custom SSL Issuer

+

The HTTP01 challenge validation is the simplest way of issuing the Let’s Encrypt SSL, but it does not work when the host behind the FQDN is not accessible from the public internet. +The common approach of validating and completing Let’s Encrypt SSL generation for private deployments is DNS01 challenge validation. +If the DNS01 does not work for you either, Cert-Manager supports a number of certificate issuers, get familiar with all types of issuers here.

+

In order to install Netris Controller with the custom SSL issuer, you need to run installation script with the specified host name:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com
+
+
+

Once the installation is complete, create a yaml file with the ClusterIssuer resource, suitable for your requirements, and apply it:

+
kubectl apply -f my-cluster-issuer.yaml
+
+
+

Then rerun the installation script with the --ctl-ssl-issuer argument:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-ssl-issuer <Your ClusterIssuer resource name>
+
+
+
+
+
+

Upgrading

+

To upgrade the Netris Controller to the latest version simply run the script:

+
curl -sfL https://get.netris.io | sh -
+
+
+

If a newer version of Netris Controller is available, it will be updated in a few minutes.

+
+
+

Uninstalling

+

To uninstall Netris Controller and K3s from a server node, run:

+
/usr/local/bin/k3s-uninstall.sh
+
+
+
+
+

Backup and Restore

+

Netris Controller stores all critical data in MariaDB. It’s highly recommended to create a cronjob with mysqldump.

+
+

Backup

+

To take database snapshot run the following command:

+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot-$(date +%Y-%m-%d-%H-%M-%S).sql
+
+
+

After command execution, you can find db-snapshot-YYYY-MM-DD-HH-MM-SS.sql file in the current working directory.

+
+

Backup the Secret Key

+

Netris Controller generates a unique secret key at the first installation. If you’re moving or reinstalling your controller, it makes sense to take note of the secret key for restoring purpose in the future. Overwise, you have to reinitiate all devices connected to the controller.

+
kubectl -n netris-controller get secret netris-controller-grpc-secret -o jsonpath='{.data.secret-key}{"\n"}'
+
+
+
+
+
+

Restore

+

In order to restore DB from a database snapshot, follow these steps:

+
    +
  1. Drop the current database by running the following command:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"'
+
+
+
    +
  1. Create a new database:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"'
+
+
+
    +
  1. Copy snapshot file to the MariaDB container:

  2. +
+
kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql
+
+
+
    +
  1. Run the restore command:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql'
+
+
+
+

Note

+

In this example the snapshot file name is db-snapshot.sql and it’s located in the current working directory

+
+
+

Restore the Secret Key

+

If you want to restore the controller secret key too (you might want to do that if you’re reinstalling or moving the controller to the other place), follow these steps:

+
    +
  1. Set OLD_SECRET environment variable (the secret key taken from the old controller):

  2. +
+
export OLD_SECRET=<Your old secret key>
+
+
+

example: export OLD_SECRET=VUdodFFSakJCU2lFVVA4T1c0cnpuUmdiMkQxem85Y2dnS3pkajlNSg==

+
    +
  1. Update the secret key of the new controller:

  2. +
+
kubectl -n netris-controller patch secret netris-controller-grpc-secret --type='json' -p='[{"op" : "replace" ,"path" : "/data/secret-key" ,"value" : "'$OLD_SECRET'"}]'
+
+
+
    +
  1. Restart Netris Controller’s all microservices

  2. +
+
kubectl -n netris-controller rollout restart deployments
+
+
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/controller-k8s-installation.html b/en/3.4/controller-k8s-installation.html new file mode 100644 index 0000000000..4e65bd55be --- /dev/null +++ b/en/3.4/controller-k8s-installation.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Helm Chart Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Helm Chart Installation

+
+

Requirements

+
    +
  • Kubernetes 1.12+

  • +
  • Helm 3.1+

  • +
  • PV provisioner support in the underlying infrastructure

  • +
+
+
+

Get Repo Info

+

Add the Netris Helm repository:

+
helm repo add netrisai https://netrisai.github.io/charts
+helm repo update
+
+
+
+
+

Installing the Chart

+

In order to install the Helm chart, you must follow these steps:

+
    +
  1. Create the namespace for netris-controller:

  2. +
+
kubectl create namespace netris-controller
+
+
+
    +
  1. Install helm chart with netris-controller:

  2. +
+
helm install netris-controller netrisai/netris-controller \
+  --namespace netris-controller \
+  --set ingress.hosts={my.domain.com}
+
+
+
+
+

Uninstalling the Chart

+

To uninstall/delete the netris-controller helm release:

+
helm uninstall netris-controller
+
+
+
+
+

Chart Configuration

+

See the netris-controller README for details about configurable parameters and their default values.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/controller-k8s-quickstart.html b/en/3.4/controller-k8s-quickstart.html new file mode 100644 index 0000000000..556bd9f3c8 --- /dev/null +++ b/en/3.4/controller-k8s-quickstart.html @@ -0,0 +1,728 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Quickstart Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Quickstart Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Quickstart Installation

+

Netris offers a simplified deployment model for users who want to quickly install the Netris Controller in the shortest amount of time.

+

This installation process is streamlined for Linux servers that do not already have Kubernetes running. The install does the following:

+ +

If you wish to install the controller on an existing Kubernetes cluster, follow these instructions instead of this Quickstart.

+
+

Quickstart Process

+
    +
  1. Install the Netris Controller w/ k3s by running the following command on your Linux server:

  2. +
+
curl -sfL https://get.netris.io | sh -
+
+
+
    +
  1. When the installation completes, you will be provided with the login for the web UI. Login with the provided credentials.

  2. +
  3. Navigate in the UI to Net→IPAM and add a new subnet that contains the desired management IP addresses you wish to use for your SoftGates and switches.

    +

    For example, if you are planning on using 192.168.1.100 as the IP address of your Ubuntu server, then create a subnet in Netris UI for 192.168.1.0/24.

    +

    Detailed configuration documentation is available here: Netris IPAM.

    +
  4. +
  5. Navigate in the UI to Topology

  6. +
  7. Click the Add in the upper right

  8. +
  9. Fill out the fields for the SoftGate you wish to add

  10. +
  11. Select the proper Management IP address from the subnet selector

  12. +
  13. Once the SoftGate is created in the Topology, right-click on the SoftGate and select the Install Agent option

  14. +
  15. Copy the agent install command to your clipboard and run the command on the Ubuntu server you are using as your SoftGate

  16. +
  17. Congratulations. The SoftGate should now be connected to your controller.

  18. +
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/controller-vm-installation.html b/en/3.4/controller-vm-installation.html new file mode 100644 index 0000000000..df7c1bf9a2 --- /dev/null +++ b/en/3.4/controller-vm-installation.html @@ -0,0 +1,863 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Virtual Machine Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Virtual Machine Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Virtual Machine Installation

+
+

Requirements

+

Minimal system requirements for the VM:

+
    +
  • CPU - 4 Core

  • +
  • RAM - 4 Gb

  • +
  • Disk - 100Gb

  • +
  • Network - 1 virtual NIC

  • +
+

Recommended system requirements for the VM:

+
    +
  • CPU - 8 Core

  • +
  • RAM - 16 Gb

  • +
  • Disk - 100Gb

  • +
  • Network - 1 virtual NIC

  • +
+
+
+

KVM Hypervisor Installation

+

If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04)

+
sudo apt-get install virt-manager
+
+
+
+
+

VM Controller Installation

+
    +
  1. Download the Netris Controller image. (contact Netris support for repository access permissions).

  2. +
+
cd /var/lib/libvirt/images
+
+sudo wget http://img.netris.io/netris-controller3.qcow2
+
+
+
    +
  1. Download VM definition file.

  2. +
+
cd /etc/libvirt/qemu
+
+sudo wget http://img.netris.io/netris-controller3.xml
+
+
+
    +
  1. Define the KVM virtual machine

  2. +
+
sudo virsh define netris-controller3.xml
+
+
+
+

Note

+

Netris Controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below for the network interface configuration example.

+
+

Example: Network configuration on host (hypervisor) machine.

+
+

Note

+

replace <Physical NIC>, <host server management IP/prefix length> and <host server default gateway> +with the correct NIC and IP for your host machine.

+
+
sudo vim /etc/network/interfaces
+
+
+
#Physical NIC connected to the management network
+auto <Physical NIC>
+iface <Physical NIC> inet static
+                        address 0.0.0.0/0
+
+#bridge interface
+auto br-mgmt
+iface br-mgmt inet static
+                        address <host server management IP/prefix length>
+                        gateway <host server default gateway>
+                        bridge-ports <Physical NIC>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Set the virtual machine to autostart and start it.

  2. +
+
sudo virsh autostart netris-controller
+
+
+
sudo virsh start netris-controller
+
+
+
+
+

Accessing the Netris Controller

+

By default, Netris Controller will obtain an IP address from a DHCP server.

+

Below steps describe how to configure a Static IP address for the Netris Controller.

+
    +
  1. Connecting to the VM console.

  2. +
+

default credentials. login: netris password: newNet0ps

+
sudo virsh console netris-controller
+
+
+
+

Note

+

Do not forget to change the default password (using passwd command).

+
+
    +
  1. Setting a static IP address.

  2. +
+

Edit network configuration file.

+
sudo vim /etc/network/interfaces
+
+
+

Example: IP configuration file.

+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <Netris Controller IP/prefix length>
+        gateway <Netris Controller default gateway>
+        dns-nameserver <a DNS server address>
+
+source /etc/network/interfaces.d/*
+
+
+

Reload the network config.

+
sudo ifreload -a
+
+
+
+

Note

+

Make sure Netris Controller has Internet access.

+
+
    +
  1. Reboot the controller

  2. +
+
sudo reboot
+
+
+

After reboot, the Netris Controller GUI should be accessible using a browser. Use netris/newNet0ps credentials.

+Netris Credentials +
+
+

Replacing the SSL certificate

+
    +
  1. Replace the below file with your SSL certificate file.

  2. +
+
/etc/nginx/ssl/controller.cert.pem;
+
+
+
    +
  1. Replace the below file with your SSL private key.

  2. +
+
/etc/nginx/ssl/controller.key.pem;
+
+
+
    +
  1. Restart Nginx service.

  2. +
+
systemctl restart nginx.service
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/definitions.html b/en/3.4/definitions.html new file mode 100644 index 0000000000..678e5b585c --- /dev/null +++ b/en/3.4/definitions.html @@ -0,0 +1,725 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Definitions — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Definitions

+

When configuring and operating a Netris system, the following nomenclature is important to understand:

+
    +
  • User - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is netris, with password newNet0ps.

  • +
  • Tenant - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request and manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs) via a DevOps/NetOps pipeline.

  • +
  • Permission Group - List of permissions on a per section basis can be attached individually to a User or a User Role

  • +
  • User Role - Group of user permissions and tenants for role-based access control (RBAC)

  • +
  • Site - Each separate deployment (each data center) should be defined as a Site. All network units and resources are attached to a site. Netris Controller comes with a “default” site preconfigured. Site entry defines global attributes such as; AS numbers, default ACL policy, and Site Mesh (site to site VPN) type.

  • +
  • Subnet - IPv4/IPv6 address resources linked to Sites and Tenants

  • +
  • Switch Port - Physical ports of all switches attached to the system

  • +
  • Inventory - Inventory of all network units that are operated using Netris Agent

  • +
  • E-BGP - Defines all External BGP peers (iBGP and eBGP)

  • +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/genindex.html b/en/3.4/genindex.html new file mode 100644 index 0000000000..ec2a289ff8 --- /dev/null +++ b/en/3.4/genindex.html @@ -0,0 +1,700 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Index — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ + +

Index

+ +
+ +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/index.html b/en/3.4/index.html new file mode 100644 index 0000000000..e0ecf7976a --- /dev/null +++ b/en/3.4/index.html @@ -0,0 +1,975 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Documentation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Documentation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Welcome to Netris Documentation

+

Learn how to get started with Netris VPC Networking for your network environment.

+

You are welcome to join our Slack channel to get additional support from our engineers and community.

+ + + + + + + + + + +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/installation.html b/en/3.4/installation.html new file mode 100644 index 0000000000..d2a65860f3 --- /dev/null +++ b/en/3.4/installation.html @@ -0,0 +1,739 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Controller Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Controller Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Controller Installation

+

Netris Controller can be installed locally as a VM, deployed as a Kubernetes application, or hosted in the Netris Cloud. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime.

+
+

Note

+

Select ONE installation option below.

+
+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/installing-netris-controller.html b/en/3.4/installing-netris-controller.html new file mode 100644 index 0000000000..30d1b6957f --- /dev/null +++ b/en/3.4/installing-netris-controller.html @@ -0,0 +1,718 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Installing a Netris Controller
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Installing a Netris Controller

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. So you can access the console, and nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface.

+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

In this example I am running my Netris controller on an AWS hosted virtual machine (EC2) which has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

I’m using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my EC2 : 54.219.211.71.

+_images/cloudflare-dns-record.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+

To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-hostname ” will instruct the installer to generate a Let’s Encrypt SSL certificate for the provided domain name. That’s why it is important to create the DNS record before this step.

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com
+
+
+

Once installation process is finished you will be able to access your newly installed Netris Controller web consonle using netris/newNet0ps credentials.

+

Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller.

+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/introduction.html b/en/3.4/introduction.html new file mode 100644 index 0000000000..bbdaaf9055 --- /dev/null +++ b/en/3.4/introduction.html @@ -0,0 +1,715 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Introduction to Netris — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Introduction to Netris
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Introduction to Netris

+

Netris is an automatic netops software for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network’s health and either applies software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation.

+_images/netris-architecture.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/inventory-profiles.html b/en/3.4/inventory-profiles.html new file mode 100644 index 0000000000..7610b84c83 --- /dev/null +++ b/en/3.4/inventory-profiles.html @@ -0,0 +1,738 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Inventory Profiles — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +
+

Inventory Profiles

+

Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/softgate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except netris-defined and user-defined custom flows. Generated rules include:

+
    +
  • SSH from user defined subnets

  • +
  • NTP from user defined ntp services

  • +
  • DNS from user defined DNS servers

  • +
  • Custom user defined rules

  • +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + +
Inventory Profile Fields

Name

Profile name

Description

Free text description

Allow SSH from IPv4

List of IPv4 subnets allowed to ssh (one address per line)

Allow SSH from IPv6

List of IPv6 subnets allowed to ssh (one address per line)

Timezone

Devices using this inventory profile will adjust their system time to the selected timezone.

NTP servers

List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate.

DNS servers

List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate.

+

Example: In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup).

+Adding an inventory profile +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/ipam.html b/en/3.4/ipam.html new file mode 100644 index 0000000000..54816ef765 --- /dev/null +++ b/en/3.4/ipam.html @@ -0,0 +1,798 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + IP Address Management — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • IP Address Management
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

IP Address Management

+

Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting.

+

Purpose: +Users define specific roles(purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-net, NAT, etc…

+
+

Allocations and Subnets

+

There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets.

+IPAM Tree View +

IPAM Tree View

+
+
+
+

Add an Allocation

+
    +
  1. Navigate to Net→IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Allocation from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + +
Allocation Fields

Name

Unique name for current allocation.

Prefix

Unique prefix for allocation, must not overlap with other allocations.

Tenant

Owner of the allocation.

+Add a New IP Allocation +

Add Allocation Window

+
+
+
+

Add a Subnet

+
    +
  1. Navigate to Net→IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Subnet from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + + + + +
Subnet fields

Name

Unique name for current subnet.

Prefix

Unique prefix for subnet, ust be included in one of allocations.

Tenant

Owner of the subnet.

Purpose

This field describes for what kind of services the current subnet can be used. It can have the following values:

+
+
    +
  • common - ordinary subnet, can be used in v-nets and ROH.

  • +
  • loopback - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates).

  • +
  • management - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates).

  • +
  • load-balancer - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience.

  • +
  • nat - hosts of this subnet or subnet itself can be used to define NAT services.

  • +
  • inactive - can’t be used in any services, useful for reserving/documenting prefixes for future use.

  • +
+
+
+Add a New Subnet +

Add Subnet Window

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/kubernetes-integration.html b/en/3.4/kubernetes-integration.html new file mode 100644 index 0000000000..c2d0bdaec6 --- /dev/null +++ b/en/3.4/kubernetes-integration.html @@ -0,0 +1,1079 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Kubernetes Integration — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Kubernetes Integration
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Kubernetes Integration

+

Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris-Kubernetes integration is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters.

+
+

Install Netris Operator

+

Integration between the Netris Controller and the Kubernetes API is completed by installing the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart:

+ +
+

Regular Manifest Method

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='http://**your-netris-controller-ip-or-host**' \
+--from-literal=login='**your-netris-admin-username**' --from-literal=password='**your-netris-admin-password**'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+
+

Using Type ‘LoadBalancer’

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox pod provisioning +

After provisioning has finished, inspect the service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.202   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9898   active   50.117.59.202   9898/TCP   US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9898   active   50.117.59.202   9898/TCP   US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.203   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.203
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+View L4 LB instances +
+
+

V-Net Custom Resource

+

You can also create Netris V-Nets (L2 segments) via Kubernetes with a simple manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+
+
+

BGP Custom Resource

+

You can create BGP peers via Kubernetes manifests:

+
    +
  1. Create a yaml file:

  2. +
+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1092
+  localIP: 50.117.59.118/30
+  remoteIP: 50.117.59.117/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.192/28 le 32
+EOF
+
+
+
    +
  1. Apply the manifest file:

  2. +
+
kubectl apply -f isp2-customer.yaml
+
+
+
    +
  1. Check created BGP:

  2. +
+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.118/30   50.117.59.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.118/30   50.117.59.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Calico CNI Integration

+

Netris Operator can integrate with Calico CNI. This annotation will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.118/30   50.117.59.117/30    7m59s
+sandbox9-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox9-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox9-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.118/30   50.117.59.117/30    8m41s
+sandbox9-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox9-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox9-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.202
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/l3-load-balancer.html b/en/3.4/l3-load-balancer.html new file mode 100644 index 0000000000..d49074780a --- /dev/null +++ b/en/3.4/l3-load-balancer.html @@ -0,0 +1,732 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + L3 Load Balancer (Anycast LB) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • L3 Load Balancer (Anycast LB)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

L3 Load Balancer (Anycast LB)

+

L3 (Anycast) load balancer leverages ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks.

+

ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server.

+

End-user traffic should be destined to the anycast IP address. The switch fabric uses ECMP to load balance the traffic towards every server, and will hash sessions based on IP/Protocol/Port such that TCP sessions will exist between the given end-user and server pair for the lifetime of the session. Optional health checks are used to identify application failures and reroute traffic in the case of a server outage.

+
+

Creating an L3 Load Balancer

+

To configure L3 (Anycast) load balancing:

+
    +
  1. Navigate to Services→Instances (ROH) and locate an existing ROH instance

  2. +
  3. Click the ellipses and then the Edit button

  4. +
  5. Select an extra IPv4 address from the select box at the bottom, and check the Anycast option.

  6. +
  7. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances.

  8. +
+Add a L3 LB +

Example: Adding an Anycast IPv4 address

+List L3 LBs +

Example: Under Services→Load Balancer, you can find the listing of L3 (Anycast) Load Balancers, service statuses, and you can add/remove more ROH instances and/or health checks.

+List L3 LB Details +

Screenshot: L3 (Anycast) Load Balancer Detail

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/l4-load-balancer.html b/en/3.4/l4-load-balancer.html new file mode 100644 index 0000000000..b0d8525d21 --- /dev/null +++ b/en/3.4/l4-load-balancer.html @@ -0,0 +1,765 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + L4 Load Balancer (L4LB) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • L4 Load Balancer (L4LB)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

L4 Load Balancer (L4LB)

+

Netris L4 Load Balancer (L4LB) leverages SoftGate(Linux router) nodes to provide Layer-4 load balancing services, including on-demand cloud load balancing with native integration with Kubernetes.

+
+

Enabling L4LB service

+

L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer).

+

The IP address pool for L4LB can be defined in the Net→IPAM section by adding an Allocation and setting the purpose field to ‘load-balancer’. You can define multiple IP pools for L4LB at any given site. See the below example.

+

Example: Adding a load-balancer IP pool assignment.

+Add an IP Allocation +

Screenshot: Listing of Net→IPAM after adding a load-balancer assignment

+List IP Subnets +
+
+

Consuming L4LB service

+

This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section.

+

Click +add under Services→L4 Load Balancer to request an L4LB service.

+

Add new L4 Load Balancer fields are described below:

+

General fields

+
    +
  • Name - Unique name.

  • +
  • Protocol - TCP or UDP.

  • +
  • Tenant - Requestor Tenant should have access to the backend IP space.

  • +
  • Site - Site where L4LB service is being requested for. Backends should belong on this site.

  • +
  • State - Administrative state.

  • +
+

Frontend

+
    +
  • Address - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses.

  • +
  • Port - TCP or UDP port to be exposed.

  • +
+

Health-check

+
    +
  • Type - Probe backends on service availability.

    +
      +
    • None - load balance unconditionally.

    • +
    • TCP - probe backend service availability through TCP connect checks.

    • +
    • HTTP - probe backend service availability through HTTP GET checks.

    • +
    +
  • +
  • Timeout(ms) - Probe timeout in milliseconds.

  • +
  • Request path - HTTP request path.

  • +
+

Backend

+
    +
  • +Add - add a backend host.

  • +
  • Address - IP address of the backend host.

  • +
  • Port - Service port on the backend host.

  • +
  • Enabled - Administrative state of particular backend.

  • +
+Request an L4 Load Balancer +

Example: Requesting an L4 Load Balancer service.

+List L4 Load Balancers +

Example: Listing of L4 Load Balancer services

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/netris-architecture.html b/en/3.4/netris-architecture.html new file mode 100644 index 0000000000..5b97bddfdd --- /dev/null +++ b/en/3.4/netris-architecture.html @@ -0,0 +1,744 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Netris Architecture — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Architecture

+

A Netris system is composed of 4 elements:

+
    +
  • Netris Controller

  • +
  • Netris Switch Agent

  • +
  • Netris SoftGate

  • +
  • Customer Application Servers

  • +
+
+

Netris Controller

+

Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. The Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud.

+

Diagram: High level Netris architecture

+_images/netris_controller_diagram.png +
    +
  • Controller HA We highly recommend running more than one copy of the controller for database replication.

  • +
  • Multiple sites Netris is designed to operate multiple sites with just a single controller with HA.

  • +
  • What if the controller is unreachable. Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers core operations will not be affected.

  • +
+
+
+

Netris Switch Agent

+

Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet.

+
+
+

Netris SoftGate

+

Netris SoftGate is automatic configuration software and reference architecture for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), and site-to-site VPN function on a regular x86 server with a SmartNIC card.

+

Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2.

+

The server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities.

+

Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels.

+

Diagram: Netris SoftGate high level architecture

+_images/softgate_diagram.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/network-policies.html b/en/3.4/network-policies.html new file mode 100644 index 0000000000..f500279c66 --- /dev/null +++ b/en/3.4/network-policies.html @@ -0,0 +1,1083 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Basic BGP — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Basic BGP

+

BGP neighbors can be declared in the Net→E-BGP section. Netris will automatically generate and program the network configuration to meet the requirements.

+
+

Adding BGP Peers

+
    +
  1. Navigate to Net→E-BGP in the web UI

  2. +
  3. Click the Add button

  4. +
  5. Fill in the fields as described in the table below

  6. +
  7. Click the Add button

  8. +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BGP Peer Fields

Name

User assigned name of BGP session

Description

Free description

Site

Selects the site (data center) where this BGP session should be terminated on.

Softgate

Only if SoftGate nodes are in use. Define on which node BGP session should be terminated on.

Neighbor AS

Autonomous System number of the remote side. (Local AS is defined at Net→Sites section)

Terminate on switch

Typically used for setups without SoftGate. For connecting with upstream routers. Instructs the system to terminate the BGP session directly on the switch.

Switch port

Switch Port for the physical cable to the BGP neighbor (any port on the fabric). Optionally can bind to a V-NET service. Typically used for peering with IXPs or systems like GGC (Google Global Cache).

VLAN ID

Optionally tag with a VLAN ID (usually untagged)

IP Version

IPv4 or IPv6

Local IP

BGP peering IP address on Netris controlled side

Remote IP

BGP peering IP address on the remote end

State

Administrative state (Enabled/Disabled)

Advanced

Advanced policy settings are described in the next section

+

Example: Declare a basic BGP neighbor

+_images/add-bgp-basic.png +

Example2: Declare BGP neighbor terminated on V-Net. Netris will automatically configure BGP session on the switch closest to the remote IP.

+_images/add-bgp-basic-2.png +
+
+
+

Advanced BGP

+

BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies.

+

Click Advanced to expand the BGP neighbor add/edit window.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
BGP Peer Fields - Advanced

Neighbor address

IP address of the neighbor when peering with the loopback IP address instead of the interface IP address (aka Multihop).

Update source

When Multihop BGP peering is used it allows the operator to choose one of the loopback IP addresses of the SoftGate node as a BGP speaker source IP address.

BGP password

Password for the BGP session

Allowas-in

Define the number of allowed occurrences of the self AS number in the received BGP NLRI to consider it valid (normally 0).

Default Originate

Originate default route to the current neighbor

Prefix Inbound Max

Drop the BGP session if the number of received prefixes exceeds this max limit. For switch termination maximum allowed is 1000 prefixes and SoftGate termination can handle up to one million prefixes.

Inbound Route-Map

Apply BGP policies described in a route-map for inbound BGP updates

Outbound Route-Map

Apply BGP policies described in a route-map for outbound BGP updates

Local Preference

Set local preference for all inbound routes for the current neighbor

Weight

Set weight for all inbound routes for the current neighbor

Prepend Inbound (times)

How many times to prepend self AS number for inbound routes

Prepend Outbound (times)

How many times to prepend self AS number for outbound routes

Prefix List Inbound

List of IP addresses prefixes to permit or deny inbound

Prefix List Outbound

List of IP addresses prefixes to permit or deny outbound

Send BGP Community

List of BGP communities to send to the current neighbor

+
+
+

BGP Objects

+
+
Under Net→E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy.
+
Supported objects include:
+
+
    +
  • IPv4 Prefix

  • +
  • IPv6 Prefix

  • +
  • AS-PATH

  • +
  • Community

  • +
  • Extended Community

  • +
  • Large Community

  • +
+
+

IPv4 Prefix

+
+
Rules defined one per line.
+
Each line in IPv4 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv4 prefix (mandatory).

  • +
  • Length - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv4 Prefix list.

+_images/IPv4-Prefix.png +
+
+

IPv6 Prefix

+
+
Rules defined one per line.
+
Each line in IPv6 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv6 prefix (mandatory).

  • +
  • Keyword - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv6 Prefix list.

+_images/IPv6-Prefix.png +
+
+

Community

+
+
Community field has two parts:
+
+
    +
  • Action - Possible values: permit or deny (mandatory).

  • +
  • Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive).

  • +
+

Example: Creating community.

+_images/community.png +
+
+
+
+

BGP route-maps

+
+
Under the Net→E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound.
+
Description of route-map fields:
+
+
    +
  • Sequence Number - Automatically assigned a sequence number. Drag and move sequences to organize the order.

  • +
  • Description - Free description.

  • +
  • Policy - Permit or deny the routes which match below all match clauses within the current sequence.

  • +
  • Match - Rules for route matching.

    +
      +
    • Type - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag.

    • +
    • Object - Select an object from the list.

    • +
    +
  • +
  • Action - Action when all match clauses are met.

    +
      +
    • Action type - Define whether to manipulate a particular BGP attribute or go to another sequence.

    • +
    • Attribute - The attribute to be manipulated.

    • +
    • Value - New attribute value.

    • +
    +
  • +
+

Example: route-map

+_images/route-map.png +
+
+
+
+

Static Routing

+

Located under Net→Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. +We recommend using the Routes only if BGP is not supported by the remote end.

+
+
Typical use cases for static routing:
+
+
    +
  • To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported.

  • +
  • Temporary interconnection with the old network for a migration.

  • +
  • Routing a subnet behind a VM hypervisor machine for an internal VM network.

  • +
  • Specifically routing traffic destined to a particular prefix through an out-of-band management network.

  • +
+
+
Add new static route fields description:
+
+
    +
  • Prefix - Route destination to match.

  • +
  • Next-Hop - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network.

  • +
  • Description - Free description.

  • +
  • Site - Site where Route belongs.

  • +
  • State - Administrative (enable/disable) state of the Route.

  • +
  • Apply to - Limit the scope to particular units. It’s typically used for Null routes.

  • +
+

Example: Default route pointing to a Next-Hop that belongs to one of V-NETs.

+_images/defaultroute.png +

Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network.

+_images/static_route.png +

Screenshot: This Shows that my back route is actually applied on leaf1 and spine1.

+_images/leaf1_spine1.png +
+
+
+

NAT

+

Netris SoftGate nodes are required to support NAT (Network Address Translation).

+
+

Enabling NAT

+

To enable NAT for a given site, you first need to create a subnet with NAT purpose in the IPAM section. NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need.

+
    +
  1. Allocate a public IP subnet for NAT under Net→IPAM.

  2. +
+

Example: Adding an IP allocation under Net→Subnets.

+_images/IP-allocation.png +
    +
  1. Attach NAT IP addresses and/or NAT IP Pools to just one SoftGate node. Other SoftGate Nodes on the same site will automatically add the same NAT IP/Pool resources for proper consistency and high availability.

  2. +
+

Example: Adding NAT IP addresses and NAT IP Address Pools to a SoftGate node.

+_images/NATIP-address.png +
+
+

Defining NAT rules

+

NAT rules are defined under Net→NAT.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NAT Rule Fields

Name

Unique name

State

State of rule (enabled or disabled)

Site

Site to apply the rule

Action

SNAT - Replace the source IP address with specified NAT IP along with port overloading +DNAT - Replace the destination IP address and/or destination port with specified NAT IP +ACCEPT - Silently forward, typically used to add an exclusion to broader SNAT or DNAT rule +MASQUERADE - Replace the source IP address with the IP address of the exit interface

Protocol

All - Match any IP protocol +TCP - Match TCP traffic and ports +UDP - Match UDP traffic and ports +ICMP - Match ICMP traffic

Source

Address - Source IP address to match +Port - Source ports range to match with this value (TCP/UDP)

Destination

Address - Destination IP address to match. In the case of DNAT it should be one of the predefined NAT IP addresses +Port - For DNAT only, to match a single destination port +Ports - For SNAT/ACCEPT only. Destination ports range to match with this value (TCP/UDP)

DNAT to IP

The global IP address for SNAT to be visible on the Public Internet. The internal IP address for DNAT to replace the original destination address with

DNAT to Port

The Port to which destination Port of the packet should be NAT’d

Status

Administrative state (enable/disable)

Comment

Free optional comment

+

Example: SNAT all hosts on 10.0.0.0/8 to the Internet using 198.51.100.65 as a global IP.

+_images/globalIP.png +

Example: Port forwarding. DNAT the traffic destined to 198.51.100.66:80 to be forwarded to the host 10.0.4.10 on port tcp/1080.

+_images/Port-Forwarding.png +
+
+
+
+

SiteMesh

+

SiteMesh is a Netris service for site-to-site interconnects over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site.

+

Edit Net->Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below.

+
    +
  • Disabled - Do not participate in SiteMesh.

  • +
  • Hub - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites)

  • +
  • Spoke - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites)

  • +
  • Dynamic Spoke - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites)

  • +
+

Screenshot: Site Mesh parameter editing a Site under Net→Sites.

+_images/Site_Mesh.png +

You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge.

+_images/SiteMesh_modes.png +

Check the Net→Site Mesh section for the listing of tunnel statuses.

+

Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh)

+_images/SiteMesh_listing.png +
+
+
+

Looking Glass

+

The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Net→Looking Glass then selecting the device from the top-left dropdown menu.

+

Looking Glass controls described for IPv4/IPv6 protocol families.

+
    +
  • BGP Summary - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes.

  • +
  • BGP Route - Lookup the BGP table (RIB) for the given address.

  • +
  • Route - Lookup switch routing table for the given address.

  • +
  • Traceroute - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address.

  • +
  • Ping - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address.

  • +
+

Example: Spine1: listing BGP neighbors and number of received prefixes.

+_images/Spine1.png +

Example: BGP Route - looking up my leaf1 switch’s loopback address from spine1’s perspective. Spine1 is load balancing between two available paths.

+_images/BGP_route.png +

Example: Ping.

+_images/ping.png +
+
Looking Glass controls described for the EVPN family.
+
+
    +
  • BGP Summary - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received.

  • +
  • VNI - List VNIs learned.

  • +
  • BGP EVPN - List detailed EVPN routing information optionally for the given route distinguisher.

  • +
  • MAC table - List MAC address table for the given VNI.

  • +
+

Example: Listing of adjacent BGP neighbors and number of EVPN prefixes received.

+_images/BGP_neighbors_listing.png +

Example: Listing MAC addresses on VNI 2.

+_images/MAC_listing.png +

Example: EVPN routing information listing for a specified route distinguisher.

+_images/EVPN_routing.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/objects.inv b/en/3.4/objects.inv new file mode 100644 index 0000000000..9cdab883b2 Binary files /dev/null and b/en/3.4/objects.inv differ diff --git a/en/3.4/reference-designs.html b/en/3.4/reference-designs.html new file mode 100644 index 0000000000..1484fe4584 --- /dev/null +++ b/en/3.4/reference-designs.html @@ -0,0 +1,707 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Network Reference Designs — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Network Reference Designs
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Network Reference Designs

+

Netris can support any type of standard network design.

+

The majority of designs fall into one of four patterns:

+
    +
  1. Unmanaged Switch

  2. +
  3. Single L2 Domain

  4. +
  5. Collapsed Core

  6. +
  7. Data Center Leaf/Spine Designs (Modern High Throughput Redundant Design)

  8. +
  9. Data Center Core/Distribution/Access (Legacy High Throughput Redundant Design)

  10. +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/release-notes.html b/en/3.4/release-notes.html new file mode 100644 index 0000000000..03629243e7 --- /dev/null +++ b/en/3.4/release-notes.html @@ -0,0 +1,725 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Release notes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Release notes

+
    +
  • DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGate performance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on.

  • +
  • L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, we now support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates with Kubernetes providing cloud-like load balancer service (type: load-balancer).

  • +
  • Kubenet​ - a network service purpose-built for Kubernetes cluster nodes. Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking with modern physical networking.

  • +
  • API logs​ - Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type.

  • +
  • Site Mesh​ - a Netris service for automatically configuring site-to-site interconnect over the public Internet. Site Mesh supports configuration for WireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a few clicks, services in one site get connectivity to services in other sites over a mesh of WireGuard tunnels.

  • +
  • Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loop cable. Removed the need for IP address reservation for V-NET, switching entirely to the anycast default gateway.

  • +
  • Controller distributions​ - Netris controller, is now available in three deployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application. 3) Managed/Hosted in the cloud.

  • +
  • Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes.

  • +
  • Switch/SoftGate agents​ - New installer with easy initial config tool. Support for IP and FQDN as a controller address. Authentication key.

  • +
  • GUI​ - Improved Net→Topology section, becoming the main and required place for defining the network topology. All sections got a column organizer, so every user can order and hide/show columns to their comfort.

  • +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/roh.html b/en/3.4/roh.html new file mode 100644 index 0000000000..3b3e80d827 --- /dev/null +++ b/en/3.4/roh.html @@ -0,0 +1,753 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + ROH (Routing on the Host) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • ROH (Routing on the Host)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

ROH (Routing on the Host)

+

To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly on their servers. This is commonly known as ROH (Routing on the Host).

+

In ROH architectures, servers use a routing daemon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon/suite is FRR.

+

Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead an IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. This is a modern and optimal design, leveraging Layer-3 networking from the fabric to the servers.

+

By using only Layer-3 interfaces, Layer-2 protocols such as Spanning Tree (STP) can be minized and the reliability of the network increases.

+

The ROH architecture that is configured by Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). For each instance of ROH, you’ll need to create a ROH entry in Netris Controller.

+
+

Adding ROH Hosts

+
    +
  1. Navigate in the Netris UI to Services→Instances (ROH)

  2. +
  3. Click the Add button

  4. +
  5. Fill out the form based on the fields in the table below.

  6. +
  7. Click the Add button

  8. +
+

Description of ROH instance fields:

+
    +
  • Name - Unique name for the ROH instance

  • +
  • Site - Site where the current ROH instance belongs

  • +
  • Type - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor.

  • +
  • ROH Routing Profile - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances

    +
      +
    • Default route only (most common design) - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch

    • +
    • Default + Aggregate - Will add prefixes of defined assignments + “Default” profile

    • +
    • Full table - Will advertise all prefixes available in the routing table of the connected switch

    • +
    • Inherit - Will inherit policy from site objects defined under Net→Sites

    • +
    +
  • +
  • Legacy Mode - Switch from default zero-config mode to using /30 IP addresses. Used for MSFT Windows Servers or other OS that doesn’t support FRR.

  • +
  • +Port - Physical Switch Ports anywhere on the network.

  • +
  • +IPv4 - IPv4 addresses for the loopback interface.

  • +
  • +Inbound Prefix List - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks.

  • +
+
+

Tip

+

Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+
+ROH Instances +

Example: Adding an ROH instance. (Yes, you can use A.B.C.0/32 and A.B.C.255/32)

+ROH Listings +

Expanded view of ROH listing. BGP sessions are up, and the expected IP is in fact received from the actual ROH server. Traffic stats are available per port.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox1/configurations.html b/en/3.4/sandbox/Sandbox1/configurations.html new file mode 100644 index 0000000000..b82de1195c --- /dev/null +++ b/en/3.4/sandbox/Sandbox1/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox1.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc1::1 (from the “2607:f358:11:ffc1::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.24 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox1/creating-services.html b/en/3.4/sandbox/Sandbox1/creating-services.html new file mode 100644 index 0000000000..f66b66ca71 --- /dev/null +++ b/en/3.4/sandbox/Sandbox1/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1012.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.0/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.1)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.4/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.4/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.4/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.8/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.24 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox1/index.html b/en/3.4/sandbox/Sandbox1/index.html new file mode 100644 index 0000000000..0c331a151f --- /dev/null +++ b/en/3.4/sandbox/Sandbox1/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox1 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox1/onprem-k8s.html b/en/3.4/sandbox/Sandbox1/onprem-k8s.html new file mode 100644 index 0000000000..044e8df69f --- /dev/null +++ b/en/3.4/sandbox/Sandbox1/onprem-k8s.html @@ -0,0 +1,1207 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox1.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox1.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.13   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.13:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.13
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.13   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.13   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.13   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.13   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.14   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.14
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.14
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.14
+
+SRV05-NYC
+curl 45.38.161.14
+
+SRV05-NYC
+curl 45.38.161.14
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1012
+  localIP: 45.38.161.22/30
+  remoteIP: 45.38.161.21/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.0/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.22/30   45.38.161.21/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.22/30   45.38.161.21/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.22/30   45.38.161.21/30    7m59s
+sandbox1-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox1-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox1-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.22/30   45.38.161.21/30    8m41s
+sandbox1-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox1-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox1-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.13
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox1/sandbox-info.html b/en/3.4/sandbox/Sandbox1/sandbox-info.html new file mode 100644 index 0000000000..00c8617290 --- /dev/null +++ b/en/3.4/sandbox/Sandbox1/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox1.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.24 -p 30061
+srv02-nyc: ssh demo@166.88.17.24 -p 30062
+srv03-nyc: ssh demo@166.88.17.24 -p 30063
+srv04-nyc: ssh demo@166.88.17.24 -p 30064
+srv05-nyc: ssh demo@166.88.17.24 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1011
+Local Address:                  45.38.161.18/30
+Remote Address:                 45.38.161.17/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.0/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1011
+Local Address:                  2607:f358:11:ffc0::3/127
+Remote Address:                 2607:f358:11:ffc0::2/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc1::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1012
+Local Address:                  45.38.161.22/30
+Remote Address:                 45.38.161.21/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.0/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.0/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.0/30
+|___ NAT Subnet:               45.38.161.4/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.8/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.12/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc1::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc1::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox10/configurations.html b/en/3.4/sandbox/Sandbox10/configurations.html new file mode 100644 index 0000000000..57686b814a --- /dev/null +++ b/en/3.4/sandbox/Sandbox10/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox10.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffca::1 (from the “2607:f358:11:ffca::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.19 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox10/creating-services.html b/en/3.4/sandbox/Sandbox10/creating-services.html new file mode 100644 index 0000000000..a385ad9da4 --- /dev/null +++ b/en/3.4/sandbox/Sandbox10/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1102.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.208/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.209)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.212/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.212/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.212/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.216/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.19 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox10/index.html b/en/3.4/sandbox/Sandbox10/index.html new file mode 100644 index 0000000000..39442a3329 --- /dev/null +++ b/en/3.4/sandbox/Sandbox10/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox10 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox10/onprem-k8s.html b/en/3.4/sandbox/Sandbox10/onprem-k8s.html new file mode 100644 index 0000000000..70084157f4 --- /dev/null +++ b/en/3.4/sandbox/Sandbox10/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox10.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox10.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.221   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.221:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.221
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.221   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.221   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.221   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.221   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.222   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.222
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.222
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.222
+
+SRV05-NYC
+curl 50.117.59.222
+
+SRV05-NYC
+curl 50.117.59.222
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1102
+  localIP: 50.117.59.126/30
+  remoteIP: 50.117.59.125/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.208/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.126/30   50.117.59.125/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.126/30   50.117.59.125/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.126/30   50.117.59.125/30    7m59s
+sandbox10-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox10-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox10-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.126/30   50.117.59.125/30    8m41s
+sandbox10-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox10-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox10-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.221
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox10/sandbox-info.html b/en/3.4/sandbox/Sandbox10/sandbox-info.html new file mode 100644 index 0000000000..2a20f4c18e --- /dev/null +++ b/en/3.4/sandbox/Sandbox10/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox10.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.19 -p 30061
+srv02-nyc: ssh demo@166.88.17.19 -p 30062
+srv03-nyc: ssh demo@166.88.17.19 -p 30063
+srv04-nyc: ssh demo@166.88.17.19 -p 30064
+srv05-nyc: ssh demo@166.88.17.19 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1101
+Local Address:                  50.117.59.122/30
+Remote Address:                 50.117.59.121/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.208/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1101
+Local Address:                  2607:f358:11:ffc0::15/127
+Remote Address:                 2607:f358:11:ffc0::14/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffca::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1102
+Local Address:                  50.117.59.126/30
+Remote Address:                 50.117.59.125/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.208/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.208/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.208/30
+|___ NAT Subnet:               50.117.59.212/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.216/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.220/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffca::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffca::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox11/configurations.html b/en/3.4/sandbox/Sandbox11/configurations.html new file mode 100644 index 0000000000..52a24b5433 --- /dev/null +++ b/en/3.4/sandbox/Sandbox11/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox11.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcb::1 (from the “2607:f358:11:ffcb::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.82 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox11/creating-services.html b/en/3.4/sandbox/Sandbox11/creating-services.html new file mode 100644 index 0000000000..3f380c58cb --- /dev/null +++ b/en/3.4/sandbox/Sandbox11/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.82 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1112.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.118.

    6. +
    7. In the Remote IP field, type in 45.38.161.117.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.96/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.97)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.82 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.100/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.100/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.100/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.104/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.104”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.104” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.104) into the browser’s address bar or simply visit http://45.38.161.104/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.104 (name_45.38.161.104)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.104 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.82 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox11/index.html b/en/3.4/sandbox/Sandbox11/index.html new file mode 100644 index 0000000000..c3b65b3af9 --- /dev/null +++ b/en/3.4/sandbox/Sandbox11/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox11 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox11/onprem-k8s.html b/en/3.4/sandbox/Sandbox11/onprem-k8s.html new file mode 100644 index 0000000000..6148d4b4c2 --- /dev/null +++ b/en/3.4/sandbox/Sandbox11/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox11.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox11.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.109   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.109:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.109
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.109   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.109   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.109   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.109   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.110   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.110
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.110
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.110
+
+SRV05-NYC
+curl 45.38.161.110
+
+SRV05-NYC
+curl 45.38.161.110
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1112
+  localIP: 45.38.161.118/30
+  remoteIP: 45.38.161.117/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.96/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.118/30   45.38.161.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.118/30   45.38.161.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.118/30   45.38.161.117/30    7m59s
+sandbox11-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox11-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox11-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.118/30   45.38.161.117/30    8m41s
+sandbox11-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox11-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox11-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.109
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox11/sandbox-info.html b/en/3.4/sandbox/Sandbox11/sandbox-info.html new file mode 100644 index 0000000000..2a6406f70d --- /dev/null +++ b/en/3.4/sandbox/Sandbox11/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox11.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@50.117.27.82 -p 30061
+srv02-nyc: ssh demo@50.117.27.82 -p 30062
+srv03-nyc: ssh demo@50.117.27.82 -p 30063
+srv04-nyc: ssh demo@50.117.27.82 -p 30064
+srv05-nyc: ssh demo@50.117.27.82 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1111
+Local Address:                  45.38.161.114/30
+Remote Address:                 45.38.161.113/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.96/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1111
+Local Address:                  2607:f358:11:ffc0::17/127
+Remote Address:                 2607:f358:11:ffc0::16/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcb::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1112
+Local Address:                  45.38.161.118/30
+Remote Address:                 45.38.161.117/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.96/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.96/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.96/30
+|___ NAT Subnet:               45.38.161.100/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.104/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.108/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcb::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcb::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox12/configurations.html b/en/3.4/sandbox/Sandbox12/configurations.html new file mode 100644 index 0000000000..b643f872e1 --- /dev/null +++ b/en/3.4/sandbox/Sandbox12/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox12.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcc::1 (from the “2607:f358:11:ffcc::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.83 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox12/creating-services.html b/en/3.4/sandbox/Sandbox12/creating-services.html new file mode 100644 index 0000000000..da71d4b937 --- /dev/null +++ b/en/3.4/sandbox/Sandbox12/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.83 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1122.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.126.

    6. +
    7. In the Remote IP field, type in 45.38.161.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.128/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.129)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.83 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.132/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.132/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.132/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.136/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.136”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.136” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.136) into the browser’s address bar or simply visit http://45.38.161.136/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.136 (name_45.38.161.136)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.136 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.83 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox12.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox12/index.html b/en/3.4/sandbox/Sandbox12/index.html new file mode 100644 index 0000000000..c9c24ff4a4 --- /dev/null +++ b/en/3.4/sandbox/Sandbox12/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox12 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox12/onprem-k8s.html b/en/3.4/sandbox/Sandbox12/onprem-k8s.html new file mode 100644 index 0000000000..929758a4a2 --- /dev/null +++ b/en/3.4/sandbox/Sandbox12/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox12.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox12.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.141   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.141:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.141
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.141   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.141   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.141   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.141   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.142   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.142
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.142
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.142
+
+SRV05-NYC
+curl 45.38.161.142
+
+SRV05-NYC
+curl 45.38.161.142
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1122
+  localIP: 45.38.161.126/30
+  remoteIP: 45.38.161.125/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.128/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.126/30   45.38.161.125/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.126/30   45.38.161.125/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.126/30   45.38.161.125/30    7m59s
+sandbox12-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox12-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox12-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.126/30   45.38.161.125/30    8m41s
+sandbox12-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox12-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox12-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.141
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox12/sandbox-info.html b/en/3.4/sandbox/Sandbox12/sandbox-info.html new file mode 100644 index 0000000000..154d928016 --- /dev/null +++ b/en/3.4/sandbox/Sandbox12/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox12.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@50.117.27.83 -p 30061
+srv02-nyc: ssh demo@50.117.27.83 -p 30062
+srv03-nyc: ssh demo@50.117.27.83 -p 30063
+srv04-nyc: ssh demo@50.117.27.83 -p 30064
+srv05-nyc: ssh demo@50.117.27.83 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1121
+Local Address:                  45.38.161.122/30
+Remote Address:                 45.38.161.121/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.128/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1121
+Local Address:                  2607:f358:11:ffc0::19/127
+Remote Address:                 2607:f358:11:ffc0::18/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcc::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1122
+Local Address:                  45.38.161.126/30
+Remote Address:                 45.38.161.125/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.128/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.128/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.128/30
+|___ NAT Subnet:               45.38.161.132/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.136/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.140/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcc::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcc::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox13/configurations.html b/en/3.4/sandbox/Sandbox13/configurations.html new file mode 100644 index 0000000000..d09e9589f4 --- /dev/null +++ b/en/3.4/sandbox/Sandbox13/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox13.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcd::1 (from the “2607:f358:11:ffcd::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.84 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox13/creating-services.html b/en/3.4/sandbox/Sandbox13/creating-services.html new file mode 100644 index 0000000000..af2ab33f2d --- /dev/null +++ b/en/3.4/sandbox/Sandbox13/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.84 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1132.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.166.

    6. +
    7. In the Remote IP field, type in 45.38.161.165.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.144/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.145)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.84 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.148/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.148/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.148/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.152/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.152”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.152” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.152) into the browser’s address bar or simply visit http://45.38.161.152/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.152 (name_45.38.161.152)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.152 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.84 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox13/index.html b/en/3.4/sandbox/Sandbox13/index.html new file mode 100644 index 0000000000..d3e61beaaa --- /dev/null +++ b/en/3.4/sandbox/Sandbox13/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox13 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox13/onprem-k8s.html b/en/3.4/sandbox/Sandbox13/onprem-k8s.html new file mode 100644 index 0000000000..2acadcf4f8 --- /dev/null +++ b/en/3.4/sandbox/Sandbox13/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox13.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox13.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.157   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.157:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.157
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.157   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.157   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.157   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.157   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.158   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.158
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.158
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.158
+
+SRV05-NYC
+curl 45.38.161.158
+
+SRV05-NYC
+curl 45.38.161.158
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1132
+  localIP: 45.38.161.166/30
+  remoteIP: 45.38.161.165/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.144/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.166/30   45.38.161.165/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.166/30   45.38.161.165/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.166/30   45.38.161.165/30    7m59s
+sandbox13-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox13-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox13-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.166/30   45.38.161.165/30    8m41s
+sandbox13-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox13-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox13-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.157
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox13/sandbox-info.html b/en/3.4/sandbox/Sandbox13/sandbox-info.html new file mode 100644 index 0000000000..c11fce84f8 --- /dev/null +++ b/en/3.4/sandbox/Sandbox13/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox13.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@50.117.27.84 -p 30061
+srv02-nyc: ssh demo@50.117.27.84 -p 30062
+srv03-nyc: ssh demo@50.117.27.84 -p 30063
+srv04-nyc: ssh demo@50.117.27.84 -p 30064
+srv05-nyc: ssh demo@50.117.27.84 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1131
+Local Address:                  45.38.161.162/30
+Remote Address:                 45.38.161.161/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.144/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1131
+Local Address:                  2607:f358:11:ffc0::1b/127
+Remote Address:                 2607:f358:11:ffc0::1a/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcd::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1132
+Local Address:                  45.38.161.166/30
+Remote Address:                 45.38.161.165/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.144/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.144/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.144/30
+|___ NAT Subnet:               45.38.161.148/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.152/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.156/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcd::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcd::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox14/configurations.html b/en/3.4/sandbox/Sandbox14/configurations.html new file mode 100644 index 0000000000..8393b8531e --- /dev/null +++ b/en/3.4/sandbox/Sandbox14/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox14.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffce::1 (from the “2607:f358:11:ffce::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.85 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox14/creating-services.html b/en/3.4/sandbox/Sandbox14/creating-services.html new file mode 100644 index 0000000000..9f10a90779 --- /dev/null +++ b/en/3.4/sandbox/Sandbox14/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.85 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1142.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.174.

    6. +
    7. In the Remote IP field, type in 45.38.161.173.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.176/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.177)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.85 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.180/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.180/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.180/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.184/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.184”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.184” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.184) into the browser’s address bar or simply visit http://45.38.161.184/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.184 (name_45.38.161.184)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.184 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.85 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox14.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox14/index.html b/en/3.4/sandbox/Sandbox14/index.html new file mode 100644 index 0000000000..0233894af5 --- /dev/null +++ b/en/3.4/sandbox/Sandbox14/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox14 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox14/onprem-k8s.html b/en/3.4/sandbox/Sandbox14/onprem-k8s.html new file mode 100644 index 0000000000..cf58d9b0d5 --- /dev/null +++ b/en/3.4/sandbox/Sandbox14/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox15.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox15.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.189   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.189:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.189
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.189   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.189   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.189   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.189   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.190   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.190
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.190
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.190
+
+SRV05-NYC
+curl 45.38.161.190
+
+SRV05-NYC
+curl 45.38.161.190
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1142
+  localIP: 45.38.161.174/30
+  remoteIP: 45.38.161.173/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.176/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.174/30   45.38.161.173/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.174/30   45.38.161.173/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.174/30   45.38.161.173/30    7m59s
+sandbox15-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox15-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox15-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.174/30   45.38.161.173/30    8m41s
+sandbox15-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox15-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox15-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.189
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox14/sandbox-info.html b/en/3.4/sandbox/Sandbox14/sandbox-info.html new file mode 100644 index 0000000000..514827de5a --- /dev/null +++ b/en/3.4/sandbox/Sandbox14/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox14.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@50.117.27.85 -p 30061
+srv02-nyc: ssh demo@50.117.27.85 -p 30062
+srv03-nyc: ssh demo@50.117.27.85 -p 30063
+srv04-nyc: ssh demo@50.117.27.85 -p 30064
+srv05-nyc: ssh demo@50.117.27.85 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1141
+Local Address:                  45.38.161.170/30
+Remote Address:                 45.38.161.169/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.176/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1141
+Local Address:                  2607:f358:11:ffc0::1d/127
+Remote Address:                 2607:f358:11:ffc0::1c/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffce::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1142
+Local Address:                  45.38.161.174/30
+Remote Address:                 45.38.161.173/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.176/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.176/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.176/30
+|___ NAT Subnet:               45.38.161.180/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.184/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.188/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffce::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffce::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox15/configurations.html b/en/3.4/sandbox/Sandbox15/configurations.html new file mode 100644 index 0000000000..de1f8a991e --- /dev/null +++ b/en/3.4/sandbox/Sandbox15/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox15.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffcf::1 (from the “2607:f358:11:ffcf::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@50.117.27.86 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox15/creating-services.html b/en/3.4/sandbox/Sandbox15/creating-services.html new file mode 100644 index 0000000000..20fac240cd --- /dev/null +++ b/en/3.4/sandbox/Sandbox15/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.86 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1152.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.214.

    6. +
    7. In the Remote IP field, type in 45.38.161.213.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.192/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.193)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.86 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.196/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.196/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.196/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.200/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.200”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.200” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.200) into the browser’s address bar or simply visit http://45.38.161.200/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “45.38.161.200 (name_45.38.161.200)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.200 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@50.117.27.86 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox15/index.html b/en/3.4/sandbox/Sandbox15/index.html new file mode 100644 index 0000000000..2bf16726a7 --- /dev/null +++ b/en/3.4/sandbox/Sandbox15/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox15 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox15/onprem-k8s.html b/en/3.4/sandbox/Sandbox15/onprem-k8s.html new file mode 100644 index 0000000000..66b54bd78b --- /dev/null +++ b/en/3.4/sandbox/Sandbox15/onprem-k8s.html @@ -0,0 +1,1198 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox15.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox15.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.205   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.205:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.205
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.205   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.205   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.205   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.205   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.206   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.206
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.206
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.206
+
+SRV05-NYC
+curl 45.38.161.206
+
+SRV05-NYC
+curl 45.38.161.206
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1152
+  localIP: 45.38.161.214/30
+  remoteIP: 45.38.161.213/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.192/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.214/30   45.38.161.213/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.214/30   45.38.161.213/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.214/30   45.38.161.213/30    7m59s
+sandbox15-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox15-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox15-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.214/30   45.38.161.213/30    8m41s
+sandbox15-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox15-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox15-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.205
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox15/sandbox-info.html b/en/3.4/sandbox/Sandbox15/sandbox-info.html new file mode 100644 index 0000000000..18ee54ab4a --- /dev/null +++ b/en/3.4/sandbox/Sandbox15/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox15.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@50.117.27.86 -p 30061
+srv02-nyc: ssh demo@50.117.27.86 -p 30062
+srv03-nyc: ssh demo@50.117.27.86 -p 30063
+srv04-nyc: ssh demo@50.117.27.86 -p 30064
+srv05-nyc: ssh demo@50.117.27.86 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1151
+Local Address:                  45.38.161.210/30
+Remote Address:                 45.38.161.209/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.192/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1151
+Local Address:                  2607:f358:11:ffc0::1f/127
+Remote Address:                 2607:f358:11:ffc0::1e/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcf::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1152
+Local Address:                  45.38.161.214/30
+Remote Address:                 45.38.161.213/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.192/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.192/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.192/30
+|___ NAT Subnet:               45.38.161.196/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.200/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.204/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcf::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcf::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox2/configurations.html b/en/3.4/sandbox/Sandbox2/configurations.html new file mode 100644 index 0000000000..2d5bc8ad70 --- /dev/null +++ b/en/3.4/sandbox/Sandbox2/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox2.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc2::1 (from the “2607:f358:11:ffc2::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.190 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox2/creating-services.html b/en/3.4/sandbox/Sandbox2/creating-services.html new file mode 100644 index 0000000000..82a3bbb8a2 --- /dev/null +++ b/en/3.4/sandbox/Sandbox2/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1022.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.32/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.33)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.36/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.36/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.36/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.40/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.190 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox2/index.html b/en/3.4/sandbox/Sandbox2/index.html new file mode 100644 index 0000000000..2b3ab98fcd --- /dev/null +++ b/en/3.4/sandbox/Sandbox2/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox2 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox2/onprem-k8s.html b/en/3.4/sandbox/Sandbox2/onprem-k8s.html new file mode 100644 index 0000000000..58d0d7a0f5 --- /dev/null +++ b/en/3.4/sandbox/Sandbox2/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox2.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.45   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.45:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.45
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.45   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.45   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.45   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.45   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.46   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.46
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.46
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.46
+
+SRV05-NYC
+curl 45.38.161.46
+
+SRV05-NYC
+curl 45.38.161.46
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1022
+  localIP: 45.38.161.30/30
+  remoteIP: 45.38.161.29/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.32/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.30/30   45.38.161.29/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.30/30   45.38.161.29/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.30/30   45.38.161.29/30    7m59s
+sandbox2-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox2-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox2-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.30/30   45.38.161.29/30    8m41s
+sandbox2-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox2-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox2-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.45
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox2/sandbox-info.html b/en/3.4/sandbox/Sandbox2/sandbox-info.html new file mode 100644 index 0000000000..8c536a3776 --- /dev/null +++ b/en/3.4/sandbox/Sandbox2/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox2.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.190 -p 30061
+srv02-nyc: ssh demo@166.88.17.190 -p 30062
+srv03-nyc: ssh demo@166.88.17.190 -p 30063
+srv04-nyc: ssh demo@166.88.17.190 -p 30064
+srv05-nyc: ssh demo@166.88.17.190 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1021
+Local Address:                  45.38.161.26/30
+Remote Address:                 45.38.161.25/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.32/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1021
+Local Address:                  2607:f358:11:ffc0::5/127
+Remote Address:                 2607:f358:11:ffc0::4/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc2::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1022
+Local Address:                  45.38.161.30/30
+Remote Address:                 45.38.161.29/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.32/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.32/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.32/30
+|___ NAT Subnet:               45.38.161.36/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.40/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.44/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc2::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc2::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox3/configurations.html b/en/3.4/sandbox/Sandbox3/configurations.html new file mode 100644 index 0000000000..072a9ebb9f --- /dev/null +++ b/en/3.4/sandbox/Sandbox3/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox3.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc3::1 (from the “2607:f358:11:ffc3::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.189 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox3/creating-services.html b/en/3.4/sandbox/Sandbox3/creating-services.html new file mode 100644 index 0000000000..b7fd412ecf --- /dev/null +++ b/en/3.4/sandbox/Sandbox3/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1032.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.48/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.49)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.52/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.52/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.52/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.56/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.189 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox3/index.html b/en/3.4/sandbox/Sandbox3/index.html new file mode 100644 index 0000000000..d7831bad4f --- /dev/null +++ b/en/3.4/sandbox/Sandbox3/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox3 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox3/onprem-k8s.html b/en/3.4/sandbox/Sandbox3/onprem-k8s.html new file mode 100644 index 0000000000..053db850a6 --- /dev/null +++ b/en/3.4/sandbox/Sandbox3/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox3.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox3.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.61   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.61:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.61
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.61   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.61   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.61   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.61   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.62   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.62
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.62
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.62
+
+SRV05-NYC
+curl 45.38.161.62
+
+SRV05-NYC
+curl 45.38.161.62
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1032
+  localIP: 45.38.161.70/30
+  remoteIP: 45.38.161.69/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.48/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.70/30   45.38.161.69/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.70/30   45.38.161.69/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.70/30   45.38.161.69/30    7m59s
+sandbox3-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox3-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox3-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.70/30   45.38.161.69/30    8m41s
+sandbox3-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox3-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox3-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.61
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox3/sandbox-info.html b/en/3.4/sandbox/Sandbox3/sandbox-info.html new file mode 100644 index 0000000000..a129f49df1 --- /dev/null +++ b/en/3.4/sandbox/Sandbox3/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox3.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.189 -p 30061
+srv02-nyc: ssh demo@166.88.17.189 -p 30062
+srv03-nyc: ssh demo@166.88.17.189 -p 30063
+srv04-nyc: ssh demo@166.88.17.189 -p 30064
+srv05-nyc: ssh demo@166.88.17.189 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1031
+Local Address:                  45.38.161.66/30
+Remote Address:                 45.38.161.65/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.48/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1031
+Local Address:                  2607:f358:11:ffc0::7/127
+Remote Address:                 2607:f358:11:ffc0::6/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc3::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1032
+Local Address:                  45.38.161.70/30
+Remote Address:                 45.38.161.69/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.48/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.48/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.48/30
+|___ NAT Subnet:               45.38.161.52/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.56/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.60/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc3::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc3::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox4/configurations.html b/en/3.4/sandbox/Sandbox4/configurations.html new file mode 100644 index 0000000000..bfce71014c --- /dev/null +++ b/en/3.4/sandbox/Sandbox4/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox4.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc4::1 (from the “2607:f358:11:ffc4::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.188 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox4/creating-services.html b/en/3.4/sandbox/Sandbox4/creating-services.html new file mode 100644 index 0000000000..bba132df27 --- /dev/null +++ b/en/3.4/sandbox/Sandbox4/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1042.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 45.38.161.80/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(45.38.161.81)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.84/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.84/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 45.38.161.84/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.188 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.88/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox4/index.html b/en/3.4/sandbox/Sandbox4/index.html new file mode 100644 index 0000000000..d849f97373 --- /dev/null +++ b/en/3.4/sandbox/Sandbox4/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox4 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox4/onprem-k8s.html b/en/3.4/sandbox/Sandbox4/onprem-k8s.html new file mode 100644 index 0000000000..8be96d03cb --- /dev/null +++ b/en/3.4/sandbox/Sandbox4/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox4.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox4.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   45.38.161.93   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.93:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 45.38.161.93
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   45.38.161.93   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   45.38.161.93   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   45.38.161.93   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   45.38.161.93   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   45.38.161.94   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.94
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.94
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.94
+
+SRV05-NYC
+curl 45.38.161.94
+
+SRV05-NYC
+curl 45.38.161.94
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1042
+  localIP: 45.38.161.78/30
+  remoteIP: 45.38.161.77/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 45.38.161.80/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.78/30   45.38.161.77/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         45.38.161.78/30   45.38.161.77/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         45.38.161.78/30   45.38.161.77/30    7m59s
+sandbox4-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox4-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox4-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         45.38.161.78/30   45.38.161.77/30    8m41s
+sandbox4-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox4-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox4-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.93
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox4/sandbox-info.html b/en/3.4/sandbox/Sandbox4/sandbox-info.html new file mode 100644 index 0000000000..276b8342ba --- /dev/null +++ b/en/3.4/sandbox/Sandbox4/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox4.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.188 -p 30061
+srv02-nyc: ssh demo@166.88.17.188 -p 30062
+srv03-nyc: ssh demo@166.88.17.188 -p 30063
+srv04-nyc: ssh demo@166.88.17.188 -p 30064
+srv05-nyc: ssh demo@166.88.17.188 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1041
+Local Address:                  45.38.161.74/30
+Remote Address:                 45.38.161.73/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.80/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1041
+Local Address:                  2607:f358:11:ffc0::9/127
+Remote Address:                 2607:f358:11:ffc0::8/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc4::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1042
+Local Address:                  45.38.161.78/30
+Remote Address:                 45.38.161.77/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.80/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.80/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.80/30
+|___ NAT Subnet:               45.38.161.84/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.88/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.92/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc4::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc4::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox5/configurations.html b/en/3.4/sandbox/Sandbox5/configurations.html new file mode 100644 index 0000000000..a390c298c9 --- /dev/null +++ b/en/3.4/sandbox/Sandbox5/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox5.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc5::1 (from the “2607:f358:11:ffc5::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.187 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox5/creating-services.html b/en/3.4/sandbox/Sandbox5/creating-services.html new file mode 100644 index 0000000000..2a30e0885e --- /dev/null +++ b/en/3.4/sandbox/Sandbox5/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1052.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.128/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.129)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.132/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.132/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.132/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.136/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.187 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox5/index.html b/en/3.4/sandbox/Sandbox5/index.html new file mode 100644 index 0000000000..6312045fa0 --- /dev/null +++ b/en/3.4/sandbox/Sandbox5/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox5 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox5/onprem-k8s.html b/en/3.4/sandbox/Sandbox5/onprem-k8s.html new file mode 100644 index 0000000000..427dea7d8c --- /dev/null +++ b/en/3.4/sandbox/Sandbox5/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox5.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox5.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.141   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.141:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.141
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.141   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.141   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.141   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.141   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.142   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.142
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.142
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.142
+
+SRV05-NYC
+curl 50.117.59.142
+
+SRV05-NYC
+curl 50.117.59.142
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1052
+  localIP: 50.117.59.86/30
+  remoteIP: 50.117.59.85/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.128/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.86/30   50.117.59.85/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.86/30   50.117.59.85/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.86/30   50.117.59.85/30    7m59s
+sandbox5-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox5-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox5-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.86/30   50.117.59.85/30    8m41s
+sandbox5-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox5-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox5-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.141
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox5/sandbox-info.html b/en/3.4/sandbox/Sandbox5/sandbox-info.html new file mode 100644 index 0000000000..bd1df5014a --- /dev/null +++ b/en/3.4/sandbox/Sandbox5/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox5.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.187 -p 30061
+srv02-nyc: ssh demo@166.88.17.187 -p 30062
+srv03-nyc: ssh demo@166.88.17.187 -p 30063
+srv04-nyc: ssh demo@166.88.17.187 -p 30064
+srv05-nyc: ssh demo@166.88.17.187 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1051
+Local Address:                  50.117.59.82/30
+Remote Address:                 50.117.59.81/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.128/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1051
+Local Address:                  2607:f358:11:ffc0::b/127
+Remote Address:                 2607:f358:11:ffc0::a/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc5::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1052
+Local Address:                  50.117.59.86/30
+Remote Address:                 50.117.59.85/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.128/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.128/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.128/30
+|___ NAT Subnet:               50.117.59.132/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.136/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.140/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc5::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc5::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox6/configurations.html b/en/3.4/sandbox/Sandbox6/configurations.html new file mode 100644 index 0000000000..2d29862e49 --- /dev/null +++ b/en/3.4/sandbox/Sandbox6/configurations.html @@ -0,0 +1,754 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox6.netris.io and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@166.88.17.186 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox6/creating-services.html b/en/3.4/sandbox/Sandbox6/creating-services.html new file mode 100644 index 0000000000..3244275991 --- /dev/null +++ b/en/3.4/sandbox/Sandbox6/creating-services.html @@ -0,0 +1,837 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1062.

    4. +
    5. In the Local IP field, type in 50.117.59.94

    6. +
    7. In the Remote IP field, type in 50.117.59.93.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.144/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.150/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@166.88.17.186 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox6/index.html b/en/3.4/sandbox/Sandbox6/index.html new file mode 100644 index 0000000000..a11c12baba --- /dev/null +++ b/en/3.4/sandbox/Sandbox6/index.html @@ -0,0 +1,733 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox6 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox6/onprem-k8s.html b/en/3.4/sandbox/Sandbox6/onprem-k8s.html new file mode 100644 index 0000000000..d88773b1fa --- /dev/null +++ b/en/3.4/sandbox/Sandbox6/onprem-k8s.html @@ -0,0 +1,1194 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox6.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox6.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.154   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.154:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.154
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.155   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.155
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.155
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1062
+  localIP: 50.117.59.94/30
+  remoteIP: 50.117.59.93/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.144/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.94/30   50.117.59.93/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.94/30   50.117.59.93/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.94/30   50.117.59.93/30    7m59s
+sandbox6-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox6-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox6-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.94/30   50.117.59.93/30    8m41s
+sandbox6-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox6-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox6-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.154
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox6/sandbox-info.html b/en/3.4/sandbox/Sandbox6/sandbox-info.html new file mode 100644 index 0000000000..5fec861b54 --- /dev/null +++ b/en/3.4/sandbox/Sandbox6/sandbox-info.html @@ -0,0 +1,770 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing the Linux servers:

+
srv01: ssh demo@166.88.17.186 -p 30061
+srv02: ssh demo@166.88.17.186 -p 30062
+srv03: ssh demo@166.88.17.186 -p 30063
+srv04: ssh demo@166.88.17.186 -p 30064
+srv05: ssh demo@166.88.17.186 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1061
+IP customer: 50.117.59.90/30
+IP Iris: 50.117.59.89/30
+
+Neighbor AS: 65007
+Vlan: 1062
+IP customer:  50.117.59.94/30
+IP Iris: 50.117.59.93/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     50.117.59.144/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox7/configurations.html b/en/3.4/sandbox/Sandbox7/configurations.html new file mode 100644 index 0000000000..c2fc166bf9 --- /dev/null +++ b/en/3.4/sandbox/Sandbox7/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox7.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc7::1 (from the “2607:f358:11:ffc7::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.185 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox7/creating-services.html b/en/3.4/sandbox/Sandbox7/creating-services.html new file mode 100644 index 0000000000..511309f488 --- /dev/null +++ b/en/3.4/sandbox/Sandbox7/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1072.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.160/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.161)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.164/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.164/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.164/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.168/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.185 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox7/index.html b/en/3.4/sandbox/Sandbox7/index.html new file mode 100644 index 0000000000..4e85f05f81 --- /dev/null +++ b/en/3.4/sandbox/Sandbox7/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox7 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox7/onprem-k8s.html b/en/3.4/sandbox/Sandbox7/onprem-k8s.html new file mode 100644 index 0000000000..5182e74a33 --- /dev/null +++ b/en/3.4/sandbox/Sandbox7/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox7.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox7.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.173   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.173:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.173
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.173   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.173   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.173   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.173   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.174   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.174
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.174
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.174
+
+SRV05-NYC
+curl 50.117.59.174
+
+SRV05-NYC
+curl 50.117.59.174
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1072
+  localIP: 50.117.59.102/30
+  remoteIP: 50.117.59.101/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.160/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.102/30   50.117.59.101/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.102/30   50.117.59.101/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.102/30   50.117.59.101/30    7m59s
+sandbox7-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox7-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox7-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.102/30   50.117.59.101/30    8m41s
+sandbox7-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox7-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox7-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.173
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox7/sandbox-info.html b/en/3.4/sandbox/Sandbox7/sandbox-info.html new file mode 100644 index 0000000000..d1ba65739d --- /dev/null +++ b/en/3.4/sandbox/Sandbox7/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox7.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.185 -p 30061
+srv02-nyc: ssh demo@166.88.17.185 -p 30062
+srv03-nyc: ssh demo@166.88.17.185 -p 30063
+srv04-nyc: ssh demo@166.88.17.185 -p 30064
+srv05-nyc: ssh demo@166.88.17.185 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1071
+Local Address:                  50.117.59.98/30
+Remote Address:                 50.117.59.97/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.160/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1071
+Local Address:                  2607:f358:11:ffc0::f/127
+Remote Address:                 2607:f358:11:ffc0::e/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc7::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1072
+Local Address:                  50.117.59.102/30
+Remote Address:                 50.117.59.101/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.160/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.160/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.160/30
+|___ NAT Subnet:               50.117.59.164/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.168/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.172/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc7::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc7::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox8/configurations.html b/en/3.4/sandbox/Sandbox8/configurations.html new file mode 100644 index 0000000000..380fdc6c43 --- /dev/null +++ b/en/3.4/sandbox/Sandbox8/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox8.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc8::1 (from the “2607:f358:11:ffc8::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.29 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox8/creating-services.html b/en/3.4/sandbox/Sandbox8/creating-services.html new file mode 100644 index 0000000000..400de58bec --- /dev/null +++ b/en/3.4/sandbox/Sandbox8/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.29 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1082.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.110.

    6. +
    7. In the Remote IP field, type in 50.117.59.109.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.176/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.177)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.29 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.180/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.180/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.180/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.184/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.29 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox8/index.html b/en/3.4/sandbox/Sandbox8/index.html new file mode 100644 index 0000000000..6a5f5807a5 --- /dev/null +++ b/en/3.4/sandbox/Sandbox8/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox8 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox8/onprem-k8s.html b/en/3.4/sandbox/Sandbox8/onprem-k8s.html new file mode 100644 index 0000000000..061325426b --- /dev/null +++ b/en/3.4/sandbox/Sandbox8/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox8.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox8.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.189   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.189:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.189
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.189   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.189   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.189   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.189   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.190   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.190
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.190
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.190
+
+SRV05-NYC
+curl 50.117.59.190
+
+SRV05-NYC
+curl 50.117.59.190
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1082
+  localIP: 50.117.59.110/30
+  remoteIP: 50.117.59.109/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.176/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.110/30   50.117.59.109/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.110/30   50.117.59.109/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.110/30   50.117.59.109/30    7m59s
+sandbox8-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox8-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox8-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.110/30   50.117.59.109/30    8m41s
+sandbox8-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox8-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox8-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.189
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox8/sandbox-info.html b/en/3.4/sandbox/Sandbox8/sandbox-info.html new file mode 100644 index 0000000000..30d03193f0 --- /dev/null +++ b/en/3.4/sandbox/Sandbox8/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox8.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.29 -p 30061
+srv02-nyc: ssh demo@166.88.17.29 -p 30062
+srv03-nyc: ssh demo@166.88.17.29 -p 30063
+srv04-nyc: ssh demo@166.88.17.29 -p 30064
+srv05-nyc: ssh demo@166.88.17.29 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1081
+Local Address:                  50.117.59.106/30
+Remote Address:                 50.117.59.105/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.176/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1081
+Local Address:                  2607:f358:11:ffc0::11/127
+Remote Address:                 2607:f358:11:ffc0::10/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc8::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1082
+Local Address:                  50.117.59.110/30
+Remote Address:                 50.117.59.109/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.176/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.176/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.176/30
+|___ NAT Subnet:               50.117.59.180/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.184/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.188/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc8::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc8::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox9/configurations.html b/en/3.4/sandbox/Sandbox9/configurations.html new file mode 100644 index 0000000000..81f9f88ccc --- /dev/null +++ b/en/3.4/sandbox/Sandbox9/configurations.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox9.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffc9::1 (from the “2607:f358:11:ffc9::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@166.88.17.22 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox9/creating-services.html b/en/3.4/sandbox/Sandbox9/creating-services.html new file mode 100644 index 0000000000..4cf0f5ddf5 --- /dev/null +++ b/en/3.4/sandbox/Sandbox9/creating-services.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.22 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1092.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.192/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.193)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.22 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.196/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.196/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.196/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.200/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@166.88.17.22 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox9/index.html b/en/3.4/sandbox/Sandbox9/index.html new file mode 100644 index 0000000000..2ad02dca51 --- /dev/null +++ b/en/3.4/sandbox/Sandbox9/index.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox9 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox9/onprem-k8s.html b/en/3.4/sandbox/Sandbox9/onprem-k8s.html new file mode 100644 index 0000000000..91e16788ce --- /dev/null +++ b/en/3.4/sandbox/Sandbox9/onprem-k8s.html @@ -0,0 +1,1196 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox9.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox9.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.205   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.205:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.205
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.205   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.205   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.205   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.205   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.206   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.206
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.206
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.206
+
+SRV05-NYC
+curl 50.117.59.206
+
+SRV05-NYC
+curl 50.117.59.206
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1092
+  localIP: 50.117.59.118/30
+  remoteIP: 50.117.59.117/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.192/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.118/30   50.117.59.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.118/30   50.117.59.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.118/30   50.117.59.117/30    7m59s
+sandbox9-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox9-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox9-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.118/30   50.117.59.117/30    8m41s
+sandbox9-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox9-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox9-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.205
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/sandbox/Sandbox9/sandbox-info.html b/en/3.4/sandbox/Sandbox9/sandbox-info.html new file mode 100644 index 0000000000..86501dd1cb --- /dev/null +++ b/en/3.4/sandbox/Sandbox9/sandbox-info.html @@ -0,0 +1,810 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox9.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@166.88.17.22 -p 30061
+srv02-nyc: ssh demo@166.88.17.22 -p 30062
+srv03-nyc: ssh demo@166.88.17.22 -p 30063
+srv04-nyc: ssh demo@166.88.17.22 -p 30064
+srv05-nyc: ssh demo@166.88.17.22 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1091
+Local Address:                  50.117.59.114/30
+Remote Address:                 50.117.59.113/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.192/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1091
+Local Address:                  2607:f358:11:ffc0::13/127
+Remote Address:                 2607:f358:11:ffc0::12/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc9::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1092
+Local Address:                  50.117.59.118/30
+Remote Address:                 50.117.59.117/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.192/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.192/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.192/30
+|___ NAT Subnet:               50.117.59.196/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.200/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.204/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc9::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc9::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/search.html b/en/3.4/search.html new file mode 100644 index 0000000000..1af9807055 --- /dev/null +++ b/en/3.4/search.html @@ -0,0 +1,709 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Search — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Search
  • + + +
  • + +
  • + +
+ + +
+
+
+
+ + + + +
+ +
+ +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/searchindex.js b/en/3.4/searchindex.js new file mode 100644 index 0000000000..66d8cb18d2 --- /dev/null +++ b/en/3.4/searchindex.js @@ -0,0 +1 @@ +Search.setIndex({"docnames": ["EdgeCore-SONiC-Switch-initial-setup", "Nvidia-Cumulus-v3.7-Switch-initial-setup", "Nvidia-Cumulus-v5-Switch-initial-setup", "SoftGate-PRO-installation", "SoftGate-installation", "Ubuntu-SwitchDev-Switch-initial-setup", "accounts", "acls", "controller-k3s-installation", "controller-k8s-installation", "controller-k8s-quickstart", "controller-vm-installation", "definitions", "index", "installation", "installing-netris-controller", "introduction", "inventory-profiles", "ipam", "kubernetes-integration", "l3-load-balancer", "l4-load-balancer", "netris-architecture", "network-policies", "reference-designs", "release-notes", "roh", "sandbox/Sandbox1/configurations", "sandbox/Sandbox1/creating-services", "sandbox/Sandbox1/index", "sandbox/Sandbox1/onprem-k8s", "sandbox/Sandbox1/sandbox-info", "sandbox/Sandbox10/configurations", "sandbox/Sandbox10/creating-services", "sandbox/Sandbox10/index", "sandbox/Sandbox10/onprem-k8s", "sandbox/Sandbox10/sandbox-info", "sandbox/Sandbox11/configurations", "sandbox/Sandbox11/creating-services", "sandbox/Sandbox11/index", "sandbox/Sandbox11/onprem-k8s", "sandbox/Sandbox11/sandbox-info", "sandbox/Sandbox12/configurations", "sandbox/Sandbox12/creating-services", "sandbox/Sandbox12/index", "sandbox/Sandbox12/onprem-k8s", "sandbox/Sandbox12/sandbox-info", "sandbox/Sandbox13/configurations", "sandbox/Sandbox13/creating-services", "sandbox/Sandbox13/index", "sandbox/Sandbox13/onprem-k8s", "sandbox/Sandbox13/sandbox-info", "sandbox/Sandbox14/configurations", "sandbox/Sandbox14/creating-services", "sandbox/Sandbox14/index", "sandbox/Sandbox14/onprem-k8s", "sandbox/Sandbox14/sandbox-info", "sandbox/Sandbox15/configurations", "sandbox/Sandbox15/creating-services", "sandbox/Sandbox15/index", "sandbox/Sandbox15/onprem-k8s", "sandbox/Sandbox15/sandbox-info", "sandbox/Sandbox2/configurations", "sandbox/Sandbox2/creating-services", "sandbox/Sandbox2/index", "sandbox/Sandbox2/onprem-k8s", "sandbox/Sandbox2/sandbox-info", "sandbox/Sandbox3/configurations", "sandbox/Sandbox3/creating-services", "sandbox/Sandbox3/index", "sandbox/Sandbox3/onprem-k8s", "sandbox/Sandbox3/sandbox-info", "sandbox/Sandbox4/configurations", "sandbox/Sandbox4/creating-services", "sandbox/Sandbox4/index", "sandbox/Sandbox4/onprem-k8s", "sandbox/Sandbox4/sandbox-info", "sandbox/Sandbox5/configurations", "sandbox/Sandbox5/creating-services", "sandbox/Sandbox5/index", "sandbox/Sandbox5/onprem-k8s", "sandbox/Sandbox5/sandbox-info", "sandbox/Sandbox6/configurations", "sandbox/Sandbox6/creating-services", "sandbox/Sandbox6/index", "sandbox/Sandbox6/onprem-k8s", "sandbox/Sandbox6/sandbox-info", "sandbox/Sandbox7/configurations", "sandbox/Sandbox7/creating-services", "sandbox/Sandbox7/index", "sandbox/Sandbox7/onprem-k8s", "sandbox/Sandbox7/sandbox-info", "sandbox/Sandbox8/configurations", "sandbox/Sandbox8/creating-services", "sandbox/Sandbox8/index", "sandbox/Sandbox8/onprem-k8s", "sandbox/Sandbox8/sandbox-info", "sandbox/Sandbox9/configurations", "sandbox/Sandbox9/creating-services", "sandbox/Sandbox9/index", "sandbox/Sandbox9/onprem-k8s", "sandbox/Sandbox9/sandbox-info", "softgate-performance", "supported-networks", "switch-agent-installation", "switch-ports", "terraform-integration", "topology-management", "tutorials/activating-bgp-on-equinix-metal-project", "tutorials/adding-netris-softgate-nodes-in-equinix-metal", "tutorials/aws-concept", "tutorials/aws-deploy-softgate", "tutorials/create-interconnection-to-fabric", "tutorials/enable-services-on-equinix-metal-project", "tutorials/equinix-metal-api-integration-enablement", "tutorials/gcp-concept", "tutorials/gcp-deploy-softgate", "tutorials/getting-started-for-equinix-metal", "tutorials/index", "tutorials/index-equinix", "tutorials/index-vpc", "tutorials/install-netris-controller-in-equinix-metal", "tutorials/installing-netris-controller", "tutorials/netris-vpc-for-aws", "tutorials/netris-vpc-for-equinix-metal", "tutorials/netris-vpc-for-gcp", "tutorials/netris-vpc-for-phoenixnap-bmc", "tutorials/phoenixnap-bmc-adding-netris-softgate-nodes", "tutorials/phoenixnap-bmc-api-integration-enablement", "tutorials/phoenixnap-bmc-concept", "tutorials/phoenixnap-bmc-install-netris-controller", "tutorials/phoenixnap-bmc-ipam-setup", "tutorials/phoenixnap-bmc-link-to-installation", "tutorials/phoenixnap-bmc-using-l4lb", "tutorials/phoenixnap-bmc-using-nat", "tutorials/phoenixnap-bmc-using-vnet", "tutorials/upgrading-netris", "tutorials/upgrading-sonic-os", "tutorials/using-l4-load-balancer", "tutorials/using-vnet-in-equinix-metal-project", "tutorials/vpc-anywhere", "tutorials/vpc-anywhere-check-default-site", "tutorials/vpc-anywhere-concept", "tutorials/vpc-anywhere-controller-installation", "tutorials/vpc-anywhere-ipam-setup", "tutorials/vpc-anywhere-softgate-installation", "tutorials/vpc-anywhere-upstream-peering", "tutorials/vpc-anywhere-using-l4lb", "tutorials/vpc-anywhere-using-multi-interface-softgate", "tutorials/vpc-anywhere-using-nat", "tutorials/vpc-anywhere-using-vnet", "visibility", "vnet"], "filenames": ["EdgeCore-SONiC-Switch-initial-setup.rst", "Nvidia-Cumulus-v3.7-Switch-initial-setup.rst", "Nvidia-Cumulus-v5-Switch-initial-setup.rst", "SoftGate-PRO-installation.rst", "SoftGate-installation.rst", "Ubuntu-SwitchDev-Switch-initial-setup.rst", "accounts.rst", "acls.rst", "controller-k3s-installation.rst", "controller-k8s-installation.rst", "controller-k8s-quickstart.rst", "controller-vm-installation.rst", "definitions.rst", "index.rst", "installation.rst", "installing-netris-controller.rst", "introduction.rst", "inventory-profiles.rst", "ipam.rst", "kubernetes-integration.rst", "l3-load-balancer.rst", "l4-load-balancer.rst", "netris-architecture.rst", "network-policies.rst", "reference-designs.rst", "release-notes.rst", "roh.rst", "sandbox/Sandbox1/configurations.rst", "sandbox/Sandbox1/creating-services.rst", "sandbox/Sandbox1/index.rst", "sandbox/Sandbox1/onprem-k8s.rst", "sandbox/Sandbox1/sandbox-info.rst", "sandbox/Sandbox10/configurations.rst", "sandbox/Sandbox10/creating-services.rst", "sandbox/Sandbox10/index.rst", "sandbox/Sandbox10/onprem-k8s.rst", "sandbox/Sandbox10/sandbox-info.rst", "sandbox/Sandbox11/configurations.rst", "sandbox/Sandbox11/creating-services.rst", "sandbox/Sandbox11/index.rst", "sandbox/Sandbox11/onprem-k8s.rst", "sandbox/Sandbox11/sandbox-info.rst", "sandbox/Sandbox12/configurations.rst", "sandbox/Sandbox12/creating-services.rst", "sandbox/Sandbox12/index.rst", "sandbox/Sandbox12/onprem-k8s.rst", "sandbox/Sandbox12/sandbox-info.rst", "sandbox/Sandbox13/configurations.rst", "sandbox/Sandbox13/creating-services.rst", "sandbox/Sandbox13/index.rst", "sandbox/Sandbox13/onprem-k8s.rst", "sandbox/Sandbox13/sandbox-info.rst", "sandbox/Sandbox14/configurations.rst", "sandbox/Sandbox14/creating-services.rst", "sandbox/Sandbox14/index.rst", "sandbox/Sandbox14/onprem-k8s.rst", "sandbox/Sandbox14/sandbox-info.rst", "sandbox/Sandbox15/configurations.rst", "sandbox/Sandbox15/creating-services.rst", "sandbox/Sandbox15/index.rst", "sandbox/Sandbox15/onprem-k8s.rst", "sandbox/Sandbox15/sandbox-info.rst", "sandbox/Sandbox2/configurations.rst", "sandbox/Sandbox2/creating-services.rst", "sandbox/Sandbox2/index.rst", "sandbox/Sandbox2/onprem-k8s.rst", "sandbox/Sandbox2/sandbox-info.rst", "sandbox/Sandbox3/configurations.rst", "sandbox/Sandbox3/creating-services.rst", "sandbox/Sandbox3/index.rst", "sandbox/Sandbox3/onprem-k8s.rst", "sandbox/Sandbox3/sandbox-info.rst", "sandbox/Sandbox4/configurations.rst", "sandbox/Sandbox4/creating-services.rst", "sandbox/Sandbox4/index.rst", "sandbox/Sandbox4/onprem-k8s.rst", "sandbox/Sandbox4/sandbox-info.rst", "sandbox/Sandbox5/configurations.rst", "sandbox/Sandbox5/creating-services.rst", "sandbox/Sandbox5/index.rst", "sandbox/Sandbox5/onprem-k8s.rst", "sandbox/Sandbox5/sandbox-info.rst", "sandbox/Sandbox6/configurations.rst", "sandbox/Sandbox6/creating-services.rst", "sandbox/Sandbox6/index.rst", "sandbox/Sandbox6/onprem-k8s.rst", "sandbox/Sandbox6/sandbox-info.rst", "sandbox/Sandbox7/configurations.rst", "sandbox/Sandbox7/creating-services.rst", "sandbox/Sandbox7/index.rst", "sandbox/Sandbox7/onprem-k8s.rst", "sandbox/Sandbox7/sandbox-info.rst", "sandbox/Sandbox8/configurations.rst", "sandbox/Sandbox8/creating-services.rst", "sandbox/Sandbox8/index.rst", "sandbox/Sandbox8/onprem-k8s.rst", "sandbox/Sandbox8/sandbox-info.rst", "sandbox/Sandbox9/configurations.rst", "sandbox/Sandbox9/creating-services.rst", "sandbox/Sandbox9/index.rst", "sandbox/Sandbox9/onprem-k8s.rst", "sandbox/Sandbox9/sandbox-info.rst", "softgate-performance.rst", "supported-networks.rst", "switch-agent-installation.rst", "switch-ports.rst", "terraform-integration.rst", "topology-management.rst", "tutorials/activating-bgp-on-equinix-metal-project.rst", "tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst", "tutorials/aws-concept.rst", "tutorials/aws-deploy-softgate.rst", "tutorials/create-interconnection-to-fabric.rst", "tutorials/enable-services-on-equinix-metal-project.rst", "tutorials/equinix-metal-api-integration-enablement.rst", "tutorials/gcp-concept.rst", "tutorials/gcp-deploy-softgate.rst", "tutorials/getting-started-for-equinix-metal.rst", "tutorials/index.rst", "tutorials/index-equinix.rst", "tutorials/index-vpc.rst", "tutorials/install-netris-controller-in-equinix-metal.rst", "tutorials/installing-netris-controller.rst", "tutorials/netris-vpc-for-aws.rst", "tutorials/netris-vpc-for-equinix-metal.rst", "tutorials/netris-vpc-for-gcp.rst", "tutorials/netris-vpc-for-phoenixnap-bmc.rst", "tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst", "tutorials/phoenixnap-bmc-api-integration-enablement.rst", "tutorials/phoenixnap-bmc-concept.rst", "tutorials/phoenixnap-bmc-install-netris-controller.rst", "tutorials/phoenixnap-bmc-ipam-setup.rst", "tutorials/phoenixnap-bmc-link-to-installation.rst", "tutorials/phoenixnap-bmc-using-l4lb.rst", "tutorials/phoenixnap-bmc-using-nat.rst", "tutorials/phoenixnap-bmc-using-vnet.rst", "tutorials/upgrading-netris.rst", "tutorials/upgrading-sonic-os.rst", "tutorials/using-l4-load-balancer.rst", "tutorials/using-vnet-in-equinix-metal-project.rst", "tutorials/vpc-anywhere.rst", "tutorials/vpc-anywhere-check-default-site.rst", "tutorials/vpc-anywhere-concept.rst", "tutorials/vpc-anywhere-controller-installation.rst", "tutorials/vpc-anywhere-ipam-setup.rst", "tutorials/vpc-anywhere-softgate-installation.rst", "tutorials/vpc-anywhere-upstream-peering.rst", "tutorials/vpc-anywhere-using-l4lb.rst", "tutorials/vpc-anywhere-using-multi-interface-softgate.rst", "tutorials/vpc-anywhere-using-nat.rst", "tutorials/vpc-anywhere-using-vnet.rst", "visibility.rst", "vnet.rst"], "titles": ["EdgeCore SONiC Switch Initial Setup", "Nvidia Cumulus v3.7 Switch Initial Setup", "Nvidia Cumulus v5 Switch Initial Setup", "SoftGate PRO Installation", "SoftGate Installation", "Ubuntu SwitchDev Switch Initial Setup", "Accounts", "Access Control Lists (ACL)", "Netris Controller installation on a generic Linux host", "Helm Chart Installation", "Quickstart Installation", "Virtual Machine Installation", "Definitions", "Welcome to Netris Documentation", "Controller Installation", "Installing a Netris Controller", "Introduction to Netris", "Inventory Profiles", "IP Address Management", "Kubernetes Integration", "L3 Load Balancer (Anycast LB)", "L4 Load Balancer (L4LB)", "Netris Architecture", "Basic BGP", "Network Reference Designs", "Release notes", "ROH (Routing on the Host)", "Provided Example Configurations", "Learn by Creating Services", "Sandbox1", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox10", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox11", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox12", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox13", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox14", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox15", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox2", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox3", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox4", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox5", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox6", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox7", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox8", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox9", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "SoftGate Performance", "Reference Network Architectures", "Network Switch Initial Setup", "Switch Ports", "Terraform: Netris provider", "Inventory", "Activating BGP on Equinix Metal Project", "Provisioning Netris SoftGate nodes in Equinix Metal Project", "Site Mesh with AWS Overview", "Deploy a Softgate in AWS", "Creating an Interconnection to Equinix Fabric", "Enabling services (NAT, V-Net, Load Balancer, IP pools)", "Enable Equinix Metal API integration", "Site Mesh with GCP Overview", "Deploy a Softgate in GCP", "Equinix Metal API integration enablement", "Netris Generic Tutorials", "Netris VPC for Equinix Metal Getting Started Guide", "VPC for Anywhere Getting Started Guide", "Installing a Netris Controller on Equinix Metal on-demand server", "Installing a Netris Controller", "Enabling Site Mesh with AWS", "Netris VPC for Equinix Metal", "Enabling Site Mesh with GCP", "Netris VPC for phoenixNAP BMC", "Provisioning Netris SoftGate nodes in phoenixNAP BMC", "Enable phoenixNAP BMC API integration", "Netris VPC for phoenixNAP BMC Overview", "Installing a Netris Controller on phoenixNAP BMC server", "IPAM Setup for Services", "Installing Netris on phoenixNAP BMC", "Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC", "Using NAT services in phoenixNAP BMC", "Using V-Net (isolated virtual network) services in phoenixNAP BMC", "Netris Upgrade and Rollback Procedures", "Upgrading SONiC Operating System", "Using on-demand (elastic) L4 Load Balancer service", "Using V-Net (isolated virtual network) services in Equinix Metal Project", "VPC Anywhere Getting Started Guide", "Check Default Site Settings", "VPC Anywhere Overview", "Install a Netris Controller", "IPAM Setup for Services", "SoftGate Installation", "Connecting VPC to upstream networks (use one of two options)", "Using on-demand (elastic) L4 Load Balancer service", "Using Multi-interface SoftGate (experimental)", "Using NAT services", "Using V-Net (isolated virtual network) services", "Visibility (Telescope)", "V-Net"], "terms": {"A": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152], "variabl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152], "further": [0, 1, 2, 3, 4, 5, 22, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 108, 145], "instal": [0, 1, 2, 5, 25, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 109, 110, 111, 115, 116, 117, 118, 119, 120, 124, 126, 127, 129, 134, 136, 137, 140, 149], "requir": [0, 1, 2, 5, 7, 13, 14, 15, 16, 18, 19, 21, 22, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 107, 111, 112, 116, 120, 121, 122, 127, 130, 134, 140, 144, 149, 152], "consol": [0, 1, 2, 5, 11, 15, 108, 109, 111, 112, 113, 114, 116, 117, 121, 122, 127, 128, 130, 131, 133, 135, 138, 139, 143, 144, 147, 148], "internet": [0, 1, 2, 3, 4, 5, 8, 11, 15, 22, 23, 25, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 108, 113, 122, 131, 134, 135, 139, 143, 146, 149, 150], "connect": [0, 1, 2, 3, 4, 5, 8, 10, 11, 13, 19, 20, 21, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 107, 111, 112, 116, 134, 135, 136, 140, 142, 145, 149, 150, 152], "via": [0, 1, 2, 3, 4, 5, 12, 18, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 105, 106, 111, 116, 129, 135, 136, 137, 139, 145], "manag": [0, 1, 2, 3, 4, 5, 6, 8, 10, 11, 12, 13, 15, 19, 22, 23, 25, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 111, 112, 116, 122, 135, 136, 139, 142, 143, 145, 150, 151, 152], "port": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 19, 20, 21, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 107, 111, 112, 116, 122, 134, 142, 143, 145, 149, 151, 152], "If": [0, 1, 2, 3, 5, 8, 10, 11, 19, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 136, 144, 148, 152], "ha": [0, 1, 2, 5, 6, 11, 13, 15, 19, 22, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 107, 108, 116, 121, 122, 129, 130, 131, 135, 136, 139, 143, 144], "pre": [0, 1, 2, 4, 5, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 101, 145], "network": [0, 1, 2, 3, 4, 5, 6, 7, 11, 12, 15, 16, 18, 19, 22, 23, 25, 26, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 106, 107, 108, 110, 111, 112, 113, 115, 116, 119, 122, 124, 126, 129, 131, 134, 140, 141, 142, 143, 145, 149, 151, 152], "oper": [0, 1, 2, 4, 5, 12, 16, 22, 23, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 107, 111, 113, 116, 121, 130, 131, 145, 150], "system": [0, 1, 2, 5, 6, 7, 8, 11, 12, 17, 22, 23, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 107, 111, 116, 136, 151], "NOS": [0, 1, 2, 5, 22, 107], "need": [0, 1, 2, 4, 5, 7, 8, 15, 19, 20, 23, 25, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 102, 105, 107, 108, 109, 110, 111, 113, 114, 115, 116, 117, 122, 127, 128, 129, 131, 135, 136, 137, 139, 142, 143, 144, 145, 146, 151, 152], "uninstal": [0, 1, 2, 5, 14], "first": [0, 1, 2, 5, 8, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 107, 110, 111, 112, 115, 116, 131, 136, 137, 144, 151], "To": [0, 1, 2, 5, 6, 7, 8, 9, 15, 19, 20, 23, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 111, 112, 113, 116, 131, 132, 136, 137, 144, 146, 148, 150, 151], "current": [0, 1, 2, 5, 8, 18, 19, 23, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 83, 85, 90, 95, 100, 107, 111, 116, 136, 146, 151, 152], "access": [0, 1, 2, 5, 6, 8, 12, 13, 15, 19, 21, 22, 23, 24, 25, 29, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 86, 89, 90, 91, 94, 95, 96, 99, 100, 101, 107, 110, 111, 113, 115, 116, 121, 122, 130, 131, 134, 135, 137, 139, 143, 144, 149, 150, 151], "oni": [0, 1, 2, 5], "from": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 13, 17, 18, 20, 21, 22, 23, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 105, 107, 109, 111, 112, 113, 114, 116, 117, 118, 119, 120, 127, 128, 129, 131, 134, 135, 136, 137, 139, 142, 144, 145, 148, 149, 150, 151, 152], "grub": [0, 1, 2, 5], "menu": [0, 1, 2, 5, 6, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 105, 109, 111, 113, 114, 116, 117, 127, 128, 131, 136, 144, 151], "select": [0, 1, 2, 3, 5, 10, 14, 17, 18, 20, 21, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 105, 107, 111, 112, 113, 114, 116, 117, 128, 130, 131, 144, 148, 151], "o": [0, 1, 2, 3, 4, 5, 8, 15, 19, 22, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 102, 109, 117, 121, 122, 127, 130, 135, 137, 143, 145], "option": [0, 1, 2, 3, 4, 5, 7, 10, 13, 14, 19, 20, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 105, 106, 111, 116, 126, 135, 136, 139, 140, 145, 151], "onc": [0, 1, 2, 3, 4, 5, 7, 8, 10, 15, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 105, 107, 110, 111, 112, 113, 115, 116, 121, 122, 129, 130, 131, 134, 135, 136, 139, 143, 144, 145, 149], "": [0, 1, 2, 4, 5, 6, 7, 15, 16, 19, 22, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 105, 106, 107, 110, 111, 113, 115, 121, 122, 127, 129, 130, 131, 134, 135, 136, 139, 143, 145, 148, 149, 150, 152], "done": [0, 1, 2, 5, 106, 109, 110, 111, 113, 115, 116, 117, 127, 129], "automat": [0, 1, 2, 4, 5, 16, 19, 21, 22, 23, 25, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 105, 106, 107, 108, 111, 113, 116, 129, 131, 135, 142, 145, 146, 151, 152], "reboot": [0, 1, 2, 3, 4, 5, 11, 109, 111, 116, 117, 127, 137, 145], "get": [0, 1, 2, 5, 7, 8, 10, 11, 13, 14, 15, 19, 21, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 106, 111, 113, 118, 121, 122, 130, 131, 135, 136, 139, 143], "readi": [0, 1, 2, 5, 19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 112, 135, 136, 139], "ec": 0, "i": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 109, 110, 111, 112, 113, 115, 116, 117, 121, 122, 127, 129, 130, 131, 133, 134, 135, 136, 137, 138, 139, 141, 142, 143, 144, 145, 146, 147, 149, 150, 151, 152], "dhcp": [0, 1, 2, 5, 11, 135, 150], "stop": [0, 1, 2, 5, 111, 136], "discoveri": [0, 1, 2, 5], "servic": [0, 1, 2, 5, 6, 7, 11, 12, 16, 17, 18, 19, 20, 22, 23, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59, 60, 61, 62, 64, 65, 66, 67, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 99, 100, 101, 102, 107, 108, 111, 112, 116, 119, 121, 122, 124, 126, 129, 130, 140, 143, 151, 152], "configur": [0, 1, 2, 4, 5, 10, 11, 12, 14, 16, 19, 20, 22, 23, 25, 26, 28, 29, 30, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96, 98, 99, 100, 101, 105, 107, 108, 110, 112, 113, 115, 123, 125, 129, 145, 150, 151, 152], "an": [0, 1, 2, 3, 4, 5, 6, 7, 10, 11, 13, 15, 16, 17, 21, 22, 23, 25, 26, 27, 28, 29, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 98, 99, 101, 107, 110, 114, 115, 116, 117, 121, 122, 123, 128, 129, 130, 131, 133, 136, 137, 138, 139, 140, 142, 143, 145, 146, 147, 148, 150, 152], "ip": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 19, 20, 21, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 107, 108, 109, 110, 111, 112, 115, 116, 117, 119, 121, 122, 124, 127, 129, 130, 131, 134, 135, 137, 139, 140, 143, 144, 145, 149, 150], "address": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 19, 20, 21, 22, 23, 25, 26, 29, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 86, 89, 90, 91, 94, 95, 96, 99, 100, 101, 106, 107, 108, 109, 111, 116, 117, 119, 121, 122, 124, 127, 129, 130, 131, 134, 135, 137, 139, 140, 143, 145, 149, 150, 151, 152], "default": [0, 1, 2, 3, 4, 5, 6, 9, 11, 12, 13, 15, 17, 19, 23, 25, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 105, 106, 107, 111, 114, 116, 117, 121, 122, 128, 129, 130, 135, 139, 140, 142, 143, 144, 145, 146, 148, 150, 152], "gatewai": [0, 1, 2, 3, 4, 5, 11, 19, 25, 29, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 86, 89, 90, 91, 94, 95, 96, 99, 100, 101, 106, 111, 113, 116, 129, 131, 135, 139, 142, 145, 150, 152], "manual": [0, 1, 2, 4, 5, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 107, 145], "addr": [0, 1, 2, 5], "add": [0, 1, 2, 3, 4, 5, 9, 10, 13, 19, 20, 21, 23, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 105, 109, 112, 114, 117, 121, 122, 128, 130, 131, 135, 139, 143, 144, 145, 151, 152], "prefix": [0, 1, 2, 3, 4, 5, 11, 13, 18, 19, 26, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106, 111, 116, 131, 144, 145, 146], "dev": [0, 1, 2, 5, 15, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 121, 122, 130, 143], "eth0": [0, 1, 2, 5, 11, 111], "rout": [0, 1, 2, 3, 4, 5, 13, 16, 22, 25, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101, 102, 108, 110, 112, 115, 125, 145, 146, 152], "echo": [0, 1, 2, 5], "nameserv": [0, 1, 2, 5, 11], "dn": [0, 1, 2, 5, 11, 15, 17, 25, 107, 121, 122, 130, 143], "server": [0, 1, 2, 3, 4, 5, 8, 10, 11, 13, 17, 19, 20, 22, 26, 27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 62, 63, 64, 65, 67, 68, 69, 70, 72, 73, 74, 75, 77, 78, 79, 80, 82, 83, 84, 85, 87, 88, 89, 90, 92, 93, 94, 95, 97, 98, 99, 100, 106, 107, 109, 111, 113, 116, 117, 119, 124, 127, 129, 131, 134, 137, 138, 139, 142, 145, 149, 150], "etc": [0, 1, 2, 3, 4, 5, 11, 18, 105, 106, 145, 151], "resolv": [0, 1, 2, 5, 15, 121, 122, 130, 143], "conf": [0, 1, 2, 5], "The": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 15, 19, 20, 21, 22, 23, 24, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 105, 106, 107, 111, 116, 121, 122, 130, 134, 135, 136, 137, 139, 141, 143, 144, 145, 149, 150], "cumulu": [0, 13, 104, 105], "imag": [0, 1, 2, 5, 11, 137], "should": [0, 1, 2, 3, 4, 8, 10, 11, 12, 19, 20, 21, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 108, 109, 111, 112, 113, 116, 117, 127, 129, 131, 135, 137, 139, 145, 146, 150, 152], "avail": [0, 1, 2, 3, 8, 10, 19, 21, 23, 25, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98, 105, 106, 137, 148, 150, 151, 152], "web": [0, 1, 2, 10, 15, 19, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 108, 109, 111, 113, 114, 116, 117, 121, 122, 127, 128, 130, 131, 133, 134, 136, 137, 138, 143, 144, 147, 148, 149], "which": [0, 1, 2, 4, 6, 15, 18, 19, 22, 23, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 107, 129, 137, 145, 151], "through": [0, 1, 2, 7, 12, 15, 21, 22, 23, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 87, 88, 91, 92, 93, 96, 97, 98, 101, 110, 113, 115, 119, 121, 122, 130, 132, 133, 134, 135, 136, 138, 139, 142, 143, 144, 147, 149, 150], "local": [0, 1, 2, 8, 14, 19, 22, 23, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106, 112, 134, 146, 149], "exampl": [0, 1, 2, 3, 6, 7, 8, 10, 11, 15, 17, 19, 20, 21, 23, 26, 28, 29, 30, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96, 98, 99, 100, 101, 105, 106, 107, 108, 111, 112, 113, 116, 121, 122, 130, 131, 134, 135, 136, 137, 139, 143, 144, 148, 149, 150, 151, 152], "nos": [0, 1, 2, 5, 106], "http": [0, 1, 2, 5, 8, 9, 10, 11, 15, 19, 21, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 121, 122, 130, 136, 137, 143], "192": [0, 1, 2, 10, 19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 144], "168": [0, 1, 2, 10, 19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 144], "100": [0, 1, 2, 10, 22, 23, 38, 41, 106], "10": [0, 1, 2, 3, 7, 23, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 102, 107, 111, 113, 116, 134, 135, 136, 137, 139, 144], "sonic_20211125_074752_ec202012_227": 0, "bin": [0, 1, 2, 5, 8, 137], "after": [0, 1, 2, 4, 7, 8, 11, 18, 19, 21, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 105, 106, 108, 111, 116, 136, 137, 145], "complet": [0, 1, 2, 3, 8, 10, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 111, 112, 116, 136, 137], "login": [0, 1, 2, 10, 11, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 137], "us": [0, 1, 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 17, 18, 20, 21, 22, 23, 26, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 83, 84, 87, 88, 89, 92, 93, 94, 97, 98, 99, 105, 106, 107, 108, 109, 111, 112, 113, 116, 117, 121, 122, 129, 130, 131, 136, 137, 141, 142, 143, 144, 145, 152], "usernam": [0, 1, 2, 5, 6, 12, 19], "password": [0, 1, 2, 5, 6, 11, 12, 15, 19, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 121, 122, 130, 137, 143], "admin": [0, 6, 7, 12, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 105, 106, 107, 113, 121, 122, 130, 131, 135, 137, 143, 144], "yourpassword": 0, "set": [0, 1, 2, 3, 5, 6, 8, 9, 11, 13, 15, 19, 21, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 110, 112, 113, 114, 115, 117, 121, 122, 129, 130, 131, 135, 136, 139, 140, 143, 144, 145, 146, 150, 151, 152], "up": [0, 1, 2, 3, 4, 5, 15, 19, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 108, 110, 112, 115, 121, 130, 136, 143, 145, 151], "out": [0, 1, 2, 3, 5, 8, 10, 18, 19, 23, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 106, 107, 136, 145], "band": [0, 1, 2, 5, 18, 23, 107], "oob": [0, 1, 2, 3, 5], "disabl": [0, 2, 3, 7, 13, 23, 105, 111, 152], "ztp": [0, 2], "y": [0, 4, 8, 137, 145], "establish": [0, 7, 19, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 108, 110, 112, 115, 134, 144, 149], "netri": [0, 1, 2, 5, 6, 7, 9, 10, 12, 14, 17, 18, 21, 23, 24, 25, 26, 27, 28, 29, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 98, 99, 105, 107, 108, 110, 112, 113, 114, 115, 120, 123, 125, 128, 131, 133, 134, 135, 137, 138, 139, 140, 141, 142, 144, 145, 146, 147, 148, 149, 150, 151, 152], "agent": [0, 1, 2, 4, 5, 7, 10, 12, 13, 25, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 107, 109, 111, 116, 117, 122, 127, 129, 136, 137, 143, 145], "navig": [0, 1, 2, 3, 4, 5, 10, 18, 19, 20, 23, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 107, 111, 116, 136, 144, 145, 148, 151], "net": [0, 1, 2, 3, 4, 5, 7, 10, 13, 18, 21, 23, 25, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 105, 107, 108, 109, 111, 112, 114, 116, 117, 119, 124, 126, 127, 128, 129, 131, 134, 136, 140, 141, 144, 145, 146, 149, 151], "inventori": [0, 1, 2, 3, 4, 5, 7, 12, 13, 25, 105, 106, 109, 111, 116, 117, 127, 136, 145, 148], "section": [0, 1, 2, 3, 4, 5, 6, 12, 19, 21, 23, 25, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 105, 107, 111, 113, 116, 131, 133, 135, 136, 138, 139, 144, 145, 146, 147, 148], "click": [0, 1, 2, 3, 4, 5, 6, 10, 18, 20, 21, 23, 25, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 107, 108, 109, 111, 112, 113, 114, 116, 117, 127, 128, 131, 144, 145, 148, 151], "three": [0, 1, 2, 3, 4, 5, 14, 23, 25, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 107, 111, 116, 135, 139, 145, 148], "vertic": [0, 1, 2, 3, 4, 5, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 111, 116, 145, 148], "dot": [0, 1, 2, 3, 4, 5, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 109, 111, 113, 116, 117, 127, 131, 136, 145, 148], "right": [0, 1, 2, 3, 4, 5, 6, 10, 15, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 107, 111, 116, 121, 122, 130, 131, 143, 144, 145, 148, 151], "side": [0, 1, 2, 3, 4, 5, 6, 19, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 108, 111, 112, 116, 129, 131, 145, 148, 150, 151], "you": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 12, 13, 15, 16, 17, 19, 20, 21, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 141, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152], "ar": [0, 1, 2, 3, 4, 5, 6, 8, 10, 12, 13, 15, 18, 19, 20, 21, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 105, 106, 107, 108, 109, 110, 111, 112, 113, 115, 116, 117, 118, 119, 120, 121, 122, 127, 130, 131, 134, 135, 136, 139, 141, 143, 144, 145, 146, 149, 152], "provis": [0, 1, 2, 5, 13, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 113, 117, 119, 124, 131, 135, 139, 140], "Then": [0, 1, 2, 4, 5, 7, 8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 109, 112, 113, 114, 117, 127, 128, 131, 135, 136, 139, 145], "copi": [0, 1, 2, 3, 4, 5, 6, 8, 10, 22, 109, 111, 114, 116, 117, 127, 128, 136, 145], "one": [0, 1, 2, 4, 5, 7, 13, 15, 17, 18, 19, 21, 22, 23, 24, 25, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 107, 109, 111, 113, 116, 117, 127, 131, 136, 137, 140, 145, 151], "line": [0, 1, 2, 3, 4, 5, 7, 17, 20, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 107, 111, 116, 145], "command": [0, 1, 2, 4, 5, 8, 10, 11, 15, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 105, 106, 109, 111, 116, 117, 127, 136, 145], "your": [0, 1, 2, 3, 4, 5, 8, 10, 11, 13, 15, 16, 17, 19, 23, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 107, 108, 110, 111, 112, 113, 114, 115, 116, 117, 121, 122, 128, 130, 131, 134, 135, 137, 139, 141, 142, 143, 144, 145, 146, 149, 150], "clipboard": [0, 1, 2, 3, 4, 5, 10, 111, 116, 145], "sudo": [0, 1, 2, 3, 4, 5, 11, 136, 137, 145], "linux": [1, 2, 3, 4, 10, 11, 13, 14, 15, 19, 21, 22, 26, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 107, 121, 122, 129, 130, 142, 143, 145], "3": [1, 7, 9, 25, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 102, 106, 107, 109, 117, 119, 122, 124, 127, 131, 136, 139, 152], "15": [1, 19, 30, 35, 36, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 102], "mlx": [1, 2], "amd64": [1, 2, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 137], "cumuluslinux": 1, "open": [1, 2, 5, 111, 116, 135, 139, 148, 151], "interfac": [1, 2, 3, 4, 5, 11, 13, 15, 19, 22, 23, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 107, 111, 116, 122, 136, 140, 143, 145], "file": [1, 2, 3, 5, 8, 11, 13, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 134, 136, 149], "other": [1, 2, 3, 4, 5, 6, 7, 8, 12, 18, 19, 22, 23, 25, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 88, 91, 93, 96, 98, 101, 105, 107, 108, 110, 111, 112, 113, 115, 116, 129, 131, 134, 135, 136, 139, 141, 142, 145, 149, 150, 152], "detail": [1, 2, 3, 5, 9, 10, 16, 20, 23, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 106, 116, 121, 122, 130, 143, 151], "vim": [1, 2, 3, 4, 5, 11, 145], "loopback": [1, 2, 3, 4, 5, 11, 18, 20, 23, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 107, 111, 116, 145], "auto": [1, 2, 3, 4, 5, 11, 26, 106, 145, 152], "lo": [1, 2, 3, 4, 5, 11, 145], "ifac": [1, 2, 3, 4, 5, 11, 145], "inet": [1, 2, 3, 4, 5, 11, 145], "primari": [1, 2, 5, 11, 112], "static": [1, 2, 3, 4, 5, 11, 13, 145], "length": [1, 2, 3, 4, 5, 11, 23, 111, 116, 145], "sourc": [1, 2, 3, 4, 5, 7, 11, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 106, 116, 134, 145, 149, 151], "d": [1, 2, 3, 4, 5, 8, 11, 23, 145], "tee": [1, 2], "ifreload": [1, 2, 5, 11], "licens": 1, "cl": 1, "past": [1, 4, 109, 111, 114, 116, 117, 127, 128, 136, 145], "string": [1, 23], "press": [1, 106, 135], "ctrl": 1, "5": [2, 3, 7, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 66, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 114, 117, 128, 131, 136, 137], "4": [2, 8, 11, 15, 19, 21, 22, 23, 27, 28, 30, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 66, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 108, 111, 116, 119, 121, 122, 124, 130, 136, 137, 143], "0": [2, 4, 7, 8, 10, 11, 19, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 106, 111, 113, 116, 134, 135, 136, 137, 139, 144, 145, 146, 149, 150], "upon": 2, "virtual": [2, 3, 13, 15, 25, 105, 111, 112, 113, 116, 119, 124, 126, 129, 131, 140, 152], "forward": [2, 7, 23, 102, 111, 113, 116, 131, 134, 149], "vrf": [2, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "mgmt": [2, 11, 106], "pleas": [2, 3, 4, 15, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 113, 121, 131, 132, 133, 136, 137, 138, 145, 147], "refer": [2, 3, 13, 15, 22, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98, 121, 122, 130, 143], "follow": [2, 3, 8, 9, 10, 12, 18, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 105, 106, 107, 111, 112, 116, 132, 136, 137], "instruct": [2, 8, 10, 15, 19, 23, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 121, 122, 130, 143], "exec": [2, 8, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97, 136], "bash": [2, 8, 136], "ensur": [2, 4, 15, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 111, 113, 116, 121, 122, 130, 136, 141, 143, 145, 150], "remov": [2, 3, 19, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 137, 145, 152], "might": [2, 8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "see": [2, 3, 4, 6, 7, 9, 11, 19, 21, 23, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 107, 108, 109, 113, 117, 121, 127, 131, 136, 145, 151], "time": [2, 7, 10, 17, 19, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 107, 136, 137, 151], "warn": 2, "output": [2, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 136], "can": [2, 4, 6, 7, 8, 12, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 108, 109, 110, 111, 112, 113, 115, 116, 117, 121, 122, 127, 129, 130, 131, 133, 134, 135, 136, 138, 139, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152], "ignor": 2, "cmd": 2, "usr": [2, 8], "lib": [2, 11], "helper": 2, "delet": [2, 3, 4, 9, 13, 19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 107, 145], "1001": [2, 112], "fail": [2, 107], "return": [2, 7, 19, 22, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "1": [2, 5, 9, 10, 11, 19, 25, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 106, 119, 122, 124, 126, 127, 135, 136, 137, 139, 143, 150], "cgroup": 2, "2": [3, 18, 22, 23, 25, 26, 27, 31, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106, 108, 109, 111, 112, 116, 117, 119, 122, 124, 126, 127, 135, 137, 139, 143, 152], "x": [3, 4, 22, 145], "intel": [3, 22], "xeon": [3, 102], "silver": 3, "processor": 3, "physic": [3, 4, 11, 12, 16, 19, 23, 25, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 107, 113, 131, 142, 145, 146, 150, 152], "core": [3, 4, 8, 11, 15, 22, 24, 102, 121, 122, 130, 143, 145], "per": [3, 6, 7, 12, 17, 23, 26, 107, 109, 117, 127, 135, 139, 151, 152], "socket": 3, "20": [3, 116, 150], "total": [3, 108], "128": [3, 43, 45, 46, 78, 80, 81, 116, 135], "gb": [3, 4, 8, 11, 15, 111, 116, 121, 122, 130, 143, 145], "64": [3, 8, 15, 19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 121, 122, 130, 143], "ram": [3, 4, 8, 11, 15, 111, 116, 121, 122, 130, 143, 145, 151], "multichannel": 3, "300": [3, 4, 145], "hdd": [3, 4, 145], "nvidia": [3, 13, 104], "mellanox": 3, "6": [3, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 71, 75, 80, 85, 90, 95, 100, 102, 106], "smartnic": [3, 22, 107], "card": [3, 22], "some": [3, 8, 19, 26, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 105, 112, 130, 135, 137], "recommend": [3, 4, 8, 11, 22, 23, 107, 111, 112, 121, 122, 130, 136, 142, 143, 145, 146, 150], "differ": [3, 6, 12, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 110, 115, 121, 122, 126, 130, 143, 151, 152], "vendor": [3, 137], "have": [3, 6, 8, 10, 12, 15, 18, 21, 22, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 111, 112, 113, 116, 122, 129, 134, 135, 136, 137, 139, 141, 143, 149, 150], "name": [3, 4, 6, 7, 15, 17, 18, 19, 21, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 109, 111, 112, 114, 116, 117, 121, 122, 128, 130, 131, 136, 143, 144, 145, 151, 152], "so": [3, 15, 19, 22, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 108, 111, 113, 135, 139, 143, 145, 151], "mainli": 3, "befor": [3, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 111, 116, 121, 122, 130, 135, 136, 143], "start": [3, 7, 11, 13, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 107, 109, 111, 117, 127, 129, 135, 136, 139, 151], "consid": [3, 23], "reset": 3, "all": [3, 4, 6, 7, 8, 12, 14, 17, 19, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 110, 111, 115, 116, 129, 135, 136, 137, 139, 142, 145, 150, 151], "power": [3, 151], "save": [3, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 106, 107, 111, 116, 148, 151], "perform": [3, 7, 13, 16, 18, 22, 25, 26, 109, 117, 127, 134, 136, 137, 146, 149], "tune": [3, 23], "cpu": [3, 4, 8, 11, 15, 22, 102, 111, 116, 121, 122, 130, 143, 145, 151], "p": [3, 8, 19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 135, 136], "state": [3, 7, 19, 21, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 111, 116, 135, 151, 152], "c3": [3, 109, 121], "report": [3, 22], "c6": 3, "polici": [3, 6, 12, 16, 22, 23, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 107], "enabl": [3, 13, 19, 22, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 108, 110, 115, 119, 124, 129, 134, 135, 139, 140, 148, 149, 150, 152], "turbo": 3, "boost": 3, "memori": 3, "frequenc": 3, "highest": 3, "number": [3, 6, 7, 8, 12, 19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 102, 107, 136, 137, 146, 151, 152], "NOT": 3, "when": [3, 6, 7, 8, 10, 12, 19, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 105, 106, 107, 109, 111, 112, 113, 116, 117, 127, 129, 131, 135, 139], "test": [3, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 102, 106], "function": [3, 4, 14, 22, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 107, 129, 135, 142, 145, 151], "nic": [3, 11, 22], "turn": [3, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 112, 129, 135, 139], "off": [3, 22, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "vt": 3, "hyper": 3, "thread": 3, "freshli": [3, 4, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 145], "ubuntu": [3, 4, 10, 11, 13, 25, 104, 109, 111, 116, 117, 127, 135, 137, 145], "18": [3, 5, 11, 19, 30, 31, 35, 40, 45, 46, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "04": [3, 4, 5, 11, 109, 111, 116, 117, 145], "lt": [3, 4, 145], "netplan": [3, 4, 145], "control": [3, 4, 6, 9, 10, 12, 13, 17, 23, 25, 26, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 86, 89, 94, 99, 105, 106, 107, 109, 112, 113, 114, 117, 118, 119, 120, 123, 124, 125, 127, 128, 129, 131, 134, 135, 136, 139, 140, 142, 144, 145, 146, 149, 150, 151, 152], "topologi": [3, 4, 10, 13, 19, 23, 25, 28, 29, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58, 59, 60, 63, 64, 65, 68, 69, 70, 73, 74, 75, 78, 79, 80, 84, 85, 88, 89, 90, 93, 94, 95, 98, 99, 100, 106, 145], "document": [3, 10, 18, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 106, 112, 133, 136, 138, 147], "here": [3, 4, 8, 10, 19, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 106, 111, 112, 114, 116, 117, 121, 122, 130, 136, 143, 145], "ad": [3, 4, 6, 13, 19, 20, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 106, 126, 145, 152], "creat": [3, 4, 6, 7, 8, 9, 10, 13, 15, 19, 23, 25, 26, 27, 29, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59, 60, 61, 62, 64, 65, 66, 67, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 99, 100, 101, 105, 107, 109, 110, 113, 114, 115, 117, 121, 122, 123, 125, 126, 127, 128, 129, 130, 131, 134, 136, 139, 140, 143, 145, 149, 151, 152], "newli": [3, 15, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 109, 113, 117, 121, 122, 127, 130, 131, 135, 139, 143], "run": [3, 8, 10, 15, 19, 22, 23, 25, 26, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106, 110, 115, 136], "ordinari": [3, 4, 18, 145], "user": [3, 4, 7, 10, 12, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 105, 107, 109, 110, 113, 115, 117, 121, 122, 127, 130, 131, 135, 137, 143, 144, 145, 151], "review": [3, 7, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 116], "ifupdown": [3, 4, 145], "verifi": [3, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 116, 136, 137], "present": [3, 4, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 136, 145], "correspond": [3, 109, 117, 127, 135, 136, 139, 150], "what": [3, 15, 18, 19, 22, 23, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 107, 114, 117, 121, 122, 128, 129, 130, 131, 141, 142, 143], "dure": [3, 22, 136, 145], "gener": [3, 7, 13, 14, 15, 17, 19, 21, 22, 23, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 107, 112, 114, 117, 121, 122, 130, 136, 137, 143], "base": [3, 6, 7, 12, 16, 18, 19, 20, 23, 25, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 113, 129, 131, 135, 139, 144], "initi": [3, 13, 25, 82, 106, 107, 111, 116, 135], "same": [3, 4, 14, 15, 20, 23, 111, 116, 121, 122, 130, 135, 139, 143, 145, 150], "No": [3, 106], "host": [3, 4, 7, 9, 11, 13, 14, 15, 18, 19, 21, 23, 25, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 108, 121, 122, 130, 135, 136, 139, 143, 150, 152], "ensz": [3, 4, 145], "comment": [3, 4, 7, 23, 145], "below": [3, 4, 7, 11, 14, 15, 18, 21, 23, 26, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 107, 108, 109, 111, 112, 113, 114, 116, 117, 119, 121, 122, 127, 128, 130, 131, 133, 134, 138, 143, 145, 147, 149], "locat": [3, 4, 8, 20, 23, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 109, 111, 112, 114, 116, 117, 128, 129, 130, 135, 137, 145, 146], "node": [3, 4, 8, 13, 15, 19, 21, 23, 25, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 83, 85, 86, 90, 91, 95, 96, 100, 101, 107, 108, 119, 121, 122, 124, 129, 130, 135, 139, 142, 143, 145, 146, 148, 150], "everyth": [3, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 135], "seem": 3, "ok": [3, 4, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 109, 117, 121, 122, 127, 130, 143, 145], "do": [3, 8, 10, 11, 18, 19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 111, 112, 114, 117, 128], "ani": [3, 4, 6, 7, 15, 18, 19, 21, 23, 24, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 111, 112, 113, 116, 121, 122, 129, 130, 131, 133, 135, 136, 138, 139, 141, 142, 143, 147, 150, 152], "addit": [3, 8, 13, 19, 20, 23, 25, 26, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 85, 87, 90, 92, 95, 97, 100, 116, 118, 119, 120, 144, 148], "than": [3, 22, 136], "those": [3, 18, 110, 111, 115, 127, 135, 144], "describ": [3, 4, 7, 11, 18, 21, 23, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 113, 119, 145], "abov": [3, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 111, 116], "boot": [3, 4, 145], "its": [3, 4, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 111, 116, 135, 144, 145], "heartbeat": [3, 4, 107, 109, 117, 127, 145], "go": [3, 4, 6, 15, 18, 19, 22, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 107, 108, 109, 111, 112, 113, 114, 116, 117, 121, 122, 127, 128, 130, 131, 143, 145], "critic": [3, 4, 8, 145, 148, 151], "telescop": [3, 4, 13, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 122, 136, 143, 145], "dashboard": [3, 4, 13, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 145], "color": [3, 4, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 145], "reflect": [3, 4, 136, 145], "health": [3, 4, 16, 20, 21, 22, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 145, 151], "8": [4, 8, 11, 15, 19, 22, 23, 28, 30, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 76, 80, 85, 90, 95, 100, 102, 106, 113, 121, 122, 130, 134, 143, 144, 145], "16": [4, 11, 19, 23, 30, 35, 40, 41, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 111, 113, 137, 144, 145, 149, 150], "22": [4, 30, 31, 97, 98, 101, 109, 111, 116, 117, 122, 134, 143, 145, 149], "ship": [4, 145], "two": [4, 6, 8, 13, 19, 23, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 107, 109, 112, 113, 117, 127, 129, 131, 140, 142, 145, 148, 151], "defin": [4, 6, 7, 11, 12, 13, 16, 17, 18, 21, 22, 25, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 88, 91, 93, 96, 98, 101, 106, 107, 111, 114, 116, 117, 128, 129, 141, 144, 145, 146, 148, 150], "site": [4, 7, 12, 13, 19, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 107, 109, 112, 114, 117, 121, 128, 129, 130, 131, 135, 139, 140, 144, 145, 146, 148, 150, 152], "softgate1": [4, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 109, 117, 127, 145], "softgate2": [4, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 145], "we": [4, 19, 22, 23, 25, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106, 107, 108, 109, 111, 112, 113, 116, 117, 121, 127, 129, 130, 131, 135, 137, 139, 142, 143, 145], "new": [4, 8, 10, 19, 21, 23, 25, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 105, 106, 107, 108, 111, 112, 114, 116, 117, 119, 121, 122, 124, 128, 129, 130, 131, 136, 137, 139, 143, 144, 145, 148, 150, 151, 152], "altern": [4, 21, 23, 144, 145, 146, 150], "learn": [4, 13, 23, 27, 29, 31, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 99, 101, 106, 145], "how": [4, 11, 13, 21, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98, 106, 108, 113, 145], "keep": [4, 7, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 145], "mind": [4, 7, 145], "uniqu": [4, 6, 7, 8, 18, 20, 21, 23, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 111, 116, 135, 139, 145, 150, 152], "each": [4, 7, 12, 15, 18, 19, 20, 22, 23, 25, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 105, 106, 107, 109, 114, 116, 117, 122, 126, 128, 129, 135, 136, 139, 143, 145, 146, 148, 150, 151, 152], "note": [4, 8, 13, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 105, 111, 113, 116, 126, 135, 136, 139, 144, 145, 150], "replac": [4, 8, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 134, 145, 149], "regular": [4, 13, 22, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 127, 129, 145], "attempt": [4, 137, 145], "migrat": [4, 23, 145], "prior": [4, 8, 111, 116], "handoff": [4, 145], "bond0": [4, 111, 116, 145, 148], "necessari": [4, 16, 22, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 87, 88, 92, 93, 97, 98, 111, 116, 129, 145, 146, 150], "subinterfac": [4, 135, 139, 145, 148, 150], "under": [4, 6, 7, 12, 20, 21, 23, 26, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 86, 87, 91, 92, 96, 97, 101, 108, 109, 111, 112, 113, 114, 116, 117, 127, 128, 131, 134, 135, 136, 139, 141, 145, 146, 149, 150, 151, 152], "xyz": [4, 145], "But": [4, 145], "bind": [4, 11, 23, 145], "onli": [4, 6, 7, 13, 15, 18, 22, 23, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 105, 106, 108, 113, 114, 117, 128, 129, 131, 135, 145, 151, 152], "make": [4, 8, 11, 19, 22, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 85, 87, 90, 92, 95, 97, 100, 106, 113, 129, 131, 135, 136, 137, 139, 145, 152], "chang": [4, 7, 11, 15, 19, 22, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 109, 111, 113, 116, 117, 121, 122, 129, 130, 131, 136, 141, 143, 145, 146, 148, 152], "remain": [4, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 145], "trunk": [4, 111, 116, 142, 145], "en": [4, 145], "more": [4, 20, 22, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 113, 131, 134, 135, 139, 145, 149], "bond": [4, 145], "ensx": [4, 145], "actual": [4, 23, 26, 145, 151], "slave": [4, 145], "adjust": [4, 17, 107, 145], "mode": [4, 23, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 105, 145, 152], "accord": [4, 22, 107, 145], "desir": [4, 10, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 107, 111, 116, 129, 131, 144, 145], "activ": [4, 7, 13, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 112, 119, 124, 134, 144, 145, 149, 152], "backup": [4, 14, 112, 136, 145], "maintain": [4, 23, 145], "custom": [5, 6, 7, 13, 17, 22, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 98, 99, 101, 107, 151], "download": [5, 11, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 134, 137, 149], "ai": [5, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "newnet0p": [5, 11, 12, 15, 121, 122, 130, 143], "descript": [6, 7, 17, 19, 23, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 106, 107, 109, 111, 114, 116, 117, 127, 128, 131, 144, 148, 151], "field": [6, 7, 10, 13, 21, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 106, 111, 113, 114, 116, 117, 127, 128, 131, 135, 144, 148, 151], "full": [6, 22, 23, 26, 146], "e": [6, 8, 12, 23, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 108, 112, 136, 137, 139, 146, 148, 151], "mail": [6, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "email": [6, 7, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101], "also": [6, 7, 15, 19, 22, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 109, 111, 113, 117, 127, 129, 135, 139, 144, 148, 151], "notif": [6, 7], "retriev": [6, 111, 116], "cc": 6, "send": [6, 22, 23, 112], "thi": [6, 7, 8, 10, 15, 17, 18, 19, 20, 21, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 106, 107, 108, 109, 110, 111, 112, 113, 115, 116, 117, 119, 121, 122, 127, 130, 131, 132, 134, 135, 136, 137, 139, 141, 143, 146, 148, 149, 150, 152], "phone": 6, "compani": [6, 26], "work": [6, 8, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 111], "usual": [6, 23, 107], "multi": [6, 13, 140, 152], "where": [6, 7, 8, 15, 21, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 107, 112, 116, 122, 134, 135, 139, 143, 149, 152], "provid": [6, 7, 10, 11, 13, 14, 15, 17, 18, 19, 21, 25, 28, 29, 30, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96, 98, 99, 100, 101, 105, 107, 109, 110, 111, 113, 115, 116, 117, 127, 131, 132, 134, 135, 139, 144, 149, 151, 152], "posit": [6, 107], "within": [6, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98, 108, 116, 134, 144, 149], "object": [6, 7, 13, 19, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "rbac": [6, 7, 12], "deactiv": 6, "view": [6, 13, 18, 26, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 82, 86, 91, 96, 101, 111, 116, 146, 151, 152], "edit": [6, 7, 11, 19, 20, 23, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 107, 111, 113, 116, 131, 135, 148, 152], "part": [6, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 106, 107, 129, 135, 139, 142, 151], "switch": [6, 7, 8, 10, 12, 16, 17, 18, 19, 20, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 111, 114, 116, 117, 128, 136, 137, 142, 151, 152], "resourc": [6, 7, 8, 12, 13, 23, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 105, 111, 113, 116, 131, 135, 144], "assign": [6, 12, 21, 23, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 105, 107, 111, 116, 129, 131, 150, 151, 152], "variou": [6, 23, 25, 102, 129, 151], "read": [6, 113, 114, 117], "form": [6, 22, 23, 25, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 146], "list": [6, 12, 13, 17, 18, 19, 20, 21, 23, 26, 29, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 89, 90, 91, 94, 95, 96, 99, 100, 101, 105, 106, 107, 109, 116, 117, 122, 127, 135, 139, 143, 151, 152], "own": [6, 12, 111, 116, 135], "concept": [6, 120, 123, 125, 126, 140], "share": [6, 12, 23, 26], "deleg": [6, 12, 113, 131, 144], "over": [6, 12, 15, 22, 23, 25, 26, 112, 113, 122, 135, 139, 143, 146, 150, 152], "typic": [6, 12, 23, 107, 129, 142, 146], "team": [6, 7, 12, 152], "grant": [6, 12], "request": [6, 7, 12, 19, 21, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 108, 109, 112, 117, 119, 121, 122, 124, 127, 130, 131, 133, 135, 138, 143, 147], "self": [6, 8, 12, 23], "portal": [6, 12, 114, 117, 128, 133, 138, 147], "programmat": [6, 12], "kubernet": [6, 9, 10, 12, 13, 14, 18, 21, 22, 25, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 113, 131, 133, 138, 144, 147], "crd": [6, 12, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 133, 138, 147], "devop": [6, 12], "netop": [6, 12, 16, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "pipelin": [6, 12], "just": [6, 8, 19, 22, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 113], "basi": [6, 7, 12, 152], "attach": [6, 12, 17, 23, 26, 107, 129, 152], "individu": [6, 12, 15, 20, 23, 26, 114, 117, 122, 128, 143, 146, 152], "everi": [6, 20, 23, 25, 26, 107, 135, 139], "attribut": [6, 12, 23, 25, 146, 151], "particular": [6, 19, 21, 22, 23, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 109, 117, 139, 151, 152], "specif": [6, 7, 18, 19, 23, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 111, 116, 134, 137, 149], "either": [6, 7, 8, 16, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 87, 88, 92, 93, 97, 98, 113, 131, 134, 135, 136, 144, 149], "link": [6, 12, 13, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 95, 100, 106, 112, 140, 142, 151], "directli": [6, 19, 22, 23, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 135], "support": [7, 8, 9, 11, 13, 22, 23, 24, 25, 26, 105, 106, 111, 113, 116, 118, 119, 120, 131, 134, 137, 144, 149, 151, 152], "acl2": 7, "destin": [7, 17, 20, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 107, 110, 115, 116, 134, 149], "format": [7, 23, 152], "orient": 7, "wai": [7, 8, 15, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 110, 115, 121, 122, 130, 135, 139, 143], "both": [7, 19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 108, 111, 112, 113, 131], "tenant": [7, 12, 13, 18, 19, 21, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 105, 106, 107, 111, 113, 116, 131, 135, 144, 151, 152], "execut": [7, 8, 23, 106, 136, 137], "hardwar": [7, 13, 18, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 95, 100, 102, 106, 107, 111, 116, 130, 140, 151], "rate": [7, 20], "secur": [7, 16, 17, 22, 25, 107, 110, 111, 115, 116, 118, 119, 120, 124, 140], "enforc": 7, "It": [7, 8, 15, 18, 19, 22, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 106, 111, 113, 122, 127, 131, 134, 143, 144, 149], "import": [7, 12, 13, 15, 113, 121, 122, 130, 131, 143], "limit": [7, 23], "size": [7, 102], "tcam": 7, "screenshot": [7, 20, 21, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 108, 109, 113, 117, 127, 131, 133, 138, 147, 151], "util": [7, 22, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 111, 112, 116, 142, 151], "seen": [7, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "appli": [7, 8, 16, 19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 111, 116, 134, 149, 150], "sever": [7, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 135, 139], "optim": [7, 26, 129, 142], "algorithm": 7, "minim": [7, 11, 113, 121], "usag": [7, 18, 22, 106, 129], "while": [7, 15, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 121, 122, 130, 136, 143], "achiev": [7, 111, 116, 148], "permit": [7, 19, 20, 23, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106], "commun": [7, 13, 22, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 107, 110, 115, 118, 119, 120, 129, 142, 148], "featur": [7, 19, 23, 25, 107, 111, 135, 139, 144], "deni": [7, 17, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 107], "given": [7, 20, 21, 23, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 152], "drop": [7, 8, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 136], "traffic": [7, 17, 20, 22, 23, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 107, 108, 110, 111, 115, 116, 129, 134, 135, 136, 139, 142, 149, 150, 151, 152], "unless": [7, 152], "sitedefault": 7, "entri": [7, 12, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 111, 116], "protocol": [7, 19, 20, 21, 22, 23, 26, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 116, 129], "match": [7, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 116], "tcp": [7, 19, 20, 21, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 122, 133, 138, 143, 147], "udp": [7, 21, 23, 133, 138, 147], "icmp": [7, 23], "ipv4": [7, 12, 13, 17, 20, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 107, 111, 112, 116, 152], "code": 7, "icmpv6": 7, "ipv6": [7, 12, 13, 17, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 87, 91, 92, 96, 97, 101, 107, 152], "until": [7, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 109, 117, 127], "date": [7, 8, 136, 137], "action": [7, 16, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 105, 111, 116, 148], "packet": [7, 22, 23, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 113, 129, 131, 142, 151], "revers": 7, "For": [7, 10, 21, 23, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 105, 106, 107, 109, 111, 113, 114, 116, 117, 127, 128, 135, 136, 139, 148], "except": [7, 17, 107, 135], "syn": 7, "flag": 7, "non": [7, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107], "swap": 7, "type": [7, 8, 12, 13, 18, 21, 23, 24, 25, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 106, 107, 111, 113, 114, 116, 117, 127, 128, 130, 131, 134, 144, 148, 149, 151], "rang": [7, 18, 23, 105, 108, 116, 129, 135, 141, 142, 148, 150], "window": [7, 18, 22, 23, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 135, 139, 148], "group": [7, 12, 13, 111, 150], "end": [7, 20, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 112, 152], "commonli": [7, 26], "check": [7, 13, 19, 20, 21, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 107, 133, 138, 140, 146, 147, 151], "button": [7, 18, 20, 23, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 82, 88, 93, 98, 107, 111, 116, 131, 135, 139, 144], "anoth": [7, 22, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 111, 116, 135, 139], "alreadi": [7, 10, 11, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 128], "24": [7, 10, 19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 111, 116, 135, 139, 149, 150], "ssh": [7, 17, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 107, 109, 111, 116, 117, 122, 127, 134, 135, 136, 137, 143, 149], "show": [7, 23, 25, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 85, 87, 90, 92, 95, 97, 100, 107, 111, 112, 116, 137, 151], "broader": [7, 23], "respons": [7, 22, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 83, 86, 91, 96, 101, 129], "repres": [7, 22], "qa_ten": 7, "belong": [7, 19, 21, 23, 26, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 90, 91, 95, 96, 100, 101, 105, 107], "stai": 7, "wait": [7, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 109, 111, 116, 117, 127], "receiv": [7, 22, 23, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 111, 146], "gui": [7, 11, 12, 21, 22, 23, 25, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 83, 84, 87, 92, 97], "reject": 7, "soon": [7, 17, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 107], "push": 7, "appropri": [7, 23, 111, 113, 116, 131, 135, 136, 139, 144], "config": [7, 11, 25, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "throughout": 7, "fabric": [7, 20, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 106, 111, 114, 116, 117, 128, 152], "rest": [7, 18, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97, 108, 114, 117, 128], "disk": [8, 11, 15, 121, 122, 130, 143, 151], "50gb": [8, 15, 121, 122, 130, 143], "bit": [8, 15, 121, 122, 130, 143, 151], "k3": [8, 10, 15, 121, 122, 130, 143], "expect": [8, 26, 137, 145], "most": [8, 26, 113, 121, 130, 131, 134, 144, 146, 149], "modern": [8, 24, 25, 26], "oss": 8, "raspbian": 8, "buster": 8, "step": [8, 9, 11, 15, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 106, 108, 109, 110, 112, 115, 117, 120, 122, 124, 127, 132, 136, 137, 140], "legaci": [8, 16, 24, 26], "iptabl": 8, "alpin": 8, "setup": [8, 13, 17, 23, 107, 129, 145], "red": [8, 136], "hat": 8, "cento": 8, "enterpris": 8, "curl": [8, 10, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 121, 122, 130, 136, 143], "sfl": [8, 10, 15, 121, 122, 130, 136, 143], "io": [8, 9, 10, 11, 15, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 121, 122, 130, 136, 143], "sh": [8, 10, 15, 121, 122, 130, 136, 143], "abl": [8, 15, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 112, 113, 121, 122, 130, 131, 135, 143], "log": [8, 13, 19, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 111, 116], "script": [8, 145], "doe": [8, 10, 141, 143], "cert": [8, 11], "helm": [8, 10, 13, 14, 15, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 121, 122, 130, 136, 143], "chart": [8, 10, 13, 14, 15, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 121, 122, 130, 143, 151], "In": [8, 9, 15, 17, 19, 23, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 107, 108, 109, 111, 112, 113, 116, 117, 121, 122, 127, 130, 131, 134, 135, 136, 137, 139, 143, 144, 146, 148, 149, 150], "order": [8, 9, 13, 19, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 107, 111, 116, 135, 139], "ingress": [8, 9, 116, 152], "ctl": [8, 15, 121, 122, 130, 136, 143], "hostnam": [8, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 121, 122, 130, 143], "argument": [8, 106], "com": [8, 9, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 121, 122, 143], "sign": 8, "certif": [8, 15, 121, 122, 130, 143], "box": [8, 16, 18, 20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "pass": [8, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 129], "cluster": [8, 10, 15, 19, 25, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 121, 122, 130, 143], "valu": [8, 9, 18, 19, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 111, 114, 116, 117, 128, 141], "clusterissu": 8, "selfsign": 8, "letsencrypt": [8, 121, 122, 130, 143], "acm": 8, "http01": 8, "challeng": 8, "valid": [8, 23, 106], "proce": [8, 108, 111, 116, 144], "successfulli": [8, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 111, 112, 116, 136, 144], "cname": 8, "record": [8, 15, 121, 122, 130, 143], "domain": [8, 9, 15, 17, 24, 107, 121, 122, 130, 143], "subdomain": 8, "exist": [8, 10, 13, 20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 110, 115, 116, 129, 139, 142, 144, 148], "must": [8, 9, 18, 105, 106, 111, 112, 116, 121, 122, 130, 143, 144], "simplest": 8, "issu": 8, "behind": [8, 23, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 108, 152], "fqdn": [8, 25], "public": [8, 15, 23, 25, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 107, 108, 111, 116, 119, 121, 122, 124, 127, 129, 130, 131, 134, 135, 140, 143, 149, 151], "common": [8, 17, 18, 19, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 107, 113, 144], "approach": [8, 16], "privat": [8, 11, 18, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 107, 111, 112, 113, 116, 129, 131, 134, 135, 144, 146, 149], "deploy": [8, 10, 12, 15, 25, 122, 143, 145], "dns01": 8, "familiar": [8, 106, 136], "specifi": [8, 18, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 111, 116, 152], "yaml": [8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "suitabl": 8, "kubectl": [8, 9, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 136], "f": [8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 91, 95, 100], "my": [8, 9, 15, 23, 106, 121, 122, 130, 143, 151], "rerun": 8, "latest": [8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "version": [8, 19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 136, 137], "simpli": [8, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 133, 135, 136, 138, 147], "newer": 8, "updat": [8, 9, 23, 25, 134, 136, 137, 149], "few": [8, 19, 23, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 108, 109, 113, 117, 127, 135], "minut": [8, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 108, 109, 117, 127, 136], "store": [8, 22, 106], "data": [8, 12, 13, 22, 23, 24, 25, 26, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106, 107, 137, 151], "mariadb": [8, 136], "highli": [8, 22, 129, 136, 142], "cronjob": 8, "mysqldump": [8, 136], "take": [8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 111, 116, 136, 137, 144, 150], "databas": [8, 22, 134, 135, 136, 139, 149], "snapshot": [8, 136], "n": [8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 136], "c": [8, 26, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 136], "u": [8, 19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 111, 136], "mariadb_us": [8, 136], "mariadb_password": [8, 136], "mariadb_databas": [8, 136], "db": [8, 136], "m": [8, 15, 21, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 113, 131, 134, 149], "h": [8, 135], "sql": [8, 136], "find": [8, 20, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 83, 87, 92, 97, 107, 113, 131, 135], "yyyi": 8, "mm": 8, "dd": 8, "hh": 8, "ss": 8, "directori": [8, 13, 136], "re": [8, 106], "move": [8, 14, 15, 23, 107, 108, 112, 121, 122, 129, 130, 142, 143], "reinstal": 8, "sens": [8, 109, 113, 117, 127, 129, 131], "purpos": [8, 18, 21, 23, 25, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 106, 107, 111, 113, 116, 126, 134, 144, 149], "futur": [8, 18, 109, 113, 117, 127, 131, 135, 152], "overwis": 8, "reiniti": 8, "devic": [8, 17, 22, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 105, 107, 111, 112, 116, 136], "grpc": [8, 22, 122, 136, 143], "jsonpath": 8, "mysql": [8, 136], "root": [8, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 109, 117, 136, 137], "mariadb_root_password": [8, 136], "contain": [8, 10, 19, 22, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 116, 136], "cp": [8, 136], "opt": [8, 136], "want": [8, 10, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 111, 112, 113, 131, 134, 141, 144, 148, 149, 151], "too": [8, 109, 117, 127], "place": [8, 25, 107, 137], "old_secret": 8, "environ": [8, 13, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 106, 110, 115, 149, 150], "taken": [8, 111, 116, 136], "old": [8, 23, 26, 137, 152], "export": [8, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "vudodffsakjcu2lfvva4t1c0cnpuumdimkqxem85y2dns3pkajlnsg": 8, "patch": [8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "json": [8, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "op": [8, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "path": [8, 21, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "restart": [8, 11, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 136], "microservic": 8, "rollout": 8, "12": [9, 31, 101, 137, 144], "pv": 9, "provision": 9, "underli": [9, 107], "infrastructur": [9, 13, 135, 139], "repositori": [9, 11], "netrisai": [9, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "github": [9, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "namespac": [9, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "releas": [9, 13, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 152], "readm": 9, "about": [9, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 87, 88, 92, 93, 97, 98, 106, 113, 129, 131], "paramet": [9, 23, 106, 114, 116, 117, 128, 152], "offer": [10, 102], "simplifi": 10, "model": [10, 23], "who": [10, 107, 144, 152], "quickli": [10, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "shortest": 10, "amount": 10, "streamlin": 10, "wish": [10, 116], "instead": [10, 16, 19, 23, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 111, 116, 134, 148, 149], "w": [10, 19, 135], "ui": [10, 19, 23, 26, 105, 106], "credenti": [10, 11, 15, 19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 106, 121, 122, 128, 130, 143], "ipam": [10, 13, 18, 21, 23, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 88, 91, 93, 96, 98, 101, 106, 111, 113, 116, 126, 129, 140, 146, 152], "subnet": [10, 12, 13, 17, 23, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 106, 107, 110, 113, 115, 126, 134, 139, 144, 146, 149, 150], "softgat": [10, 13, 17, 18, 19, 21, 23, 25, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 85, 86, 88, 91, 93, 96, 98, 101, 106, 108, 110, 112, 113, 115, 119, 123, 124, 125, 129, 131, 134, 135, 136, 139, 140, 142, 146, 149, 150], "plan": [10, 13, 107], "upper": [10, 129], "fill": [10, 18, 23, 26, 107, 112], "proper": [10, 23, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106, 135, 150], "selector": 10, "congratul": [10, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "now": [10, 19, 25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 108, 112, 113, 114, 117, 128, 131, 135, 151], "100gb": 11, "qemu": 11, "apt": 11, "virt": 11, "contact": 11, "permiss": [11, 12, 13], "cd": 11, "var": [11, 137], "libvirt": 11, "wget": 11, "img": 11, "controller3": 11, "qcow2": 11, "definit": [11, 13, 107], "xml": 11, "virsh": 11, "br": 11, "correct": [11, 136], "bridg": [11, 152], "autostart": 11, "By": [11, 17, 26, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106, 107, 111, 148, 151], "obtain": [11, 135, 137, 150], "forget": 11, "passwd": 11, "reload": 11, "sure": [11, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 134, 136, 137, 149], "browser": [11, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 137], "nginx": [11, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "pem": 11, "kei": [11, 19, 23, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 114, 117, 137], "systemctl": [11, 105, 136], "nomenclatur": 12, "understand": [12, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "account": [12, 13, 15, 121, 122, 130, 143], "restapi": [12, 22], "role": [12, 13, 18, 113, 131, 135, 144], "separ": [12, 106, 113, 126, 135, 148], "center": [12, 13, 22, 23, 24, 26, 107], "unit": [12, 23, 105, 107], "come": [12, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 144], "preconfigur": [12, 149, 150], "global": [12, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 107, 135, 139, 150], "AS": [12, 19, 23, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 107, 112, 146], "acl": [12, 13, 29, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 113, 134, 135, 139, 149, 150], "mesh": [12, 13, 19, 22, 23, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "vpn": [12, 22, 23, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "bgp": [12, 13, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 106, 112, 113, 119, 124, 139, 140, 151, 152], "extern": [12, 19, 22, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 116, 135, 151], "peer": [12, 13, 19, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 108, 112, 146], "ibgp": 12, "ebgp": [12, 112], "vpc": [13, 108, 110, 112, 113, 115, 123, 125, 131, 134, 135, 144, 149], "join": [13, 118, 119, 120], "our": [13, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 118, 119, 120, 133, 138, 147], "slack": [13, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 118, 119, 120, 136], "channel": [13, 118, 119, 120], "engin": [13, 22, 118, 119, 120], "upgrad": [13, 14, 109, 117, 118, 127], "rollback": [13, 118], "procedur": [13, 118, 129, 142], "anywher": [13, 26, 111, 152], "guid": [13, 21, 110, 111, 115, 116, 132], "overview": [13, 120, 123, 125, 126, 140], "upstream": [13, 23, 27, 29, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 84, 87, 89, 92, 94, 97, 99, 107, 140], "nat": [13, 18, 22, 25, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 107, 108, 119, 124, 126, 129, 135, 139, 140, 150], "v": [13, 18, 23, 25, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 105, 112, 119, 124, 126, 129, 140, 144, 151], "isol": [13, 119, 124, 126, 129, 140, 142], "demand": [13, 19, 21, 25, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 119, 124, 126, 129, 140], "elast": [13, 119, 124, 126, 129, 140], "l4": [13, 19, 25, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 119, 124, 126, 129, 140], "load": [13, 16, 18, 19, 22, 23, 25, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 108, 119, 124, 126, 129, 135, 137, 139, 140, 151], "balanc": [13, 16, 18, 19, 22, 23, 25, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 108, 119, 124, 126, 129, 140, 151], "experiment": [13, 140], "equinix": [13, 113], "metal": [13, 26, 112, 113, 127, 129, 131, 132, 134, 135], "api": [13, 19, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 119, 124, 129, 135, 139], "integr": [13, 21, 22, 25, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 119, 124, 129, 142], "project": [13, 112, 113, 114, 117, 124], "pool": [13, 18, 21, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 107, 108, 119, 124], "phoenixnap": [13, 131], "bmc": [13, 131], "aw": [13, 15], "deploi": [13, 14, 15, 18, 19, 28, 29, 31, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 84, 86, 88, 89, 91, 93, 94, 96, 98, 99, 101, 106, 121, 122, 123, 125, 129, 130, 132, 143], "gcp": 13, "method": [13, 23, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 133, 138, 147, 151], "manifest": [13, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "loadbalanc": [13, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "introduct": [13, 120, 123, 125, 126, 140], "l4lb": [13, 18, 25], "reclaim": 13, "calico": [13, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99], "cni": [13, 25, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99], "terraform": [13, 113, 131, 133, 135, 138, 139, 144, 147], "prepar": 13, "architectur": [13, 23, 26, 144, 146], "unmanag": [13, 24], "scalabl": 13, "v5": [13, 104], "switchdev": [13, 25, 104], "edgecor": [13, 104, 137], "sonic": [13, 104], "v3": [13, 104, 139], "7": [13, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 71, 75, 80, 85, 90, 95, 100, 102, 104], "minimum": [13, 111, 116, 130, 140], "softwar": [13, 16, 22, 110, 111, 115, 116, 121, 122, 130, 134, 137, 140, 143, 149], "pro": [13, 109, 117, 127], "bio": 13, "alloc": [13, 21, 23, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 91, 96, 101, 106, 111, 116, 126, 127, 129, 130, 140], "profil": [13, 25, 26], "hairpin": [13, 25], "basic": [13, 106, 113], "advanc": [13, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 116], "map": 13, "rule": [13, 17, 25, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 83, 87, 88, 92, 93, 97, 98, 102, 107, 111, 113, 116, 131, 134, 135, 144, 149, 150], "sitemesh": 13, "look": [13, 19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 111, 112, 116, 146], "glass": [13, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98, 146], "l3": [13, 25, 26, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 86, 89, 91, 94, 96, 99, 101, 106, 107], "anycast": [13, 25, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 86, 89, 91, 94, 96, 99, 101], "lb": [13, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 151], "consum": [13, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 113, 129, 131, 133, 135, 138, 142, 144, 147], "approv": [13, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "workflow": 13, "process": [13, 15, 19, 22, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 110, 115, 121, 122, 130, 132, 136, 137, 143], "roh": [13, 18, 20, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 86, 88, 91, 93, 96, 98, 101, 107], "visibl": [13, 23, 135, 139], "graph": 13, "board": 13, "intro": [13, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99], "applic": [13, 14, 19, 20, 22, 25, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 107, 136], "On": [13, 25, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 86, 89, 94, 99, 112], "vm": [14, 22, 23, 26, 113, 115, 121, 125, 131, 134, 135, 139, 150], "cloud": [14, 16, 18, 19, 21, 22, 25, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 110, 115, 116, 129, 132, 133, 138, 147], "prem": [14, 18, 22, 25], "anytim": 14, "ONE": 14, "restor": [14, 136], "repo": 14, "info": [14, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 108, 121, 122, 130, 135, 143], "almost": [15, 121, 122, 130, 143], "mai": [15, 19, 26, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 112, 113, 122, 129, 131, 141, 142, 143, 144], "fact": [15, 26, 122, 143], "multipl": [15, 20, 21, 22, 105, 121, 122, 130, 135, 139, 143, 150, 151, 152], "doesn": [15, 26, 107, 122, 131, 134, 143, 149], "t": [15, 18, 19, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 113, 122, 129, 131, 134, 135, 139, 143, 149, 152], "matter": [15, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 107, 118, 119, 120, 124, 131, 140], "am": [15, 134, 149], "machin": [15, 23, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 121, 122, 130, 143], "ec2": [15, 110, 123], "got": [15, 25, 121, 130, 143], "54": [15, 111, 122, 143], "219": 15, "211": 15, "71": 15, "like": [15, 16, 18, 19, 23, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 108, 112, 113, 131, 134, 139, 144, 149], "easier": [15, 121, 122, 130, 143], "potenti": [15, 107, 121, 122, 130, 136, 143], "somewher": [15, 121, 122, 130, 143], "cloudflar": [15, 121, 122, 130, 143], "point": [15, 23, 106, 109, 112, 113, 117, 127, 135, 139], "inde": [15, 121, 130, 143], "liner": [15, 109, 117, 127, 136], "stand": [15, 121, 130, 143], "top": [15, 16, 23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 87, 88, 92, 93, 97, 98, 111, 116, 121, 122, 130, 131, 143, 144, 151], "let": [15, 16, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 111, 116, 121, 122, 130, 135, 139, 143], "encrypt": [15, 22, 23, 25, 121, 122, 129, 130, 142, 143], "ssl": [15, 121, 122, 130, 143], "That": [15, 113, 121, 122, 129, 130, 131, 135, 139, 143], "why": [15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 108, 121, 122, 130, 143], "finish": [15, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 109, 111, 116, 117, 121, 122, 130, 136, 143], "consonl": 15, "immedi": [15, 149], "someth": [15, 112, 113, 131], "strong": 15, "whitelist": 15, "restrict": [15, 122, 143], "continu": [16, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 135], "monitor": [16, 107, 122, 135, 143, 151], "remedi": [16, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "inform": [16, 22, 23, 106, 112, 148], "human": 16, "intervent": 16, "abstract": 16, "awai": 16, "complex": [16, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "effici": [16, 110, 115, 135, 139], "down": [16, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 105, 151], "allow": [17, 18, 19, 23, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 82, 83, 85, 88, 90, 93, 95, 98, 100, 107, 110, 111, 115, 116, 134, 135, 142, 149, 152], "harden": [17, 107], "flow": [17, 107, 129, 134, 142, 149], "As": [17, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 111, 116, 135, 150], "includ": [17, 18, 21, 23, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 83, 86, 91, 96, 101, 107, 135, 137, 139, 144, 149, 150], "ntp": [17, 25, 107], "free": [17, 19, 23, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 107, 108, 141], "text": [17, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 107], "timezon": [17, 25, 107], "track": 18, "design": [18, 19, 22, 25, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 134, 142, 146, 149], "tree": [18, 19, 26], "opportun": 18, "kind": [18, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "There": [18, 19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 106, 107, 129], "main": [18, 22, 25, 106, 107, 111, 116, 134, 135, 149], "organ": [18, 23, 25, 152], "rir": 18, "lir": 18, "alwai": [18, 23, 113, 116, 131, 134, 149], "child": 18, "parent": 18, "bottom": [18, 20, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98], "overlap": [18, 141], "owner": [18, 19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 105, 107, 152], "ust": 18, "experi": [18, 19], "itself": 18, "inact": [18, 111, 116, 131], "reserv": [18, 22, 25, 131, 142], "kube": [19, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "complement": [19, 25], "between": [19, 20, 22, 23, 25, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 107, 108, 110, 111, 113, 115, 129, 131, 135, 136, 139, 142, 146, 150, 152], "accomplish": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 112, 116], "master": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 136], "secret": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 128], "nnetri": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "cred": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "liter": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "inspect": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "pod": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 136], "l": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100], "demonstr": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100], "success": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 106, 107], "level": [19, 22, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 151], "1629994653": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "6441543": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "logger": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "msg": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "worker": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "reconcilergroup": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "k8": [19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101], "reconcilerkind": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "count": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 102, 107, 137, 151], "plane": [19, 22, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 152], "scenario": [19, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 131, 134, 135, 136, 139, 146, 149], "simpl": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100], "podinfo": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "k": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "stefanprodan": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "kustom": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "po": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "svc": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "clusterip": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "statu": [19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 107, 111, 112, 116, 136], "ag": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 136], "576d5bf6bd": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "7z9jl": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "49": [19, 30, 35, 40, 45, 50, 55, 60, 65, 68, 70, 75, 80, 85, 90, 95, 100, 137], "nhlmh": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "33": [19, 30, 35, 40, 45, 50, 55, 60, 63, 65, 70, 75, 80, 85, 90, 95, 100], "172": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 91, 95, 100, 106, 111, 144, 149, 150], "21": [19, 27, 30, 31, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 85, 87, 90, 92, 95, 97, 100, 102], "65": [19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 71, 75, 80, 85, 90, 95, 100], "106": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 96, 100], "none": [19, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 112], "9898": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "9999": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "50": [19, 28, 30, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 63, 65, 68, 70, 73, 75, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101], "outsid": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 112, 134, 135, 139, 144, 149, 150, 152], "spec": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "again": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100], "pend": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "32584": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "30365": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "8m57": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "real": [19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 107, 146], "becaus": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 131, 137], "xxxxxxxx": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "been": [19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 105, 107, 112, 116, 129, 131, 135, 136, 144], "inject": [19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "117": [19, 28, 33, 35, 36, 37, 38, 40, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 63, 68, 73, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101], "59": [19, 28, 33, 35, 36, 63, 68, 73, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101], "202": 19, "9m17": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "creation": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 111, 116, 129, 135], "previou": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 111, 112, 116, 135, 139], "detect": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 135, 139], "them": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 113, 129, 131, 135, 139, 142, 144, 151], "frontend": [19, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 151], "66d44feb": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "0278": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "412a": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "a32d": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "73afe011f2c6": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "nyc": [19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101], "33m": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "32m": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "recreat": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "wa": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 116, 144], "origin": [19, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 134, 149], "being": [19, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 111, 116, 135, 136, 137, 139], "nativ": [19, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 133, 138, 147], "schema": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "backend": [19, 21, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 145, 151], "srv04": [19, 27, 30, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101], "srv05": [19, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101], "80": [19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 73, 75, 76, 80, 85, 90, 95, 100, 122, 143], "These": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "index": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "html": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 137], "cat": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "eof": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "apivers": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "v1alpha1": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "metadata": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 108, 129, 135, 139], "ownerten": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "45": [19, 27, 28, 30, 31, 32, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101], "46": [19, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101], "timeout": [19, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "3000": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 129, 135], "And": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 113, 131, 135, 139], "d07acd0f": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "51ea": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "429a": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "89dd": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "8e4c1d6d0a86": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "2m17": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "3m47": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "203": [19, 106], "try": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 137], "displai": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 106, 111, 116], "l2": [19, 24, 107], "segment": [19, 26, 152], "vnet": [19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106], "guestten": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "switchport": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "swp2": [19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106], "sw22": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100], "incom": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 122, 134, 143, 149], "isp2": [19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100], "neighbora": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "65007": [19, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101], "transport": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 129, 142], "swp14": [19, 83, 85], "sw02": [19, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 82, 83, 85, 88, 90, 91, 93, 95, 96, 98, 100, 101], "vlanid": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "1092": [19, 98, 100, 101], "localip": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "118": [19, 38, 40, 41, 100, 101], "30": [19, 26, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106, 111, 112, 113, 116, 146], "remoteip": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "prefixlistoutbound": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "28": [19, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 113, 131], "le": [19, 23, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106], "32": [19, 26, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106, 111, 116], "session": [19, 20, 23, 26, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 106, 108, 112, 136, 146, 151], "neighbor": [19, 23, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 112, 146], "remot": [19, 23, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 88, 90, 91, 93, 95, 96, 98, 100, 101, 112, 134, 135, 139, 149, 150, 152], "similar": [19, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 116, 134, 136, 142, 149], "00": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 137], "51": [19, 23, 85, 106], "2m3": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "feel": [19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 141], "annot": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "previous": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 111, 116, 136], "true": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "otherwis": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 145, 152], "complain": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "thei": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 106, 108, 127, 131, 135, 139, 152], "won": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "put": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100], "back": [19, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 112, 135], "declar": [19, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 135], "suppos": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 113, 131], "prevent": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "reclaimpolici": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "retain": [19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100], "Or": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "ll": [19, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 110, 115, 152], "leaf": [19, 20, 24, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101], "tor": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "clean": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "doc": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 113, 121, 122, 130, 143], "veri": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "bgpconfigur": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "nodetonodemeshen": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "asnumb": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "64512": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "projectcalico": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "org": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "v1": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "logseverityscreen": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "06": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 137], "7m59": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "sandbox9": [19, 97, 98, 100, 101], "srv06": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "110": [19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 93, 95, 96, 100, 101], "66": [19, 23, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 71, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100], "4200070000": [19, 85], "26": [19, 30, 35, 40, 45, 50, 55, 60, 65, 66, 70, 75, 80, 85, 90, 95, 100, 106], "srv07": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "67": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "4200070001": [19, 85], "srv08": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "68": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "4200070002": [19, 85], "notic": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 109, 117, 127], "07": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "48": [19, 30, 35, 40, 45, 50, 55, 60, 65, 68, 70, 71, 75, 80, 85, 90, 95, 100], "8m41": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "44": [19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 90, 91, 95, 96, 100, 101], "19": [19, 30, 32, 33, 35, 36, 40, 45, 46, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 102, 137], "still": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 136], "mean": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 135, 139], "fals": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "final": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 106, 111, 112, 116], "earlier": [19, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100], "ye": [19, 26, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "mfpdt": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "revis": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "34577c": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "logo": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "raw": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "githubusercont": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "gh": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "page": [19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 111, 116, 137], "cuddle_clap": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "gif": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "messag": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 137], "greet": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "v6": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "goo": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "goarch": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "runtim": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "go1": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "num_goroutin": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "num_cpu": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "leverag": [20, 21, 26, 134, 144, 149], "ecmp": [20, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "hash": [20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "capabl": [20, 26, 111, 113, 116, 131, 134, 149], "spine": [20, 24, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 82, 86, 91, 96, 101, 107], "deliv": 20, "besid": [20, 129, 135, 151], "advertis": [20, 23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 86, 88, 91, 93, 96, 98, 101, 108, 112, 113, 146], "unicast": 20, "toward": [20, 23, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "pair": [20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 112], "lifetim": [20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "identifi": [20, 135, 139, 150], "failur": [20, 107], "rerout": 20, "case": [20, 23, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 113, 131, 135, 136, 152], "outag": 20, "instanc": [20, 26, 110, 115, 123, 125, 130, 134, 149, 150], "ellips": 20, "extra": 20, "status": [20, 23, 107, 146, 151], "router": [21, 22, 23, 26, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 87, 88, 91, 92, 93, 96, 97, 98, 101, 108, 112, 146], "layer": [21, 22, 26, 112, 139, 152], "least": [21, 111], "well": [21, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 133, 134, 135, 138, 139, 147, 149, 150], "kubenet": [21, 25], "requestor": 21, "space": [21, 22, 83, 111, 116], "administr": [21, 23, 141], "expos": [21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "next": [21, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 106, 107, 108, 109, 111, 113, 116, 117, 127, 129, 131, 144, 148, 150], "probe": 21, "uncondition": 21, "millisecond": 21, "compos": 22, "element": [22, 106], "statist": [22, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 151], "analyt": 22, "modul": 22, "diagram": [22, 28, 29, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 84, 88, 89, 93, 94, 98, 99, 146], "high": [22, 23, 24, 26, 109, 117, 127, 152], "replic": [22, 113, 131], "singl": [22, 23, 24, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 121, 126, 130, 151, 152], "unreach": [22, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "toler": [22, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "stat": [22, 26], "collect": [22, 136], "unavail": [22, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 136], "howev": [22, 111, 116, 135, 144, 148], "affect": [22, 136], "border": [22, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 108, 146], "translat": [22, 23, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 134, 140, 149], "x86": [22, 109, 121], "dpdk": [22, 25, 109, 117, 127], "enter": [22, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 106, 111, 116], "bypass": 22, "kernel": [22, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 137], "travel": 22, "pcie": 22, "bu": 22, "closest": [22, 23], "last": [22, 148], "cach": [22, 23], "header": 22, "rewrit": 22, "mac": [22, 23, 107, 135, 151], "vlan": [22, 23, 29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 105, 112, 129, 135, 139, 140, 141, 142, 148, 150, 152], "id": [22, 23, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 88, 91, 93, 96, 98, 101, 105, 106, 112, 114, 117, 128, 135, 139, 141, 142, 146, 148, 150, 152], "One": 22, "frr": [22, 23, 25, 26, 108], "standard": [22, 24, 113, 131, 134, 144, 149], "wireguard": [22, 23, 25], "tunnel": [22, 23, 25], "dynam": [22, 23, 25, 107, 135, 139], "program": 23, "meet": [23, 111, 116], "tabl": [23, 26, 111, 146], "termin": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 106, 111, 116], "autonom": 23, "without": [23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 129, 135, 142, 151], "cabl": [23, 25, 106, 107], "ixp": 23, "ggc": 23, "googl": [23, 116], "tag": [23, 109, 113, 116, 117, 129, 142, 148, 152], "untag": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 152], "example2": 23, "fine": 23, "expand": [23, 26, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 152], "aka": 23, "multihop": 23, "choos": [23, 111, 112, 116, 127, 130], "speaker": 23, "allowa": 23, "occurr": 23, "nlri": 23, "normal": [23, 151], "inbound": [23, 26, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 88, 91, 93, 96, 98, 101, 108, 111], "max": [23, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "exce": 23, "maximum": [23, 105], "1000": [23, 112], "handl": [23, 108, 113, 129, 131, 146], "million": 23, "outbound": [23, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 88, 91, 93, 96, 98, 101, 111, 113, 134, 149], "prefer": [23, 148, 150], "weight": 23, "prepend": 23, "mani": [23, 26, 106, 152], "referenc": 23, "extend": [23, 105], "larg": [23, 113], "consist": [23, 134, 137, 144, 149], "possibl": [23, 113, 135, 139, 144, 152], "mandatori": [23, 107, 137], "len": 23, "ge": 23, "keyword": 23, "aa": 23, "nn": 23, "65535": 23, "known": [23, 26], "associ": [23, 111, 113, 116, 131, 135, 139], "sequenc": [23, 111, 116], "drag": [23, 107], "claus": 23, "hop": [23, 111, 116], "med": 23, "met": 23, "whether": [23, 106], "manipul": 23, "isp": [23, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 106, 151], "situat": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 134, 149], "dual": [23, 152], "home": [23, 152], "temporari": 23, "interconnect": [23, 25, 107], "hypervisor": [23, 26, 142], "intern": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 83, 87, 88, 92, 93, 97, 98, 116, 148], "scope": [23, 112, 128], "null": 23, "254": [23, 27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 101, 106, 116, 135, 139], "leaf1": 23, "spine1": 23, "snat": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 83, 87, 88, 92, 93, 97, 98, 102, 107, 113, 126, 131, 140, 144], "dnat": [23, 102, 107, 113, 126, 131, 135, 140, 144], "roll": [23, 107, 135], "larger": [23, 109, 127], "scale": 23, "carrier": 23, "grade": 23, "overload": 23, "along": 23, "accept": 23, "silent": 23, "exclus": [23, 148], "masquerad": [23, 126, 140, 150], "exit": 23, "predefin": [23, 144], "198": [23, 106], "1080": 23, "particip": [23, 25, 152], "henc": 23, "reach": [23, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101, 108, 136], "hub": [23, 111, 116], "carri": 23, "transit": 23, "major": [23, 24], "spoke": 23, "small": [23, 109, 121, 127, 130], "offic": 23, "measur": 23, "underneath": 23, "mathemat": 23, "randomli": 23, "converg": 23, "tool": [23, 25, 133, 138, 147], "perspect": 23, "left": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 113, 131, 144, 151], "dropdown": [23, 105, 111, 113, 114, 116, 117, 128, 131, 144, 151], "famili": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 112], "summari": [23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 151], "adjac": [23, 26, 82], "queri": 23, "lookup": 23, "rib": 23, "tracerout": 23, "conduct": 23, "determin": 23, "ping": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "evpn": [23, 152], "brief": 23, "vni": 23, "distinguish": 23, "fall": [24, 144], "four": [24, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "pattern": 24, "collaps": 24, "throughput": 24, "redund": [24, 112, 129, 142], "distribut": [24, 25, 137], "higher": [25, 26, 107], "27mpp": [25, 102], "100gbp": 25, "12mpp": [25, 102], "built": [25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 137], "andoth": 25, "comprehens": [25, 151], "call": [25, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 151], "sent": [25, 146, 151], "abil": [25, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 151], "search": [25, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 151], "sort": [25, 151], "column": [25, 151], "filter": [25, 116, 151], "andautomat": 25, "loop": [25, 107], "entir": [25, 111, 113, 136, 151, 152], "kvm": [25, 135, 139, 142, 150], "construct": 25, "easi": [25, 129, 142], "authent": 25, "improv": 25, "becom": [25, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 129, 142], "hide": 25, "comfort": [25, 141], "resili": 26, "ecosystem": 26, "daemon": [26, 105, 108], "bare": [26, 113, 127, 129, 131, 132, 134, 135, 139], "even": [26, 27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 129, 135, 139, 142, 151], "suit": [26, 134, 149], "don": [26, 113, 129, 131, 135], "ethernet": [26, 29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99], "span": [26, 152], "stp": 26, "miniz": 26, "reliabl": 26, "increas": 26, "interim": 26, "proxmox": 26, "Will": 26, "aggreg": [26, 105], "inherit": 26, "zero": 26, "msft": 26, "sometim": 26, "autodetect": [26, 152], "1gbp": [26, 152], "10gpb": 26, "speed": [26, 105, 152], "bulk": [26, 105, 152], "b": [26, 81], "255": [26, 111, 121], "certain": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 105, 111, 116, 134, 149], "explor": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "interact": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106], "yourself": [27, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100], "visit": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "sandbox1": [27, 28, 30, 31], "examin": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "indic": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "second": [27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 109, 117, 127, 131, 148, 151], "sw21": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "2607": [27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 87, 91, 92, 96, 97, 101], "f358": [27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 87, 91, 92, 96, 97, 101], "11": [27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 87, 91, 92, 96, 97, 101, 102, 111], "ffc1": [27, 31], "respect": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "properli": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 111], "unchang": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "label": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "submit": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98, 112], "result": [27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 135, 150], "thank": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 135], "vrf_netri": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "56": [27, 32, 37, 42, 47, 52, 57, 62, 67, 68, 71, 72, 77, 82, 87, 92, 97], "84": [27, 32, 37, 42, 47, 48, 51, 52, 57, 62, 67, 72, 73, 76, 77, 82, 87, 92, 97], "byte": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "icmp_seq": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "ttl": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "562": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "745": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "690": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "737": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "666": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "transmit": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "loss": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "4092m": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "rtt": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "min": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "avg": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "mdev": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "680": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "065": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "interest": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "found": [27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 136], "asid": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "toggl": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 87, 88, 92, 93, 97, 98, 105], "iri": [27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101], "isp1": [27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101], "sandbox": [27, 28, 29, 30, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 62, 63, 64, 65, 67, 68, 69, 70, 72, 73, 74, 75, 77, 78, 79, 80, 82, 83, 84, 85, 87, 88, 89, 90, 92, 93, 94, 95, 97, 98, 99, 100], "fault": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "observ": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 87, 88, 92, 93, 97, 98], "g": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 83, 87, 88, 92, 93, 97, 98, 148], "demo": [27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101], "166": [27, 28, 31, 32, 33, 36, 48, 50, 51, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101], "88": [27, 28, 31, 32, 33, 36, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101], "17": [27, 28, 31, 32, 33, 36, 41, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 102], "30064": [27, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 101], "introductori": [27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100], "ping4": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 87, 92, 97], "repli": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "62": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 70, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "wan": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "impos": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "granular": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 113, 134, 135, 139, 146, 149, 150], "implement": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97], "would": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 152], "short": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "exercis": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "conjunct": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "intellig": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "automag": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "across": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 151], "give": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 113, 135], "30065": [28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101], "eth1": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 148], "proto": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "onlink": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "fulli": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "reachabl": [28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 107], "judg": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "corner": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 111, 116, 131, 144, 151], "mark": [28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 111, 135, 151], "wire": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "55": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "world": [28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101], "swp16": [28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 88, 90, 91, 93, 95, 96, 98, 100, 101, 106], "easili": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 112, 134, 149], "uncheck": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "1012": [28, 30, 31], "126": [28, 33, 35, 36, 43, 45, 46, 63, 68, 73, 78, 88, 98], "125": [28, 33, 35, 36, 43, 45, 46, 63, 68, 73, 78, 88, 98], "38": [28, 30, 31, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76], "161": [28, 30, 31, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 88], "green": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 112, 136], "glean": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "insight": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "leav": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 114, 117, 128], "face": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "aim": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 148], "23": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 102, 122, 143], "srv01": [28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 86, 88, 91, 93, 96, 98, 101], "srv02": [28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 86, 88, 91, 93, 96, 98, 101], "webpag": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "appear": [28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 105, 116], "216": [28, 33, 36, 63, 68, 73, 78, 88, 93, 98], "repeat": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 107, 109, 111, 116, 117, 127, 148], "had": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "tab": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 111, 116], "bar": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "calcul": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "factor": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "websit": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "land": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "trigger": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "direct": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 111, 116], "could": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "around": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "simul": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "mainten": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "vip": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98, 151], "name_50": [28, 33, 63, 68, 73, 78, 88, 93, 98], "pop": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "opposit": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "al3lb": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "were": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "uc": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "longer": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 135, 139], "sync": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 105, 107, 127], "member": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 151], "resum": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98], "content": [29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99], "welcom": [29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 118, 119, 120], "vxlan": [29, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 152], "exterior": [29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99], "l3lb": [29, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 89, 94, 99], "kubesprai": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "dedic": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 107, 134, 142, 149], "apiserv": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "kubeconfig": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "ve": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 135, 139], "6443": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "debug": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "diagnos": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "problem": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "dump": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 136], "13": [30, 101, 102, 131, 137], "rememb": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "develop": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "decid": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 121, 129], "ic": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "coffe": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "14": [30, 36], "unhealthi": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "didn": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "behavior": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 148], "round": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "robin": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "intermitt": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "went": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "btw": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "prefixlistinbound": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 95, 100, 106], "160": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 88, 90, 91, 95, 100], "01": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 95, 100, 106, 135, 139], "27": [30, 35, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 65, 70, 75, 80, 90, 95, 100], "4230000000": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 95, 100], "4230000001": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 95, 100], "4230000002": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 90, 95, 100], "beer": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "depend": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 121, 122, 130, 143, 146, 152], "question": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "five": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "integrat": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "routabl": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "srv03": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "30061": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "30062": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "30063": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "softage1": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 91, 96, 101], "sw01": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 82, 91, 96, 101], "1011": 31, "ffc0": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 91, 96, 101], "127": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 91, 96, 101, 106], "softage2": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 91, 96, 101], "___": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 91, 96, 101], "sandbox10": [32, 33, 35, 36], "ffca": [32, 36], "1102": [33, 35, 36], "208": [33, 35, 36], "209": [33, 61], "212": [33, 36], "221": 35, "222": 35, "1101": 36, "122": [36, 46], "121": [36, 46], "220": 36, "sandbox11": [37, 38, 40, 41], "ffcb": [37, 41], "82": [37, 38, 41, 81], "1112": [38, 40, 41], "96": [38, 40, 41], "97": [38, 91], "104": [38, 41], "name_45": [38, 43, 48, 53, 58], "109": [40, 93, 95, 96], "1111": 41, "114": [41, 101], "113": [41, 101, 106], "108": 41, "sandbox12": [42, 43, 45, 46], "ffcc": [42, 46], "83": [42, 43, 46], "1122": [43, 45, 46], "129": [43, 78], "132": [43, 46, 78, 81], "136": [43, 46, 78, 81], "141": [45, 80], "142": [45, 80], "1121": 46, "140": [46, 81], "sandbox13": [47, 48, 50, 51], "ffcd": [47, 51], "1132": [48, 50, 51], "165": [48, 50, 51], "144": [48, 50, 51, 83, 85, 86, 111], "145": 48, "148": [48, 51], "152": [48, 51], "157": 50, "158": 50, "1131": 51, "162": 51, "1b": 51, "1a": 51, "156": 51, "sandbox14": [52, 53, 56], "ffce": [52, 56], "85": [52, 53, 56, 80, 81, 116], "1142": [53, 55, 56], "174": [53, 55, 56, 90], "173": [53, 55, 56, 90], "176": [53, 55, 56, 93, 95, 96, 111], "177": [53, 93], "180": [53, 56, 93, 96], "184": [53, 56, 93, 96], "sandbox15": [55, 57, 58, 60, 61], "189": [55, 67, 68, 71, 95], "190": [55, 62, 63, 66, 95], "1141": 56, "170": 56, "169": 56, "1d": 56, "1c": 56, "188": [56, 72, 73, 76, 96], "ffcf": [57, 61], "86": [57, 58, 61, 80, 81], "1152": [58, 60, 61], "214": [58, 60, 61], "213": [58, 60, 61], "193": [58, 98], "196": [58, 61, 98, 101], "200": [58, 61, 98, 101], "205": [60, 100], "206": [60, 100], "1151": 61, "210": 61, "1f": 61, "1e": 61, "204": [61, 101], "sandbox2": [62, 63, 65, 66], "ffc2": [62, 66], "1022": [63, 65, 66], "36": [63, 66, 137], "40": [63, 66], "29": [65, 66, 92, 93, 96, 131, 137], "1021": 66, "25": [66, 106], "sandbox3": [67, 68, 70, 71], "ffc3": [67, 71], "1032": [68, 70, 71], "52": [68, 71], "61": [70, 130], "70": [70, 71], "69": [70, 71], "1031": 71, "60": 71, "sandbox4": [72, 73, 75, 76], "ffc4": [72, 76], "1042": [73, 75, 76], "81": [73, 81], "93": [75, 83, 85, 86], "94": [75, 83, 85, 86], "78": [75, 76], "77": [75, 76], "1041": 76, "74": 76, "73": 76, "9": 76, "92": 76, "sandbox5": [77, 78, 80, 81], "ffc5": [77, 81], "187": [77, 78, 81], "1052": [78, 80, 81], "1051": 81, "sandbox6": [82, 83, 85, 86], "radio": 82, "row": 82, "choic": 82, "543": 82, "436": 82, "445": 82, "354": 82, "345": 82, "4074m": 82, "424": 82, "075": 82, "insid": [82, 108, 129, 152], "screen": [82, 111, 116], "pubblic": 82, "186": [82, 83, 86], "disallow": 82, "nfv": [83, 107], "unselect": 83, "1062": [83, 85, 86], "150": [83, 116], "durat": 83, "v0": 85, "154": [85, 130], "155": 85, "1061": 86, "90": 86, "89": [86, 121], "sandbox7": [87, 88, 90, 91], "ffc7": [87, 91], "185": [87, 88, 91], "1072": [88, 90, 91], "164": [88, 91], "102": [90, 91], "101": [90, 91], "1071": 91, "98": 91, "sandbox8": [92, 93, 95, 96], "ffc8": [92, 96], "1082": [93, 95, 96], "1081": 96, "105": 96, "ffc9": [97, 101], "1091": 101, "help": 102, "mpp": 102, "unmatch": 102, "e5": 102, "2660": 102, "fw": 102, "schedul": 102, "kni": 102, "17mpp": 102, "7mpp": 102, "9mpp": 102, "2mpp": 102, "gold": 102, "6130": 102, "odd": 102, "31mpp": 102, "22mpp": 102, "15mpp": 102, "11mpp": 102, "3mpp": 102, "3m": 102, "conntrack": 102, "20k": 102, "38mpp": 102, "28mpp": 102, "19mpp": 102, "whom": 105, "breakout": 105, "split": [105, 126], "shown": 105, "platform": [105, 106, 116, 135, 137, 139, 150], "switchd": 105, "mtu": 105, "transmiss": 105, "autoneg": 105, "autonegoti": 105, "duplex": 105, "extens": [105, 106], "quick": 105, "lag": [105, 152], "detach": 105, "hand": 106, "tutori": [106, 137], "hashicorp": 106, "tf": 106, "At": [106, 109, 112, 113, 117, 127], "begin": [106, 112], "required_provid": 106, "netris_address": 106, "netris_login": 106, "netris_password": 106, "init": 106, "folder": [106, 137], "properti": 106, "Such": 106, "netris_sit": 106, "clear": 106, "tenantid": 106, "netris_ten": 106, "netris_alloc": 106, "netris_subnet": 106, "defaultgatewai": 106, "siteid": 106, "depends_on": 106, "With": [106, 107], "netris_softg": 106, "mainip": 106, "mgmtip": 106, "netris_switch": 106, "cumulus_linux": 106, "portcount": 106, "netris_link": 106, "sg": [106, 136], "sw": [106, 136], "swp1": 106, "netris_vnet": 106, "1050": 106, "swp3": 106, "10th": 106, "netris_port": 106, "swp10_my_switch": 106, "swp10": 106, "netris_bgp": 106, "23456": 106, "portid": 106, "error": [106, 151], "confirm": 106, "progress": 106, "destroi": 106, "resid": 107, "preliminari": 107, "regist": [107, 111, 116], "administ": [107, 113, 131, 135, 144], "chapter": [107, 108, 113, 135, 139], "watch": 107, "against": 107, "long": 107, "capac": 107, "lift": 107, "v2": 107, "know": [107, 108, 142], "visual": [107, 151], "scene": [108, 152], "pull": 108, "bring": [108, 109, 117, 127], "happen": [109, 117, 127], "acceler": [109, 117, 127], "signal": [109, 113, 117, 127, 131, 151], "crit": [109, 117, 127], "heart": [109, 117, 127], "beat": [109, 117, 127], "premis": [110, 115], "therefor": [110, 111, 115, 116, 129, 134, 142, 149], "commenc": [111, 116], "due": [111, 116, 129, 136, 139], "numer": [111, 116], "equip": [111, 116], "firewal": [111, 116], "advis": [111, 116], "afterward": [111, 116, 136], "essenti": [111, 116, 134, 144, 149], "t2": 111, "medium": [111, 116], "t3": 111, "satisfi": [111, 116, 144], "drive": [111, 116], "crucial": [111, 116], "upcom": [111, 116], "involv": [111, 116], "dot1q": [111, 116], "input": [111, 116], "65500": [111, 116], "asn": [111, 112, 116], "subsequ": [111, 116], "netmask": [111, 116], "unlik": [111, 116, 134, 136, 149], "cidr": 111, "block": [111, 119, 124, 135], "31": [111, 127, 130], "int": [111, 116, 148], "modifi": [111, 148], "target": [111, 116], "checkbox": [111, 135], "secondari": 112, "coupl": 112, "unus": [112, 131], "vc": 112, "bill": 112, "token": 112, "edg": 112, "onto": 112, "shoudl": 112, "yellow": 112, "although": [113, 131], "intend": [113, 131, 144, 152], "crush": 113, "smaller": [113, 131], "worri": [113, 129, 131], "tenanc": [113, 131, 144], "word": [113, 131], "colleagu": [113, 131, 135, 144], "later": [113, 135, 151], "repurpos": [113, 131], "deal": [114, 117, 128], "write": [114, 117], "e2": 116, "34": 116, "167": 116, "ens4": 116, "region": [116, 121, 130], "relev": 116, "manner": 116, "regardless": 116, "accur": 116, "smallest": [117, 121, 130], "flavor": [117, 121, 130], "video": 119, "walk": [119, 132], "enough": [121, 130, 136], "139": 121, "178": 121, "idea": [121, 130, 143], "netrisctl": [121, 122, 130, 143], "issuer": [121, 122, 130, 143], "hint": [121, 122, 130, 143], "183": [122, 143], "201": [122, 143], "443": [122, 143], "2003": [122, 143], "stream": [122, 143], "telemetri": [122, 143], "collectd": [122, 143], "3033": [122, 143], "50051": [122, 143], "s2": 127, "c1": 127, "jammi": 127, "client": 128, "solut": [129, 142, 148], "product": [129, 134, 136, 142, 146, 149], "introduc": [129, 136, 142], "disrupt": [129, 142], "synchron": [129, 148, 151], "4094": 129, "thu": [129, 134, 135, 137, 142, 149], "safe": [129, 142], "workload": [129, 142], "802": [129, 142], "1q": [129, 142], "s0": 130, "d1": 130, "131": 130, "153": 130, "usabl": 131, "broadcast": 131, "offici": 132, "meanwhil": [133, 138, 147], "goal": [134, 149], "ideal": [134, 149], "packag": [134, 149], "task": [134, 149], "redirect": [134, 149], "often": [134, 149], "55022": [134, 149], "vmware": [135, 139, 142, 150], "comput": [135, 139, 150], "02": [135, 139], "03": [135, 139], "launch": [135, 139, 150], "serv": [135, 139, 144, 150], "sinc": [135, 139], "live": [135, 139], "modif": 135, "made": 135, "cannot": 135, "perman": 135, "jumphost": 135, "proxycommand": 135, "exclud": [135, 139], "flexibl": [135, 139], "build": [135, 137, 139], "sai": [135, 139], "dialogu": [135, 139], "convert": [135, 139], "structur": 136, "event": 136, "sshing": 136, "check_ag": 136, "impact": 136, "svclb": 136, "haproxi": 136, "6tkgj": 136, "38d": 136, "bcb944b7c": 136, "qcbf8": 136, "13d": 136, "squid": 136, "7f6fdc6cf9": 136, "7fdx8": 136, "58rnp": 136, "graphit": 136, "mongodb": 136, "redi": 136, "smtp": 136, "76778cf85f": 136, "lw5v5": 136, "10d": 136, "8b9dbbcd8": 136, "8snhd": 136, "notifi": 136, "647975848f": 136, "fs5dn": 136, "app": 136, "b9b8d8f8d": 136, "4ssqb": 136, "987669fb9": 136, "jjskp": 136, "777c98c5d9": 136, "mqwl6": 136, "lqmq7": 136, "20h": 136, "advers": 136, "downgrad": 136, "031": 136, "www": 137, "sonic_20220929_052156_ec202012_420": 137, "wipe": 137, "author": [137, 144], "ident": 137, "prompt": 137, "mismatch": 137, "switch15": 137, "debian": 137, "commit": 137, "895d178f6": 137, "sep": 137, "05": 137, "utc": 137, "2022": 137, "225": 137, "x86_64": 137, "accton_as7326_56x": 137, "r0": 137, "hwsku": 137, "accton": 137, "as7326": 137, "56x": 137, "asic": 137, "broadcom": 137, "serial": 137, "redact": 137, "uptim": 137, "averag": 137, "39": 137, "08": 137, "aren": 139, "dmz": 140, "700": [141, 148, 150], "900": [141, 148], "advantag": 144, "imper": 144, "align": 144, "reconfigur": 144, "rfc": 144, "1918": 144, "encompass": 144, "worth": 144, "illustr": 144, "meant": 144, "uncom": 145, "tradit": 146, "logic": 146, "especi": 146, "1m": 146, "alter": 146, "trobleshoot": 146, "propos": 148, "interface_nam": 148, "mission": 148, "eth2": 148, "facilit": 149, "sum": 151, "titl": 151, "bp": 151, "pp": 151, "optic": 151, "histori": 151, "iris1": 151, "iris2": 151, "explan": 151, "summar": 151, "icon": 151, "pie": 151, "suppli": 151, "fan": 151, "temperatur": 151, "sensor": 151, "alarm": 151, "unrout": 152, "collabor": 152, "unnumb": 152, "underlai": 152, "awar": 152, "rare": 152, "guest": 152, "circuit": 152, "backbon": 152, "alia": 152, "swp": 152, "signific": 152, "egress": 152, "standbi": 152, "assum": 152, "svi": 152, "nose": 152, "10gbp": 152}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"edgecor": 0, "sonic": [0, 137], "switch": [0, 1, 2, 5, 13, 22, 103, 104, 105, 107], "initi": [0, 1, 2, 5, 104], "setup": [0, 1, 2, 5, 104, 119, 123, 124, 125, 126, 131, 140, 144], "nvidia": [1, 2, 107], "cumulu": [1, 2, 107], "v3": 1, "7": 1, "v5": 2, "softgat": [3, 4, 22, 102, 103, 107, 109, 111, 116, 117, 127, 145, 148], "pro": 3, "instal": [3, 4, 8, 9, 10, 11, 13, 14, 15, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 121, 122, 130, 132, 143, 145], "minimum": [3, 4, 145], "hardwar": [3, 4, 145], "requir": [3, 4, 8, 9, 11, 143, 145], "bio": 3, "configur": [3, 9, 13, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106, 111, 116, 146], "netri": [3, 4, 8, 11, 13, 15, 16, 19, 22, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 103, 106, 109, 111, 116, 117, 118, 119, 121, 122, 124, 126, 127, 129, 130, 132, 136, 143], "agent": [3, 22], "provis": [4, 109, 111, 116, 127, 145], "softwar": [4, 145], "ubuntu": 5, "switchdev": 5, "account": 6, "user": 6, "tenant": 6, "permiss": 6, "group": 6, "role": 6, "access": [7, 11, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "control": [7, 8, 11, 14, 15, 19, 22, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 111, 116, 121, 122, 130, 143], "list": [7, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "acl": [7, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "default": [7, 141], "polici": [7, 13, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "rule": [7, 23], "approv": 7, "workflow": 7, "process": [7, 10], "order": 7, "gener": [8, 118], "linux": [8, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "host": [8, 26], "specif": 8, "name": 8, "let": 8, "": [8, 116], "encrypt": 8, "ssl": [8, 11], "custom": [8, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "issuer": 8, "upgrad": [8, 136, 137], "uninstal": [8, 9], "backup": 8, "restor": 8, "secret": 8, "kei": 8, "helm": [9, 19], "chart": [9, 19], "get": [9, 119, 120, 140], "repo": 9, "info": 9, "quickstart": 10, "virtual": [11, 135, 139, 150], "machin": 11, "kvm": 11, "hypervisor": 11, "vm": [11, 116], "replac": 11, "certif": 11, "definit": 12, "welcom": [13, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "document": 13, "tutori": [13, 118], "cloud": 13, "nativ": 13, "tool": 13, "fundament": 13, "detail": 13, "fabric": [13, 112], "network": [13, 24, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 103, 104, 135, 139, 144, 146, 150], "servic": [13, 21, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 113, 131, 133, 134, 135, 138, 139, 144, 147, 149, 150], "oper": [13, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 137], "updat": 13, "lab": 13, "scenario": 13, "introduct": [16, 19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 110, 115, 129, 142], "inventori": [17, 107], "profil": [17, 107], "field": [17, 18, 23, 107, 152], "ip": [18, 113, 146], "address": [18, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 113, 144, 146], "manag": [18, 103, 107], "alloc": [18, 131, 144], "subnet": [18, 111, 116, 131, 135], "add": [18, 107, 111, 116], "an": [18, 20, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106, 111, 112, 135, 144], "kubernet": [19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101], "integr": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 114, 117, 128], "method": 19, "regular": 19, "manifest": 19, "us": [19, 30, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 119, 124, 126, 133, 134, 135, 138, 139, 140, 146, 147, 148, 149, 150], "type": 19, "loadbalanc": 19, "resourc": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 106], "l4lb": [19, 21, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "v": [19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 113, 135, 139, 148, 150, 152], "net": [19, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 113, 135, 139, 148, 150, 152], "bgp": [19, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 108, 146], "import": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "exist": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 135], "from": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "reclaim": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "calico": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "cni": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "disabl": [19, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "l3": [20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "load": [20, 21, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 113, 131, 133, 138, 144, 147], "balanc": [20, 21, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 85, 88, 90, 93, 95, 98, 100, 113, 131, 133, 138, 144, 147], "anycast": [20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "lb": 20, "creat": [20, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 106, 111, 112, 116, 135, 144, 148, 150], "l4": [21, 133, 138, 144, 147], "enabl": [21, 23, 111, 113, 114, 116, 117, 123, 125, 128, 131, 144], "consum": 21, "architectur": [22, 103], "basic": 23, "ad": [23, 26, 107, 111, 117, 135], "peer": 23, "advanc": 23, "object": 23, "ipv4": 23, "prefix": 23, "ipv6": 23, "commun": 23, "rout": [23, 26, 111, 116], "map": 23, "static": [23, 111, 116], "nat": [23, 27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 113, 131, 134, 144, 149], "defin": 23, "sitemesh": 23, "look": 23, "glass": 23, "refer": [24, 103], "design": 24, "releas": 25, "note": [25, 131], "roh": 26, "provid": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 106], "exampl": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 119], "ethernet": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "vlan": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 146], "vxlan": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "e": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "exterior": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "border": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "gatewai": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "protocol": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98], "translat": [27, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 144], "learn": [28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100], "l3lb": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 88, 93, 98], "sandbox1": 29, "intro": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "deploi": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 111, 116, 135], "applic": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "On": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "demand": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 113, 121, 131, 133, 138, 144, 147], "mileston": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100], "1": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 113, 131, 134], "2": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 113, 131, 134], "sandbox": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "topologi": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 107], "diagram": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "server": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 121, 130, 135], "cluster": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "upstream": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 146], "isp": [31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101], "sandbox10": 34, "sandbox11": 39, "sandbox12": 44, "sandbox13": 49, "sandbox14": 54, "sandbox15": 59, "sandbox2": 64, "sandbox3": 69, "sandbox4": 74, "sandbox5": 79, "sandbox6": 84, "gui": 86, "sandbox7": 89, "sandbox8": 94, "sandbox9": 99, "perform": 102, "unmanag": [103, 135, 139], "ha": 103, "scalabl": 103, "data": 103, "center": 103, "port": 105, "terraform": 106, "To": 106, "your": 106, "first": 106, "directori": 106, "file": 106, "prepar": 106, "infrastructur": 106, "plan": 106, "delet": 106, "view": 107, "link": [107, 148], "hairpin": 107, "onli": 107, "activ": 108, "equinix": [108, 109, 112, 114, 117, 119, 121, 124, 139], "metal": [108, 109, 114, 117, 119, 121, 124, 139], "project": [108, 109, 119, 139], "node": [109, 111, 116, 117, 127], "site": [110, 111, 115, 116, 123, 125, 141], "mesh": [110, 111, 115, 116, 123, 125], "aw": [110, 111, 123], "overview": [110, 115, 129, 142, 146], "concept": [110, 115, 129, 142], "ec2": 111, "instanc": [111, 116], "pre": [111, 116], "requisit": [111, 116], "step": [111, 116, 119, 121, 130, 143], "vpc": [111, 116, 119, 120, 124, 126, 129, 140, 142, 146], "sourc": 111, "destin": 111, "check": [111, 141], "interconnect": 112, "pool": 113, "request": 113, "new": [113, 135], "public": [113, 146], "block": 113, "elast": [113, 131, 133, 138, 144, 147], "3": 113, "4": 113, "api": [114, 117, 128, 151], "gcp": [115, 116, 125], "start": [119, 120, 140], "guid": [119, 120, 140], "anywher": [120, 140, 142], "secur": [121, 122, 130, 143], "matter": [121, 122, 130, 143], "phoenixnap": [126, 127, 128, 129, 130, 132, 133, 134, 135], "bmc": [126, 127, 128, 129, 130, 132, 133, 134, 135], "ipam": [131, 144], "option": [131, 134, 146], "separ": 131, "each": 131, "purpos": 131, "split": 131, "singl": 131, "differ": 131, "masquerad": [134, 149], "snat": [134, 149], "dnat": [134, 149], "isol": [135, 139, 150], "tag": [135, 139], "rollback": 136, "procedur": [136, 137], "system": 137, "set": 141, "connect": 146, "one": 146, "two": 146, "dmz": 146, "multi": 148, "interfac": 148, "experiment": 148, "visibl": 151, "telescop": 151, "graph": 151, "board": 151, "log": 151, "dashboard": 151}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.todo": 2, "sphinx.ext.viewcode": 1, "sphinx": 57}, "alltitles": {"EdgeCore SONiC Switch Initial Setup": [[0, "edgecore-sonic-switch-initial-setup"]], "Nvidia Cumulus v3.7 Switch Initial Setup": [[1, "nvidia-cumulus-v3-7-switch-initial-setup"]], "Nvidia Cumulus v5 Switch Initial Setup": [[2, "nvidia-cumulus-v5-switch-initial-setup"]], "SoftGate PRO Installation": [[3, "softgate-pro-installation"]], "Minimum Hardware Requirements": [[3, "minimum-hardware-requirements"], [4, "minimum-hardware-requirements"], [145, "minimum-hardware-requirements"]], "BIOS Configuration": [[3, "bios-configuration"]], "Install the Netris Agent": [[3, "install-the-netris-agent"]], "SoftGate Installation": [[4, "softgate-installation"], [145, "softgate-installation"]], "Provision Netris SoftGate software": [[4, "provision-netris-softgate-software"]], "Ubuntu SwitchDev Switch Initial Setup": [[5, "ubuntu-switchdev-switch-initial-setup"]], "Accounts": [[6, "accounts"]], "Users": [[6, "users"]], "Tenants": [[6, "tenants"]], "Permission Groups": [[6, "permission-groups"]], "User Roles": [[6, "user-roles"]], "Access Control Lists (ACL)": [[7, "access-control-lists-acl"]], "ACL Default Policy": [[7, "acl-default-policy"]], "ACL Rules": [[7, "acl-rules"]], "ACL Approval Workflow": [[7, "acl-approval-workflow"]], "ACL Processing Order": [[7, "acl-processing-order"]], "Netris Controller installation on a generic Linux host": [[8, "netris-controller-installation-on-a-generic-linux-host"]], "Linux Host requirements": [[8, "linux-host-requirements"]], "Installation": [[8, "installation"]], "Installation with the specific host name": [[8, "installation-with-the-specific-host-name"]], "Installation with the Let\u2019s Encrypt SSL": [[8, "installation-with-the-let-s-encrypt-ssl"]], "Installation with the Custom SSL Issuer": [[8, "installation-with-the-custom-ssl-issuer"]], "Upgrading": [[8, "upgrading"]], "Uninstalling": [[8, "uninstalling"]], "Backup and Restore": [[8, "backup-and-restore"]], "Backup": [[8, "backup"]], "Backup the Secret Key": [[8, "backup-the-secret-key"]], "Restore": [[8, "restore"]], "Restore the Secret Key": [[8, "restore-the-secret-key"]], "Helm Chart Installation": [[9, "helm-chart-installation"]], "Requirements": [[9, "requirements"], [11, "requirements"]], "Get Repo Info": [[9, "get-repo-info"]], "Installing the Chart": [[9, "installing-the-chart"]], "Uninstalling the Chart": [[9, "uninstalling-the-chart"]], "Chart Configuration": [[9, "chart-configuration"]], "Quickstart Installation": [[10, "quickstart-installation"]], "Quickstart Process": [[10, "quickstart-process"]], "Virtual Machine Installation": [[11, "virtual-machine-installation"]], "KVM Hypervisor Installation": [[11, "kvm-hypervisor-installation"]], "VM Controller Installation": [[11, "vm-controller-installation"]], "Accessing the Netris Controller": [[11, "accessing-the-netris-controller"]], "Replacing the SSL certificate": [[11, "replacing-the-ssl-certificate"]], "Definitions": [[12, "definitions"]], "Welcome to Netris Documentation": [[13, "welcome-to-netris-documentation"]], "Tutorials": [[13, null]], "Cloud Native Tools": [[13, null]], "Netris Fundamentals": [[13, null]], "Detailed Installation": [[13, null]], "Switch-fabric Configuration": [[13, null]], "Network Policies": [[13, null]], "Network Services": [[13, null]], "Operations": [[13, null]], "Updates": [[13, null]], "Lab Scenarios": [[13, null]], "Controller Installation": [[14, "controller-installation"], [14, null]], "Installing a Netris Controller": [[15, "installing-a-netris-controller"], [122, "installing-a-netris-controller"]], "Introduction to Netris": [[16, "introduction-to-netris"]], "Inventory Profiles": [[17, "inventory-profiles"], [107, "inventory-profiles"]], "Inventory Profile Fields": [[17, "id1"], [107, "id1"]], "IP Address Management": [[18, "ip-address-management"]], "Allocations and Subnets": [[18, "allocations-and-subnets"]], "Add an Allocation": [[18, "add-an-allocation"]], "Allocation Fields": [[18, "id1"]], "Add a Subnet": [[18, "add-a-subnet"]], "Subnet fields": [[18, "id2"]], "Kubernetes Integration": [[19, "kubernetes-integration"]], "Install Netris Operator": [[19, "install-netris-operator"], [30, "install-netris-operator"], [35, "install-netris-operator"], [40, "install-netris-operator"], [45, "install-netris-operator"], [50, "install-netris-operator"], [55, "install-netris-operator"], [60, "install-netris-operator"], [65, "install-netris-operator"], [70, "install-netris-operator"], [75, "install-netris-operator"], [80, "install-netris-operator"], [85, "install-netris-operator"], [90, "install-netris-operator"], [95, "install-netris-operator"], [100, "install-netris-operator"]], "Helm Chart Method": [[19, "helm-chart-method"]], "Regular Manifest Method": [[19, "regular-manifest-method"]], "Using Type \u2018LoadBalancer\u2019": [[19, "using-type-loadbalancer"]], "Using Netris Custom Resources": [[19, "using-netris-custom-resources"], [30, "using-netris-custom-resources"], [35, "using-netris-custom-resources"], [40, "using-netris-custom-resources"], [45, "using-netris-custom-resources"], [50, "using-netris-custom-resources"], [55, "using-netris-custom-resources"], [60, "using-netris-custom-resources"], [65, "using-netris-custom-resources"], [70, "using-netris-custom-resources"], [75, "using-netris-custom-resources"], [80, "using-netris-custom-resources"], [85, "using-netris-custom-resources"], [90, "using-netris-custom-resources"], [95, "using-netris-custom-resources"], [100, "using-netris-custom-resources"]], "Introduction to Netris Custom Resources": [[19, "introduction-to-netris-custom-resources"], [30, "introduction-to-netris-custom-resources"], [35, "introduction-to-netris-custom-resources"], [40, "introduction-to-netris-custom-resources"], [45, "introduction-to-netris-custom-resources"], [50, "introduction-to-netris-custom-resources"], [55, "introduction-to-netris-custom-resources"], [60, "introduction-to-netris-custom-resources"], [65, "introduction-to-netris-custom-resources"], [70, "introduction-to-netris-custom-resources"], [75, "introduction-to-netris-custom-resources"], [80, "introduction-to-netris-custom-resources"], [85, "introduction-to-netris-custom-resources"], [90, "introduction-to-netris-custom-resources"], [95, "introduction-to-netris-custom-resources"], [100, "introduction-to-netris-custom-resources"]], "L4LB Custom Resource": [[19, "l4lb-custom-resource"], [30, "l4lb-custom-resource"], [35, "l4lb-custom-resource"], [40, "l4lb-custom-resource"], [45, "l4lb-custom-resource"], [50, "l4lb-custom-resource"], [55, "l4lb-custom-resource"], [60, "l4lb-custom-resource"], [65, "l4lb-custom-resource"], [70, "l4lb-custom-resource"], [75, "l4lb-custom-resource"], [80, "l4lb-custom-resource"], [85, "l4lb-custom-resource"], [90, "l4lb-custom-resource"], [95, "l4lb-custom-resource"], [100, "l4lb-custom-resource"]], "V-Net Custom Resource": [[19, "v-net-custom-resource"], [30, "v-net-custom-resource"], [35, "v-net-custom-resource"], [40, "v-net-custom-resource"], [45, "v-net-custom-resource"], [50, "v-net-custom-resource"], [55, "v-net-custom-resource"], [60, "v-net-custom-resource"], [65, "v-net-custom-resource"], [70, "v-net-custom-resource"], [75, "v-net-custom-resource"], [80, "v-net-custom-resource"], [85, "v-net-custom-resource"], [90, "v-net-custom-resource"], [95, "v-net-custom-resource"], [100, "v-net-custom-resource"]], "BGP Custom Resource": [[19, "bgp-custom-resource"], [30, "bgp-custom-resource"], [35, "bgp-custom-resource"], [40, "bgp-custom-resource"], [45, "bgp-custom-resource"], [50, "bgp-custom-resource"], [55, "bgp-custom-resource"], [60, "bgp-custom-resource"], [65, "bgp-custom-resource"], [70, "bgp-custom-resource"], [75, "bgp-custom-resource"], [80, "bgp-custom-resource"], [85, "bgp-custom-resource"], [90, "bgp-custom-resource"], [95, "bgp-custom-resource"], [100, "bgp-custom-resource"]], "Importing existing resources from Netris Controller to Kubernetes": [[19, "importing-existing-resources-from-netris-controller-to-kubernetes"]], "Reclaim Policy": [[19, "reclaim-policy"], [30, "reclaim-policy"], [35, "reclaim-policy"], [40, "reclaim-policy"], [45, "reclaim-policy"], [50, "reclaim-policy"], [55, "reclaim-policy"], [60, "reclaim-policy"], [65, "reclaim-policy"], [70, "reclaim-policy"], [75, "reclaim-policy"], [80, "reclaim-policy"], [85, "reclaim-policy"], [90, "reclaim-policy"], [95, "reclaim-policy"], [100, "reclaim-policy"]], "Calico CNI Integration": [[19, "calico-cni-integration"]], "Disabling Netris-Calico Integration": [[19, "disabling-netris-calico-integration"], [30, "disabling-netris-calico-integration"], [35, "disabling-netris-calico-integration"], [40, "disabling-netris-calico-integration"], [45, "disabling-netris-calico-integration"], [50, "disabling-netris-calico-integration"], [55, "disabling-netris-calico-integration"], [60, "disabling-netris-calico-integration"], [65, "disabling-netris-calico-integration"], [70, "disabling-netris-calico-integration"], [75, "disabling-netris-calico-integration"], [80, "disabling-netris-calico-integration"], [85, "disabling-netris-calico-integration"], [90, "disabling-netris-calico-integration"], [95, "disabling-netris-calico-integration"], [100, "disabling-netris-calico-integration"]], "L3 Load Balancer (Anycast LB)": [[20, "l3-load-balancer-anycast-lb"]], "Creating an L3 Load Balancer": [[20, "creating-an-l3-load-balancer"]], "L4 Load Balancer (L4LB)": [[21, "l4-load-balancer-l4lb"]], "Enabling L4LB service": [[21, "enabling-l4lb-service"]], "Consuming L4LB service": [[21, "consuming-l4lb-service"]], "Netris Architecture": [[22, "netris-architecture"]], "Netris Controller": [[22, "netris-controller"], [31, "netris-controller"], [36, "netris-controller"], [41, "netris-controller"], [46, "netris-controller"], [51, "netris-controller"], [56, "netris-controller"], [61, "netris-controller"], [66, "netris-controller"], [71, "netris-controller"], [76, "netris-controller"], [81, "netris-controller"], [91, "netris-controller"], [96, "netris-controller"], [101, "netris-controller"]], "Netris Switch Agent": [[22, "netris-switch-agent"]], "Netris SoftGate": [[22, "netris-softgate"]], "Basic BGP": [[23, "basic-bgp"]], "Adding BGP Peers": [[23, "adding-bgp-peers"]], "BGP Peer Fields": [[23, "id1"]], "Advanced BGP": [[23, "advanced-bgp"]], "BGP Peer Fields - Advanced": [[23, "id2"]], "BGP Objects": [[23, "bgp-objects"]], "IPv4 Prefix": [[23, "ipv4-prefix"]], "IPv6 Prefix": [[23, "ipv6-prefix"]], "Community": [[23, "community"]], "BGP route-maps": [[23, "bgp-route-maps"]], "Static Routing": [[23, "static-routing"]], "NAT": [[23, "nat"]], "Enabling NAT": [[23, "enabling-nat"]], "Defining NAT rules": [[23, "defining-nat-rules"]], "NAT Rule Fields": [[23, "id3"]], "SiteMesh": [[23, "sitemesh"]], "Looking Glass": [[23, "looking-glass"]], "Network Reference Designs": [[24, "network-reference-designs"]], "Release notes": [[25, "release-notes"]], "ROH (Routing on the Host)": [[26, "roh-routing-on-the-host"]], "Adding ROH Hosts": [[26, "adding-roh-hosts"]], "Provided Example Configurations": [[27, "provided-example-configurations"], [32, "provided-example-configurations"], [37, "provided-example-configurations"], [42, "provided-example-configurations"], [47, "provided-example-configurations"], [52, "provided-example-configurations"], [57, "provided-example-configurations"], [62, "provided-example-configurations"], [67, "provided-example-configurations"], [72, "provided-example-configurations"], [77, "provided-example-configurations"], [82, "provided-example-configurations"], [87, "provided-example-configurations"], [92, "provided-example-configurations"], [97, "provided-example-configurations"]], "V-Net (Ethernet/Vlan/VXlan) Example": [[27, "v-net-ethernet-vlan-vxlan-example"], [32, "v-net-ethernet-vlan-vxlan-example"], [37, "v-net-ethernet-vlan-vxlan-example"], [42, "v-net-ethernet-vlan-vxlan-example"], [47, "v-net-ethernet-vlan-vxlan-example"], [52, "v-net-ethernet-vlan-vxlan-example"], [57, "v-net-ethernet-vlan-vxlan-example"], [62, "v-net-ethernet-vlan-vxlan-example"], [67, "v-net-ethernet-vlan-vxlan-example"], [72, "v-net-ethernet-vlan-vxlan-example"], [77, "v-net-ethernet-vlan-vxlan-example"], [82, "v-net-ethernet-vlan-vxlan-example"], [87, "v-net-ethernet-vlan-vxlan-example"], [92, "v-net-ethernet-vlan-vxlan-example"], [97, "v-net-ethernet-vlan-vxlan-example"]], "E-BGP (Exterior Border Gateway Protocol) Example": [[27, "e-bgp-exterior-border-gateway-protocol-example"], [32, "e-bgp-exterior-border-gateway-protocol-example"], [37, "e-bgp-exterior-border-gateway-protocol-example"], [42, "e-bgp-exterior-border-gateway-protocol-example"], [47, "e-bgp-exterior-border-gateway-protocol-example"], [52, "e-bgp-exterior-border-gateway-protocol-example"], [57, "e-bgp-exterior-border-gateway-protocol-example"], [62, "e-bgp-exterior-border-gateway-protocol-example"], [67, "e-bgp-exterior-border-gateway-protocol-example"], [72, "e-bgp-exterior-border-gateway-protocol-example"], [77, "e-bgp-exterior-border-gateway-protocol-example"], [82, "e-bgp-exterior-border-gateway-protocol-example"], [87, "e-bgp-exterior-border-gateway-protocol-example"], [92, "e-bgp-exterior-border-gateway-protocol-example"], [97, "e-bgp-exterior-border-gateway-protocol-example"]], "NAT (Network Address Translation) Example": [[27, "nat-network-address-translation-example"], [32, "nat-network-address-translation-example"], [37, "nat-network-address-translation-example"], [42, "nat-network-address-translation-example"], [47, "nat-network-address-translation-example"], [52, "nat-network-address-translation-example"], [57, "nat-network-address-translation-example"], [62, "nat-network-address-translation-example"], [67, "nat-network-address-translation-example"], [72, "nat-network-address-translation-example"], [77, "nat-network-address-translation-example"], [82, "nat-network-address-translation-example"], [87, "nat-network-address-translation-example"], [92, "nat-network-address-translation-example"], [97, "nat-network-address-translation-example"]], "ACL (Access Control List) Example": [[27, "acl-access-control-list-example"], [32, "acl-access-control-list-example"], [37, "acl-access-control-list-example"], [42, "acl-access-control-list-example"], [47, "acl-access-control-list-example"], [52, "acl-access-control-list-example"], [57, "acl-access-control-list-example"], [62, "acl-access-control-list-example"], [67, "acl-access-control-list-example"], [72, "acl-access-control-list-example"], [77, "acl-access-control-list-example"], [82, "acl-access-control-list-example"], [87, "acl-access-control-list-example"], [92, "acl-access-control-list-example"], [97, "acl-access-control-list-example"]], "Learn by Creating Services": [[28, "learn-by-creating-services"], [33, "learn-by-creating-services"], [38, "learn-by-creating-services"], [43, "learn-by-creating-services"], [48, "learn-by-creating-services"], [53, "learn-by-creating-services"], [58, "learn-by-creating-services"], [63, "learn-by-creating-services"], [68, "learn-by-creating-services"], [73, "learn-by-creating-services"], [78, "learn-by-creating-services"], [83, "learn-by-creating-services"], [88, "learn-by-creating-services"], [93, "learn-by-creating-services"], [98, "learn-by-creating-services"]], "V-Net (Ethernet/Vlan/VXlan)": [[28, "v-net-ethernet-vlan-vxlan"], [33, "v-net-ethernet-vlan-vxlan"], [38, "v-net-ethernet-vlan-vxlan"], [43, "v-net-ethernet-vlan-vxlan"], [48, "v-net-ethernet-vlan-vxlan"], [53, "v-net-ethernet-vlan-vxlan"], [58, "v-net-ethernet-vlan-vxlan"], [63, "v-net-ethernet-vlan-vxlan"], [68, "v-net-ethernet-vlan-vxlan"], [73, "v-net-ethernet-vlan-vxlan"], [78, "v-net-ethernet-vlan-vxlan"], [83, "v-net-ethernet-vlan-vxlan"], [88, "v-net-ethernet-vlan-vxlan"], [93, "v-net-ethernet-vlan-vxlan"], [98, "v-net-ethernet-vlan-vxlan"]], "E-BGP (Exterior Border Gateway Protocol)": [[28, "e-bgp-exterior-border-gateway-protocol"], [33, "e-bgp-exterior-border-gateway-protocol"], [38, "e-bgp-exterior-border-gateway-protocol"], [43, "e-bgp-exterior-border-gateway-protocol"], [48, "e-bgp-exterior-border-gateway-protocol"], [53, "e-bgp-exterior-border-gateway-protocol"], [58, "e-bgp-exterior-border-gateway-protocol"], [63, "e-bgp-exterior-border-gateway-protocol"], [68, "e-bgp-exterior-border-gateway-protocol"], [73, "e-bgp-exterior-border-gateway-protocol"], [78, "e-bgp-exterior-border-gateway-protocol"], [83, "e-bgp-exterior-border-gateway-protocol"], [88, "e-bgp-exterior-border-gateway-protocol"], [93, "e-bgp-exterior-border-gateway-protocol"], [98, "e-bgp-exterior-border-gateway-protocol"]], "NAT (Network Address Translation)": [[28, "nat-network-address-translation"], [33, "nat-network-address-translation"], [38, "nat-network-address-translation"], [43, "nat-network-address-translation"], [48, "nat-network-address-translation"], [53, "nat-network-address-translation"], [58, "nat-network-address-translation"], [63, "nat-network-address-translation"], [68, "nat-network-address-translation"], [73, "nat-network-address-translation"], [78, "nat-network-address-translation"], [83, "nat-network-address-translation"], [88, "nat-network-address-translation"], [93, "nat-network-address-translation"], [98, "nat-network-address-translation"]], "L3LB (Anycast L3 load balancer)": [[28, "l3lb-anycast-l3-load-balancer"], [33, "l3lb-anycast-l3-load-balancer"], [38, "l3lb-anycast-l3-load-balancer"], [43, "l3lb-anycast-l3-load-balancer"], [48, "l3lb-anycast-l3-load-balancer"], [53, "l3lb-anycast-l3-load-balancer"], [58, "l3lb-anycast-l3-load-balancer"], [63, "l3lb-anycast-l3-load-balancer"], [68, "l3lb-anycast-l3-load-balancer"], [73, "l3lb-anycast-l3-load-balancer"], [78, "l3lb-anycast-l3-load-balancer"], [88, "l3lb-anycast-l3-load-balancer"], [93, "l3lb-anycast-l3-load-balancer"], [98, "l3lb-anycast-l3-load-balancer"]], "ACL (Access Control List)": [[28, "acl-access-control-list"], [33, "acl-access-control-list"], [38, "acl-access-control-list"], [43, "acl-access-control-list"], [48, "acl-access-control-list"], [53, "acl-access-control-list"], [58, "acl-access-control-list"], [63, "acl-access-control-list"], [68, "acl-access-control-list"], [73, "acl-access-control-list"], [78, "acl-access-control-list"], [83, "acl-access-control-list"], [88, "acl-access-control-list"], [93, "acl-access-control-list"], [98, "acl-access-control-list"]], "Sandbox1": [[29, "sandbox1"]], "Learn Netris Operations with Kubernetes": [[30, "learn-netris-operations-with-kubernetes"], [35, "learn-netris-operations-with-kubernetes"]], "Intro": [[30, "intro"], [35, "intro"], [40, "intro"], [45, "intro"], [50, "intro"], [55, "intro"], [60, "intro"], [65, "intro"], [70, "intro"], [75, "intro"], [80, "intro"], [85, "intro"], [90, "intro"], [95, "intro"], [100, "intro"]], "Deploy an Application with an On-Demand Netris Load Balancer": [[30, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [35, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [40, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [45, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [50, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [55, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [60, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [65, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [70, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [75, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [80, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [85, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [90, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [95, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [100, "deploy-an-application-with-an-on-demand-netris-load-balancer"]], "Milestone 1": [[30, null], [35, null], [40, null], [45, null], [50, null], [55, null], [60, null], [65, null], [70, null], [75, null], [80, null], [85, null], [90, null], [95, null], [100, null]], "Importing Existing Resources from Netris Controller to Kubernetes": [[30, "importing-existing-resources-from-netris-controller-to-kubernetes"], [35, "importing-existing-resources-from-netris-controller-to-kubernetes"], [40, "importing-existing-resources-from-netris-controller-to-kubernetes"], [45, "importing-existing-resources-from-netris-controller-to-kubernetes"], [50, "importing-existing-resources-from-netris-controller-to-kubernetes"], [55, "importing-existing-resources-from-netris-controller-to-kubernetes"], [60, "importing-existing-resources-from-netris-controller-to-kubernetes"], [65, "importing-existing-resources-from-netris-controller-to-kubernetes"], [70, "importing-existing-resources-from-netris-controller-to-kubernetes"], [75, "importing-existing-resources-from-netris-controller-to-kubernetes"], [80, "importing-existing-resources-from-netris-controller-to-kubernetes"], [85, "importing-existing-resources-from-netris-controller-to-kubernetes"], [90, "importing-existing-resources-from-netris-controller-to-kubernetes"], [95, "importing-existing-resources-from-netris-controller-to-kubernetes"], [100, "importing-existing-resources-from-netris-controller-to-kubernetes"]], "Netris Calico CNI Integration": [[30, "netris-calico-cni-integration"], [35, "netris-calico-cni-integration"], [40, "netris-calico-cni-integration"], [45, "netris-calico-cni-integration"], [50, "netris-calico-cni-integration"], [55, "netris-calico-cni-integration"], [60, "netris-calico-cni-integration"], [65, "netris-calico-cni-integration"], [70, "netris-calico-cni-integration"], [75, "netris-calico-cni-integration"], [80, "netris-calico-cni-integration"], [85, "netris-calico-cni-integration"], [90, "netris-calico-cni-integration"], [95, "netris-calico-cni-integration"], [100, "netris-calico-cni-integration"]], "Milestone 2": [[30, null], [35, null], [40, null], [45, null], [50, null], [55, null], [60, null], [65, null], [70, null], [75, null], [80, null], [85, null], [90, null], [95, null], [100, null]], "Welcome to Netris Sandbox": [[31, "welcome-to-netris-sandbox"], [36, "welcome-to-netris-sandbox"], [41, "welcome-to-netris-sandbox"], [46, "welcome-to-netris-sandbox"], [51, "welcome-to-netris-sandbox"], [56, "welcome-to-netris-sandbox"], [61, "welcome-to-netris-sandbox"], [66, "welcome-to-netris-sandbox"], [71, "welcome-to-netris-sandbox"], [76, "welcome-to-netris-sandbox"], [81, "welcome-to-netris-sandbox"], [86, "welcome-to-netris-sandbox"], [91, "welcome-to-netris-sandbox"], [96, "welcome-to-netris-sandbox"], [101, "welcome-to-netris-sandbox"]], "Topology diagram": [[31, "topology-diagram"], [36, "topology-diagram"], [41, "topology-diagram"], [46, "topology-diagram"], [51, "topology-diagram"], [56, "topology-diagram"], [61, "topology-diagram"], [66, "topology-diagram"], [71, "topology-diagram"], [76, "topology-diagram"], [81, "topology-diagram"], [86, "topology-diagram"], [91, "topology-diagram"], [96, "topology-diagram"], [101, "topology-diagram"]], "Linux servers": [[31, "linux-servers"], [36, "linux-servers"], [41, "linux-servers"], [46, "linux-servers"], [51, "linux-servers"], [56, "linux-servers"], [61, "linux-servers"], [66, "linux-servers"], [71, "linux-servers"], [76, "linux-servers"], [81, "linux-servers"], [86, "linux-servers"], [91, "linux-servers"], [96, "linux-servers"], [101, "linux-servers"]], "Kubernetes cluster": [[31, "kubernetes-cluster"], [36, "kubernetes-cluster"], [41, "kubernetes-cluster"], [46, "kubernetes-cluster"], [51, "kubernetes-cluster"], [56, "kubernetes-cluster"], [61, "kubernetes-cluster"], [66, "kubernetes-cluster"], [71, "kubernetes-cluster"], [76, "kubernetes-cluster"], [81, "kubernetes-cluster"], [86, "kubernetes-cluster"], [91, "kubernetes-cluster"], [96, "kubernetes-cluster"], [101, "kubernetes-cluster"]], "Upstream ISP": [[31, "upstream-isp"], [36, "upstream-isp"], [41, "upstream-isp"], [46, "upstream-isp"], [51, "upstream-isp"], [56, "upstream-isp"], [61, "upstream-isp"], [66, "upstream-isp"], [71, "upstream-isp"], [76, "upstream-isp"], [81, "upstream-isp"], [86, "upstream-isp"], [91, "upstream-isp"], [96, "upstream-isp"], [101, "upstream-isp"]], "Networks Used": [[31, "networks-used"], [36, "networks-used"], [41, "networks-used"], [46, "networks-used"], [51, "networks-used"], [56, "networks-used"], [61, "networks-used"], [66, "networks-used"], [71, "networks-used"], [76, "networks-used"], [81, "networks-used"], [91, "networks-used"], [96, "networks-used"], [101, "networks-used"]], "Sandbox10": [[34, "sandbox10"]], "Sandbox11": [[39, "sandbox11"]], "Learn Netris operations with Kubernetes": [[40, "learn-netris-operations-with-kubernetes"], [45, "learn-netris-operations-with-kubernetes"], [50, "learn-netris-operations-with-kubernetes"], [55, "learn-netris-operations-with-kubernetes"], [60, "learn-netris-operations-with-kubernetes"], [65, "learn-netris-operations-with-kubernetes"], [70, "learn-netris-operations-with-kubernetes"], [75, "learn-netris-operations-with-kubernetes"], [80, "learn-netris-operations-with-kubernetes"], [85, "learn-netris-operations-with-kubernetes"], [90, "learn-netris-operations-with-kubernetes"], [95, "learn-netris-operations-with-kubernetes"], [100, "learn-netris-operations-with-kubernetes"]], "Sandbox12": [[44, "sandbox12"]], "Sandbox13": [[49, "sandbox13"]], "Sandbox14": [[54, "sandbox14"]], "Sandbox15": [[59, "sandbox15"]], "Sandbox2": [[64, "sandbox2"]], "Sandbox3": [[69, "sandbox3"]], "Sandbox4": [[74, "sandbox4"]], "Sandbox5": [[79, "sandbox5"]], "Sandbox6": [[84, "sandbox6"]], "Netris GUI": [[86, "netris-gui"]], "Networks used": [[86, "networks-used"]], "Sandbox7": [[89, "sandbox7"]], "Sandbox8": [[94, "sandbox8"]], "Sandbox9": [[99, "sandbox9"]], "SoftGate Performance": [[102, "softgate-performance"], [102, "id1"]], "Reference Network Architectures": [[103, "reference-network-architectures"]], "Unmanaged Switch & Netris SoftGate": [[103, "unmanaged-switch-netris-softgate"]], "Unmanaged Switch & SoftGate (HA)": [[103, "unmanaged-switch-softgate-ha"]], "Netris Managed Switch & SoftGate (HA)": [[103, "netris-managed-switch-softgate-ha"]], "Netris Managed Switch & SoftGate scalable data center (HA)": [[103, "netris-managed-switch-softgate-scalable-data-center-ha"]], "Network Switch Initial Setup": [[104, "id1"]], "Switch Ports": [[105, "switch-ports"]], "Terraform: Netris provider": [[106, "terraform-netris-provider"]], "To create your first Terraform configuration:": [[106, "to-create-your-first-terraform-configuration"]], "Install Terraform": [[106, "install-terraform"]], "Create a directory for Terraform files": [[106, "create-a-directory-for-terraform-files"]], "Configure a provider": [[106, "configure-a-provider"]], "Prepare an infrastructure plan": [[106, "prepare-an-infrastructure-plan"]], "Create resources": [[106, "create-resources"]], "Delete resources": [[106, "delete-resources"]], "Inventory": [[107, "inventory"]], "Adding Switches": [[107, "adding-switches"]], "Add Inventory Fields - Switch": [[107, "id2"]], "Adding SoftGates": [[107, "adding-softgates"]], "Add Inventory Fields - SoftGate": [[107, "id3"]], "Viewing Inventory": [[107, "viewing-inventory"]], "Topology Manager": [[107, "topology-manager"]], "Adding Links": [[107, "adding-links"]], "Hairpin Links (Nvidia Cumulus only)": [[107, "hairpin-links-nvidia-cumulus-only"]], "Activating BGP on Equinix Metal Project": [[108, "activating-bgp-on-equinix-metal-project"]], "Provisioning Netris SoftGate nodes in Equinix Metal Project": [[109, "provisioning-netris-softgate-nodes-in-equinix-metal-project"]], "Site Mesh with AWS Overview": [[110, "site-mesh-with-aws-overview"]], "Introduction": [[110, "introduction"], [115, "introduction"], [129, "introduction"], [142, "introduction"]], "Concept": [[110, "concept"], [115, "concept"], [129, "concept"], [142, "concept"]], "Deploy a Softgate in AWS": [[111, "deploy-a-softgate-in-aws"]], "Create an EC2 instance": [[111, "create-an-ec2-instance"]], "Configure Netris Controller": [[111, "configure-netris-controller"], [116, "configure-netris-controller"]], "Pre-Requisite Steps": [[111, "pre-requisite-steps"], [116, "pre-requisite-steps"]], "Add AWS VPC Subnet into the Netris Controller": [[111, "add-aws-vpc-subnet-into-the-netris-controller"]], "Static route for AWS VPC Subnet in Netris Controller": [[111, "static-route-for-aws-vpc-subnet-in-netris-controller"]], "Create the Softgate in the Netris Controller": [[111, "create-the-softgate-in-the-netris-controller"], [116, "create-the-softgate-in-the-netris-controller"]], "Netris SoftGate node provisioning": [[111, "netris-softgate-node-provisioning"], [116, "netris-softgate-node-provisioning"]], "Configure AWS VPC": [[111, "configure-aws-vpc"]], "Adding routes": [[111, "adding-routes"]], "EC2 Source / destination check": [[111, "ec2-source-destination-check"]], "Enabling Site Mesh": [[111, "enabling-site-mesh"], [116, "enabling-site-mesh"]], "Creating an Interconnection to Equinix Fabric": [[112, "creating-an-interconnection-to-equinix-fabric"]], "Enabling services (NAT, V-Net, Load Balancer, IP pools)": [[113, "enabling-services-nat-v-net-load-balancer-ip-pools"]], "1) Requesting new Public IP address block": [[113, "requesting-new-public-ip-address-block"]], "2) Enable on-demand (elastic) Load Balancer": [[113, "enable-on-demand-elastic-load-balancer"]], "3) Enable V-Net": [[113, "enable-v-net"]], "4) Enable NAT": [[113, "enable-nat"]], "Enable Equinix Metal API integration": [[114, "enable-equinix-metal-api-integration"]], "Site Mesh with GCP Overview": [[115, "site-mesh-with-gcp-overview"]], "Deploy a Softgate in GCP": [[116, "deploy-a-softgate-in-gcp"]], "Create a VM instance": [[116, "create-a-vm-instance"]], "Add GCP VPC Subnet(s) into the Netris Controller": [[116, "add-gcp-vpc-subnet-s-into-the-netris-controller"]], "Static route for GCP VPC Subnet(s) in Netris Controller": [[116, "static-route-for-gcp-vpc-subnet-s-in-netris-controller"]], "Configure routing in GCP VPC": [[116, "configure-routing-in-gcp-vpc"]], "Equinix Metal API integration enablement": [[117, "equinix-metal-api-integration-enablement"]], "Adding Netris SoftGate nodes": [[117, "adding-netris-softgate-nodes"]], "Netris Generic Tutorials": [[118, "netris-generic-tutorials"]], "Netris VPC for Equinix Metal Getting Started Guide": [[119, "netris-vpc-for-equinix-metal-getting-started-guide"]], "Step-by-step Netris Setup Guide for Equinix Metal": [[119, null]], "Examples of Using Netris VPC on Equinix Metal Project": [[119, null]], "VPC for Anywhere Getting Started Guide": [[120, "vpc-for-anywhere-getting-started-guide"]], "Installing a Netris Controller on Equinix Metal on-demand server": [[121, "installing-a-netris-controller-on-equinix-metal-on-demand-server"]], "Installation Steps": [[121, "installation-steps"], [130, "installation-steps"]], "Security Matters": [[121, "security-matters"], [122, "security-matters"], [130, "security-matters"], [143, "security-matters"]], "Enabling Site Mesh with AWS": [[123, "enabling-site-mesh-with-aws"]], "Setup": [[123, null], [124, null], [125, null], [126, null], [140, null]], "Netris VPC for Equinix Metal": [[124, "netris-vpc-for-equinix-metal"]], "Use": [[124, null], [126, null], [140, null]], "Enabling Site Mesh with GCP": [[125, "enabling-site-mesh-with-gcp"]], "Netris VPC for phoenixNAP BMC": [[126, "netris-vpc-for-phoenixnap-bmc"]], "Provisioning Netris SoftGate nodes in phoenixNAP BMC": [[127, "provisioning-netris-softgate-nodes-in-phoenixnap-bmc"]], "Enable phoenixNAP BMC API integration": [[128, "enable-phoenixnap-bmc-api-integration"]], "Netris VPC for phoenixNAP BMC Overview": [[129, "netris-vpc-for-phoenixnap-bmc-overview"]], "Installing a Netris Controller on phoenixNAP BMC server": [[130, "installing-a-netris-controller-on-phoenixnap-bmc-server"]], "IPAM Setup for Services": [[131, "ipam-setup-for-services"], [144, "ipam-setup-for-services"]], "Option 1 - Separated allocation for each purpose": [[131, "option-1-separated-allocation-for-each-purpose"]], "1. Enable on-demand (elastic) Load Balancer": [[131, "enable-on-demand-elastic-load-balancer"], [131, "id1"]], "2. Enable NAT": [[131, "enable-nat"], [131, "id2"]], "Option 2 - Splitting single allocation into different subnets": [[131, "option-2-splitting-single-allocation-into-different-subnets"]], "Note*": [[131, "note"]], "Installing Netris on phoenixNAP BMC": [[132, "installing-netris-on-phoenixnap-bmc"]], "Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC": [[133, "using-on-demand-elastic-l4-load-balancer-service-in-phoenixnap-bmc"]], "Using NAT services in phoenixNAP BMC": [[134, "using-nat-services-in-phoenixnap-bmc"]], "Option 1 - MASQUERADE": [[134, "option-1-masquerade"]], "Option 2 - SNAT": [[134, "option-2-snat"]], "DNAT": [[134, "dnat"], [149, "dnat"]], "Using V-Net (isolated virtual network) services in phoenixNAP BMC": [[135, "using-v-net-isolated-virtual-network-services-in-phoenixnap-bmc"]], "Adding Subnets for V-Net": [[135, "adding-subnets-for-v-net"]], "Creating a V-Net": [[135, "creating-a-v-net"], [148, "creating-a-v-net"], [150, "creating-a-v-net"]], "Deploy a new server into an existing V-Net": [[135, "deploy-a-new-server-into-an-existing-v-net"]], "Tags": [[135, "tags"], [139, "tags"]], "Unmanaged": [[135, "unmanaged"], [139, "unmanaged"]], "Netris Upgrade and Rollback Procedures": [[136, "netris-upgrade-and-rollback-procedures"]], "Upgrade Procedure": [[136, "upgrade-procedure"], [137, "upgrade-procedure"]], "Rollback Procedure": [[136, "rollback-procedure"]], "Upgrading SONiC Operating System": [[137, "upgrading-sonic-operating-system"]], "Using on-demand (elastic) L4 Load Balancer service": [[138, "using-on-demand-elastic-l4-load-balancer-service"], [147, "using-on-demand-elastic-l4-load-balancer-service"]], "Using V-Net (isolated virtual network) services in Equinix Metal Project": [[139, "using-v-net-isolated-virtual-network-services-in-equinix-metal-project"]], "VPC Anywhere Getting Started Guide": [[140, "vpc-anywhere-getting-started-guide"]], "Check Default Site Settings": [[141, "check-default-site-settings"]], "VPC Anywhere Overview": [[142, "vpc-anywhere-overview"]], "Install a Netris Controller": [[143, "install-a-netris-controller"]], "Requirements and Installation steps": [[143, "requirements-and-installation-steps"]], "Create an allocation": [[144, "create-an-allocation"]], "Enable on-demand L4 (elastic) Load Balancer": [[144, "enable-on-demand-l4-elastic-load-balancer"]], "Enable Network Address Translation (NAT)": [[144, "enable-network-address-translation-nat"]], "SoftGate software provisioning": [[145, "softgate-software-provisioning"]], "Connecting VPC to upstream networks (use one of two options)": [[146, "connecting-vpc-to-upstream-networks-use-one-of-two-options"]], "Using a VLAN with public IP addresses (DMZ)": [[146, "using-a-vlan-with-public-ip-addresses-dmz"]], "BGP Upstream": [[146, "bgp-upstream"]], "BGP Upstream: overview": [[146, "bgp-upstream-overview"]], "BGP Upstream: configuration": [[146, "bgp-upstream-configuration"]], "Using Multi-interface SoftGate (experimental)": [[148, "using-multi-interface-softgate-experimental"]], "SoftGate to SoftGate Link": [[148, "softgate-to-softgate-link"]], "Using NAT services": [[149, "using-nat-services"]], "MASQUERADE": [[149, "masquerade"]], "SNAT": [[149, "snat"]], "Using V-Net (isolated virtual network) services": [[150, "using-v-net-isolated-virtual-network-services"]], "Visibility (Telescope)": [[151, "visibility-telescope"]], "Graph Boards": [[151, "graph-boards"]], "API Logs": [[151, "api-logs"]], "Dashboard": [[151, "dashboard"]], "V-Net": [[152, "v-net"]], "V-Net Fields": [[152, "v-net-fields"]]}, "indexentries": {}}) \ No newline at end of file diff --git a/en/3.4/softgate-performance.html b/en/3.4/softgate-performance.html new file mode 100644 index 0000000000..d68e320e6e --- /dev/null +++ b/en/3.4/softgate-performance.html @@ -0,0 +1,760 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate Performance — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

SoftGate Performance

+

The following tested results are offered to help properly size the hardware needed for a SoftGate with various types of services:

+ + +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SoftGate Performance

Routing(mpps)

Routing with unmatched DNAT rules (mpps)

Routing with unmatched DNAT and SNAT rules (mpps)

DNAT

SNAT

DNAT with unmatched SNAT rules

Xeon E5-2660 CPU cores: (6 core for fw) Scheduler 0 Forward 1-7 KNI 8-10 OS 11-15

~27mpps

~17mpps

7mpps

9mpps

~2mpps

Xeon Gold 6130 CPU cores: (6 core for fw) Scheduler 0 Forward 3-13(odd numbers) KNI 15 17 19

~31mpps

~22mpps

~15mpps

~11mpps

~3mpps(3M conntrack count) ~7mpps(20k conntrack count)

~11mpps

Xeon Gold 6130 CPU cores: (8 core for fw) Scheduler 0 Forward 3-17(odd numbers) KNI 19 21 23

~38mpps

~28mpps

19mpps

12mpps

~12mpps

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/supported-networks.html b/en/3.4/supported-networks.html new file mode 100644 index 0000000000..35449305cd --- /dev/null +++ b/en/3.4/supported-networks.html @@ -0,0 +1,732 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Reference Network Architectures — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Reference Network Architectures
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Reference Network Architectures

+
+

Unmanaged Switch & Netris SoftGate

+Unmanaged Switch & SoftGate +
+
+
+

Unmanaged Switch & SoftGate (HA)

+Unmanaged Switch & SoftGate (HA) +
+
+
+

Netris Managed Switch & SoftGate (HA)

+Netris Managed Switch & SoftGate small data center (HA) +
+
+
+

Netris Managed Switch & SoftGate scalable data center (HA)

+Netris Managed Switch & SoftGate scalable data center (HA) +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/switch-agent-installation.html b/en/3.4/switch-agent-installation.html new file mode 100644 index 0000000000..560bc76a61 --- /dev/null +++ b/en/3.4/switch-agent-installation.html @@ -0,0 +1,721 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Network Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Network Switch Initial Setup
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/switch-ports.html b/en/3.4/switch-ports.html new file mode 100644 index 0000000000..0df9d6309b --- /dev/null +++ b/en/3.4/switch-ports.html @@ -0,0 +1,737 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Switch Ports — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Switch Ports

+

Switch ports can be directly managed in the Switch Port UI section. Both physical and virtual ports (extended, aggregate, etc…) will appear in this section once they have been added to inventory. The Netris Controller will automatically sync the list of available ports that appear on each device.

+

The following options are available for editing on each port:

+
    +
  • Description - Description of the port.

  • +
  • Tenant - Tenant to whom the port is assigned, by default it is the owner tenant of the device to whom the port belongs to.

  • +
  • Breakout - Available only for physical switch ports, used to split physical ports into multiple physical ports. When there is a need to use other supported option supported by switch not shown in the dropdown list user must set breakout to “Manual” and configure breakout manually on the switch. For certain platforms some ports need to be disabled to support breakout into other ports, for that option use “Disable” mode of breakout. For Cumulus, after configuration, user must manually restart switchd daemon on the switch via command “systemctl restart switchd”.

  • +
  • MTU - Maximum transmission unit of the port.

  • +
  • Autoneg - Toggle autonegotiation. Available only for physical ports.

  • +
  • Speed - Toggle speed. Available only for physical ports.

  • +
  • Duplex - Toggle duplex. Available only for physical ports.

  • +
  • Extension - Create extension ports. Available for physical and aggregate ports.

  • +
  • Extension Name - Name for new extension.

  • +
  • VLAN Range - VLAN id range for new extension port.

  • +
+_images/edit-port.png +

Example: Edit physical port

+

Quick action menu provides following actions for ports (note that Bulk Action also available for multiple ports:

+

Edit - Edit the port. +Admin UP/Down - Toggle admin status of the port. +Add to V-net - Add selected port(s) to a V-net.

+_images/quick-action-ports.png +

Add to LAG - Add selected ports into a LAG.

+_images/add-to-lag-port.png +

Free Up Port - Detach port from all resources.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/terraform-integration.html b/en/3.4/terraform-integration.html new file mode 100644 index 0000000000..08247c08d7 --- /dev/null +++ b/en/3.4/terraform-integration.html @@ -0,0 +1,998 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Terraform: Netris provider — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Terraform: Netris provider
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Terraform: Netris provider

+

Use Netris provider to interact with the many resources supported by Netris. You must configure the provider with the proper credentials before you can use it. +To learn the basics of Terraform using this provider, follow the hands-on get started tutorials on HashiCorp’s Learn platform.

+_images/diagrams_terraform.png +

When you make changes in the Terraform files and apply them, Terraform automatically decides which part of your configuration is already deployed into Netris controller and what should be added or removed.

+ +
+

Install Terraform

+

Download and install the Terraform

+
+
+

Create a directory for Terraform files

+
    +
  1. Create a directory with any name, for example, netris-terraform. It stores the configuration files and saved states for Terraform and your infrastructure.

  2. +
  3. Create a configuration file with the .tf extension in this directory, such as main.tf.

  4. +
+
+
+

Configure a provider

+
    +
  1. At the beginning of the configuration file, specify the provider settings.

  2. +
+
terraform {
+  required_providers {
+    netris = {
+      source  = "netrisai/netris"
+      version = ">= 2.0.0"
+    }
+  }
+}
+
+provider "netris" {
+  address = "<controller address>"
+  login = "<controller login>"
+  password = "<controller password>"
+}
+
+
+

Specify the provider required arguments:

+
    +
  • address - This is your Netris-Controller address (http://example.com). This can also be specified with the NETRIS_ADDRESS environment variable.

  • +
  • login - This is your Netris-Controller login. This can also be specified with the NETRIS_LOGIN environment variable.

  • +
  • password - This is your Netris-Controller password. This can also be specified with the NETRIS_PASSWORD environment variable.

  • +
+
    +
  1. Execute the command terraform init in the folder with the configuration file. This command initializes the providers specified in the configuration files and lets you work with the provider resources and data sources.

  2. +
+
+
+

Prepare an infrastructure plan

+

By using Netris Terraform Provider, you can create all kinds of resources, such as Sites, IPAMs, Topology, Inventory, etc. +To create a resource, specify a set of required and optional parameters that define the resource properties. Such resource descriptions make up an infrastructure plan.

+

Infrastructure provisioning in Netris starts with Site resources. The Netris-Controller comes with the initial site Default. You can use it in your Terraform configuration files by getting its ID with the Terraform Data source element.

+

Let’s create a separate file for site resource, and get its ID via Terraform Data source element.

+
cat << EOF > site.tf
+data "netris_site" "default" {
+  name = "Default"
+}
+EOF
+
+
+

Or, you can create a new Site resource, here is the detailed documentation with examples.

+

Now, when we’re clear on Site resource usage, let’s define our IPAM. There are two types of IPAM resources in the Netris-Controller it’s Allocation and Subnet. +IPAM resources only require tenantid field, let’s get our default Admin tenant ID with the Data source element.

+
cat << EOF > tenant.tf
+data "netris_tenant" "admin"{
+  name = "Admin"
+}
+EOF
+
+
+

Then, when we have the tenantid, we can create IPAM resources.

+
cat << EOF > ipam.tf
+resource "netris_allocation" "my-allocation-mgmt" {
+  name = "my-allocation-mgmt"
+  prefix = "192.0.2.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_allocation" "my-allocation-loopback" {
+  name = "my-allocation-loopback"
+  prefix = "198.51.100.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_allocation" "my-allocation-common" {
+  name = "my-allocation-common"
+  prefix = "203.0.113.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_subnet" "my-subnet-mgmt" {
+  name = "my-subnet-mgmt"
+  prefix = "192.0.2.0/24"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "management"
+  defaultgateway = "192.0.2.254"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-mgmt,
+  ]
+}
+
+resource "netris_subnet" "my-subnet-loopback" {
+  name = "my-subnet-loopback"
+  prefix = "198.51.100.0/24"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "loopback"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-loopback,
+  ]
+}
+
+resource "netris_subnet" "my-subnet-common" {
+  name = "my-subnet-common"
+  prefix = "203.0.113.0/25"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "common"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-common,
+  ]
+}
+EOF
+
+
+

With the command above, we’ve defined 6 resources, 3 of the type of Allocation, 3 of the type of Subnet, each Subnet resource has a different purpose. +For more details, get familiar with the IPAM docs.

+

Now, when we have all the required resources let’s define our Inventory. +We’re going to create 1 SoftGate, 1 switch and connect them with a link.

+
cat << EOF > inventory.tf
+resource "netris_softgate" "my-softgate" {
+  name = "my-softgate"
+  tenantid = data.netris_tenant.admin.id
+  siteid = data.netris_site.default.id
+  description = "Softgate 1"
+  mainip = "auto"
+  mgmtip = "auto"
+  depends_on = [
+    netris_subnet.my-subnet-mgmt,
+    netris_subnet.my-subnet-loopback,
+  ]
+}
+
+resource "netris_switch" "my-switch" {
+  name = "my-switch"
+  tenantid = data.netris_tenant.admin.id
+  siteid = data.netris_site.default.id
+  description = "Switch 01"
+  nos = "cumulus_linux"
+  asnumber = "auto"
+  mainip = "auto"
+  mgmtip = "auto"
+  portcount = 16
+  depends_on = [
+    netris_subnet.my-subnet-mgmt,
+    netris_subnet.my-subnet-loopback,
+  ]
+}
+
+resource "netris_link" "sg-to-sw" {
+  ports = [
+    "swp1@my-softgate",
+    "swp16@my-switch"
+  ]
+  depends_on = [
+    netris_softgate.my-softgate,
+    netris_switch.my-switch,
+  ]
+}
+EOF
+
+
+

Next, let’s define a local L3 network for our servers, suppose we want to connect 3 servers to our switch first 3 ports

+
cat << EOF > vnet.tf
+resource "netris_vnet" "my-vnet" {
+  name = "my-vnet"
+  tenantid = data.netris_tenant.admin.id
+  state = "active"
+  sites{
+    id = data.netris_site.default.id
+    gateways {
+      prefix = "203.0.113.1/25"
+    }
+    ports {
+      name = "swp1@my-switch"
+      vlanid = 1050
+    }
+    ports {
+      name = "swp2@my-switch"
+      vlanid = 1050
+    }
+    ports {
+      name = "swp3@my-switch"
+    }
+  }
+  depends_on = [
+    netris_switch.my-switch,
+    netris_subnet.my-subnet-common,
+  ]
+}
+EOF
+
+
+

And finally, we have to provide internet connectivity to our fabric, for that we’ll define BGP resource. Suppose we’re going to connect our ISP cable to the 10th port of our switch, and want to establish the BGP session on our Softgate.

+
cat << EOF > bgp.tf
+data "netris_port" "swp10_my_switch"{
+  name = "swp10@my-switch"
+  depends_on = [netris_switch.my-switch]
+}
+
+resource "netris_bgp" "my-bgp" {
+  name = "my-bgp"
+  siteid = data.netris_site.default.id
+  hardware = "my-softgate"
+  neighboras = 23456
+  portid = data.netris_port.swp10_my_switch.id
+  vlanid = 3000
+  localip = "172.16.0.2/30"
+  remoteip = "172.16.0.1/30"
+  description = "My First BGP"
+  prefixlistinbound = ["deny 127.0.0.0/8 le 32", "permit 0.0.0.0/0 le 24"]
+  prefixlistoutbound = ["permit 192.0.2.0/24", "permit 198.51.100.0/24 le 25", "permit 203.0.113.0/24 le 26"]
+  depends_on = [netris_link.sg-to-sw]
+}
+EOF
+
+
+
+

Note

+

For more information about all resources, how to create and manage them in Terraform, see the provider’s documentation.

+
+

Now, when we’ve done with the configuration files, let’s check whether they are valid

+
terraform validate
+
+
+

If the configuration is valid, the following message is returned:

+
Success! The configuration is valid.
+
+
+
+
+

Create resources

+
    +
  1. After preparing and checking the configuration, run the command:

  2. +
+
terraform plan
+
+
+

The terminal will display a list of resources with parameters. This is a test step. No resources are created. If there are errors in the configuration, Terraform points them out.

+
    +
  1. To create resources, run the command:

  2. +
+
terraform apply
+
+
+
    +
  1. Confirm the resource creation: type yes in the terminal and press Enter.

  2. +
+

Terraform will create all the required resources and the terminal will display the progress. After creation, you can check resource availability and their settings in the Netris-Controller UI.

+
+
+

Delete resources

+
    +
  1. To delete resources created using Terraform:

  2. +
+

Run the command:

+
terraform destroy
+
+
+

After the command is executed, the terminal will display a list of resources to be deleted.

+
    +
  1. Type yes to confirm their deletion and press Enter.

  2. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/topology-management.html b/en/3.4/topology-management.html new file mode 100644 index 0000000000..6e25a1a825 --- /dev/null +++ b/en/3.4/topology-management.html @@ -0,0 +1,933 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Inventory — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Inventory

+

The Inventory section allows you to add/edit/delete network switches and SoftGates. Initial setup of a Netris managed network is a three part process:

+
    +
  1. Create Inventory Profiles

  2. +
  3. Adding Switches

  4. +
  5. Adding Softgates

  6. +
+
+

Inventory Profiles

+

Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/SoftGate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except Netris-defined and user-defined custom flows. Generated rules include:

+
    +
  • SSH from user defined subnets

  • +
  • NTP from user defined ntp services

  • +
  • DNS from user defined DNS servers

  • +
  • Custom user defined rules

  • +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + +
Inventory Profile Fields

Name

Profile name

Description

Free text description

Allow SSH from IPv4

List of IPv4 subnets allowed to ssh (one address per line)

Allow SSH from IPv6

List of IPv6 subnets allowed to ssh (one address per line)

Timezone

Devices using this inventory profile will adjust their system time to the selected timezone.

NTP servers

List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate.

DNS servers

List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate.

+

Example: In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup).

+_images/inventory-profile.png +
+
+

Adding Switches

+

Every switch needs to be added to the Netris Controller inventory. You can add new devices with the following process:

+
    +
  1. Navigate to Net→Inventory

  2. +
  3. Click the Add button

  4. +
  5. Fill in the fields as described below

  6. +
  7. Click the Add button

  8. +
+
+
+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Add Inventory Fields - Switch

Name

Name of the device

Owner Tenant

Owner tenant of the device

Description

Description of the device

Type

There are 3 types of devices that users can add/edit - Switch/Softgate/Controller. Other types are added automatically when creating services like ROH

NOS

Operating system of the device; applicable for switches only

Site

Site where the devices reside

AS Number

Private AS number of the device; applicable for switches only; recommended to be assigned automatically

Profile

Inventory profile for current device. Profiles are used for security hardening the devices

Main IP address

Main loopback IP address for the device. Can be configured manually or assigned automatically from subnet with loopback purpose defined for current site.

Management IP address

Management IP address for the device. Can be configured manually or assigned automatically from subnet with management purpose defined for current site. This IP address is configured on the out-of-band management interface of the device.

MAC address

MAC address of the device; applicable for switches only

Preliminary port count

Used for definition of topology. When the device registers with the controller the real ports are synced with inventory

Add Link

Provides functionality to define the connections between devices; mandatory for Switch and Softgate physical interconnections

+
+

Example: Add a new Switch.

+
+
_images/add-new-hardware.png +
+
+

Note

+

Repeat this process to define all your switches.

+
+
+
+

Adding SoftGates

+

Every SoftGate node needs to be added to the Netris Controller inventory. To add a SoftGate node:

+
    +
  1. Navigate to Net→Topology

  2. +
  3. Click Add

  4. +
  5. Fill in the fields as described below

  6. +
  7. Click the Add button

  8. +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Add Inventory Fields - SoftGate

Name

Descriptive name

Owner Tenant

Tenant(typically Admin); who administers this node

Description

Free text description

Hardware Type

Select SoftGate

Site

The data center where the current SoftGate node belongs.

Inventory Profile

Profile describing the timezone; DNS; NTP; and Security features

IP Address

IPv4 address for the loopback interface

Management IP address

IPv4 address for the out of band management interface

NFV Node Port

A physical port on a spine switch where the SoftGate node’s first SmartNIC port is connected. Typically each spine switch has one SoftGate node connected to it.

+NAT address

Public IP addresses to be used as global IP for SNAT/DNAT. (check Enabling NAT section of Network Policies chapter)

+NAT address pool

Public IP address subnets to be used as rolling global IP addresses for SNAT. (check Enabling NAT section of Network Policies chapter)

+

Example: Adding a SoftGate Node to Topology.

+_images/add-softgate.png +
+
+

Viewing Inventory

+

Inventory Listing shows also Heartbeat and monitoring statuses of each device.

+

Heartbeat - Shows the status of device reachability. +Health - Shows number of successful and failed checks on the device.

+
+
_images/inventory-listing.png +
+
+

Note

+

You can also add new devices in the Topology view.

+
+
+
+
+

Topology Manager

+

The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents will configure the underlying network devices according to this topology dynamically and start watching against potential failures.

+ + +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/activating-bgp-on-equinix-metal-project.html b/en/3.4/tutorials/activating-bgp-on-equinix-metal-project.html new file mode 100644 index 0000000000..d42b42142b --- /dev/null +++ b/en/3.4/tutorials/activating-bgp-on-equinix-metal-project.html @@ -0,0 +1,725 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Activating BGP on Equinix Metal Project — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Activating BGP on Equinix Metal Project

+

Why use BGP with Equinix Metal? +SoftGate nodes are like border routers to your VPC, they are routing traffic between hosts inside your project and the Internet. We are going to establish 2 BGP sessions between SoftGate nodes and Equinix Metal. So there will be 4 BGP sessions total.

+

We need these BGP sessions for moving further. In the next chapters we are going to request pools of public IP addresses, that Netris will automatically advertise to Equinix Metal, so inbound traffic “knows” how to reach Load Balancer, NAT, and other services that you will use within your VPC.

+../_images/equinix-metal-bgp-diagram.png +

You only need to activate BGP on the Equinix Metal Project. Netris will handle the rest. +In the Equinix Metal web console go to IPs & Networks → BGP then click Activate BGP on This Project. (see below screenshots)

+../_images/equinix-metal-activate-bgp.png +

Netris will handle the rest behind the scenes automatically. Netris will enable BGP peering on the Equinix Metal side, Netris will pull the metadata with the BGP info, and will automatically configure FRR (Free Range Routing BGP daemon) on both SoftGate nodes to bring up the BGP sessions up.

+

After a few minutes you should see 4 new BGP sessions in your Netris web console under Net → E-BGP. (example screenshot below).

+../_images/equinix-metal-netris-bgp-up.png +

Now your Netris VPC has established BGP sessions with Equinix Metal Project, and you can proceed to the next step.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html b/en/3.4/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html new file mode 100644 index 0000000000..fd43e9b358 --- /dev/null +++ b/en/3.4/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html @@ -0,0 +1,738 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provisioning Netris SoftGate nodes in Equinix Metal Project — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Provisioning Netris SoftGate nodes in Equinix Metal Project

+

For SoftGate nodes you can start with two c3.small.x86 or larger servers. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one.

+

Request two servers(c3.small.x86) from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned.

+
    +
  1. At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory

  2. +
+../_images/softgate-nodes-created-in-equinix.png +
    +
  1. When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”.

    +
    +

    Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location).

    +
    +
  2. +
+

Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+../_images/softgate-nodes-recognized-in-netris.png +
    +
  1. Provision SoftGate nodes.

  2. +
+

Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there.

+../_images/softgate-one-liner-provisioning.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/softgate-green.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/aws-concept.html b/en/3.4/tutorials/aws-concept.html new file mode 100644 index 0000000000..bef9f20291 --- /dev/null +++ b/en/3.4/tutorials/aws-concept.html @@ -0,0 +1,724 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Site Mesh with AWS Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Site Mesh with AWS Overview

+
+

Introduction

+

This guide provides a step-by-step process to set up and configure Netris Softgate in AWS for establishing a site mesh network between the user’s on-premises, AWS, and other cloud environments.

+
+
+

Concept

+

Netris Softgate in AWS is an EC2 instance that runs the Netris software. Therefore, you’ll first need to create an EC2 instance for Netris Softgate and install the Netris software on it. Once that’s done, you’ll need to configure the routes in your AWS VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your AWS VPC to access those destinations through the Netris Softgate EC2 instance.

+../_images/aws-concept-traffic-flows.png +

Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/aws-deploy-softgate.html b/en/3.4/tutorials/aws-deploy-softgate.html new file mode 100644 index 0000000000..30a5b4c9a7 --- /dev/null +++ b/en/3.4/tutorials/aws-deploy-softgate.html @@ -0,0 +1,842 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Deploy a Softgate in AWS — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Deploy a Softgate in AWS

+

As stated in the previous section, the following sequence of actions must be taken in order to proceed: create an EC2 instance, add Softgate into the Netris Controller, install Netris Softgate software on the EC2 instance, and configure routes in AWS VPC. Let us commence with these steps in the specified order.

+
+

Create an EC2 instance

+

Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated EC2. To achieve this, create a security group with the “All traffic” type and “Anywhere” source for both inbound and outbound rules. Afterward, an EC2 instance can be created using the security group above.

+../_images/aws-security-group.png +

To enable connectivity with other Netris sites, it is essential to create the EC2 instance in the desired VPC. Therefore, deploy a new EC2 instance with the Ubuntu 22.04 operating system installed, utilizing an instance type that meets the minimum hardware requirements of 2 virtual CPUs and 4 GB of RAM, such as t2.medium/t3.medium or any other type that satisfies these specifications. It is also recommended to allocate at least 30 GB of drive space.

+../_images/aws-softgate-deployed.png +

After successfully deploying the EC2 instance, it is crucial to take note of its Public IPv4 address. This address will be required in the upcoming step.

+
+
+

Configure Netris Controller

+

Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed.

+
+

Pre-Requisite Steps

+

In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below:

+
    +
  1. Open the Netris Web Console.

  2. +
  3. Navigate to “Net” and select “Sites”.

  4. +
  5. Click on the “+ Add” button.

  6. +
  7. Select “Dot1q Trunk” as the “Switch Fabric”.

  8. +
  9. Input a descriptive name for the site.

  10. +
  11. Specify 65500 in the “Public ASN” field.

  12. +
  13. Click “Add” to create the new site.

  14. +
+../_images/aws-netris-site-create.png +

Subsequently, it is necessary to create a private subnet for Softgate management. To achieve this, follow the steps below:

+
    +
  1. Go to the “IPAM” section under the “Net” tab.

  2. +
  3. Click on the “+ Add” button located at the top-right corner.

  4. +
  5. Enter a unique “Prefix” for the new subnet, such as 10.255.255.0/24.

  6. +
  7. Type a descriptive “Name” for the subnet.

  8. +
  9. Select the desired tenant name from the “Tenant” dropdown menu.

  10. +
  11. From the “Type” dropdown menu, select “Subnet”.

  12. +
  13. Select “management” from the “Purpose” dropdown menu.

  14. +
  15. Choose the appropriate site from the “Sites” dropdown menu.

  16. +
  17. Click on the “Add” button to create the subnet.

  18. +
+../_images/aws-netris-ipam-mgmt.png +

Following the creation of the private subnet for Softgate management, it is necessary to create another subnet with the “Purpose” of loopback. The “Prefix” for this subnet will be the Public IPv4 address of the AWS EC2 instance with a netmask length of /32. For instance, if the EC2 instance’s IP address is 54.176.11.144, then the “Prefix” for the loopback subnet will be 54.176.11.144/32. However, unlike the subnet for Softgate management, an allocation for that “Prefix” must first be created before creating the subnet for loopback.

+../_images/aws-netris-ipam-lo.png +
+
+

Add AWS VPC Subnet into the Netris Controller

+

To register your AWS VPC’s entire CIDR block into Netris IPAM, follow these steps:

+
    +
  1. From the AWS Console, navigate to your VPC and take note of your CIDR blocks.

  2. +
  3. In Netris Controller, go to the “IPAM” section under the “Net” tab.

  4. +
  5. Click the “+ Add” button located at the top-right corner.

  6. +
  7. Enter the VPC CIDR block into the “Prefix” field for the new subnet. For example, if your CIDR block is “172.31.0.0/16”, enter that value.

  8. +
  9. Type a descriptive name for the subnet.

  10. +
  11. From the “Tenant” dropdown menu, select the desired tenant name.

  12. +
  13. From the “Type” dropdown menu, select “Subnet”.

  14. +
  15. Select “inactive” from the “Purpose” dropdown menu.

  16. +
  17. Choose the appropriate site from the “Sites” dropdown menu.

  18. +
  19. Click the “Add” button.

  20. +
+../_images/aws-vpc-cidr-to-netris.png +
+
+

Static route for AWS VPC Subnet in Netris Controller

+

We need to create route entry in Netris. The prefix for the route will be the AWS VPC CIDR block, and the next-hop will be the default gateway of Netris Softgate EC2.

+

Here are the steps to create the static route:

+
    +
  1. Securely log in to the Netris Softgate’s EC2 instance using SSH.

  2. +
  3. Retrieve the default gateway address by typing the command ip route show default.

  4. +
  5. In Netris Controller, go to the “Routes” section under the “Net” tab.

  6. +
  7. Click on the “+ Add” button located at the top-right corner of the screen.

  8. +
  9. Enter your VPC CIDR block in the “Prefix” field.

  10. +
  11. Enter the default gateway of the Netris Softgate EC2 instance in the “Next-Hop” field.

  12. +
  13. Select the appropriate site from the “Site” dropdown menu.

  14. +
  15. Finally, click on the “Add” button to create the static route.

  16. +
+../_images/aws-netris-static-route.png +
+
+

Create the Softgate in the Netris Controller

+

After completing all the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide:

+
    +
  1. Ensure that you have completed all the pre-required steps.

  2. +
  3. Navigate to the “Net” tab in the Netris Controller and select the “Inventory” section.

  4. +
  5. Click on the “+Add” button to create a new Softgate.

  6. +
  7. Provide a descriptive name for the Softgate in the “Name” field.

  8. +
  9. From the “Tenant” dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets.

  10. +
  11. From the “Type” dropdown menu, select “SoftGate”.

  12. +
  13. Choose the appropriate site from the “Site” dropdown menu.

  14. +
  15. For the “Main IP address” and “Management IP Address” fields, select “Assign automatically”.

  16. +
  17. In the “Description” field, add int=eth0 to specify that Netris should use the softgate’s eth0 interface instead of the default bond0 interface that Netris Softgate Agent looks for.

  18. +
  19. Finally, click on the “Add” button to create the Softgate.

  20. +
+../_images/aws-netris-create-sg.png +
+
+

Netris SoftGate node provisioning

+

After creating a softgate resource in Netris Controller, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting “Install Agent”. Copy the one-line installer command to your clipboard and connect to your EC2 instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it’s done, reboot the server.

+../_images/aws-netris-provision-sg.png +
+
+
+

Configure AWS VPC

+
+

Adding routes

+

To enable specific traffic to be routed to the Netris Softgate EC2 instance in your AWS VPC, you need to modify your VPC’s Route Table. To get started, go to the AWS Console and navigate to your VPC’s Route Table. From there, click the “Edit routes” button to access the routing table, and then click “Add route” to create a new routing entry.

+

In the “Destination” field, enter the subnet CIDR block for the Netris other Sites’ subnets you want to access from this VPC. Next, in the “Target” field, select the “Instance” option and then select the Netris Softgate EC2 instance you previously created. This will ensure that traffic for those subnets is directed to the Softgate instance.

+../_images/aws-vpc-routes-created.png +
+
+

EC2 Source / destination check

+

To allow Netris SoftGate to work properly, it is necessary to disable the ‘Source / destination check’ for the SoftGate’s EC2 Instance. To do so, follow the steps below:

+
    +
  1. Go to the AWS Console and navigate to the EC2 service page.

  2. +
  3. Select the SoftGate EC2 instance.

  4. +
  5. Click on the “Actions” button in the “Networking” section.

  6. +
  7. Select “Change Source / destination check”.

  8. +
  9. Mark the “Stop” checkbox to disable the feature.

  10. +
  11. Click “Save” to apply the changes.

  12. +
+../_images/aws-ec2-stop-fwd-check.png +

By disabling the “Source / destination check”, the SoftGate EC2 instance can receive and forward traffic between the Netris other Sites and AWS VPC subnets.

+
+
+
+

Enabling Site Mesh

+

To enable Site Mesh, in Netris Controller, navigate to the “Net” tab and select “Sites”. Next, click on the three vertical dots (⋮) on the right side of the AWS site and select “Edit”, and then from the “Site Mesh” dropdown menu, select “Hub”. Save the changes. Repeat these steps for all sites that need to have meshed.

+../_images/aws-netris-enable-site-mesh.png +

The Site Mesh status can be viewed by navigating to the “Site Mesh” section under the “Net” tab. This will display the current status of Site Mesh for all Sites.

+../_images/aws-netris-site-mesh-status.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/create-interconnection-to-fabric.html b/en/3.4/tutorials/create-interconnection-to-fabric.html new file mode 100644 index 0000000000..1e0f36bceb --- /dev/null +++ b/en/3.4/tutorials/create-interconnection-to-fabric.html @@ -0,0 +1,757 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Creating an Interconnection to Equinix Fabric — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Creating an Interconnection to Equinix Fabric
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Creating an Interconnection to Equinix Fabric

+

Once you have successfully set up Netris on Equinix Metal, you may want to use it to connect your private networks from Equinix Fabric back to Metal.

+

Netris can easily manage the routing and other required VPC services for this.

+

Create VLANs in Metal

+

To be able to create an interconnect into Equinix Fabric, first you must create a pair of VLANs for the Primary and Secondary connections.

+

To do this, go to Equinix Console -> IP&Networks -> Layer 2. +Click “Add Vlan”. +Choose the location of your Netris Softgates, then use a couple of VLAN IDs that are unused.

+

In the example below, VLAN 1000 is used for Primary, and VLAN 1001 is used for the Secondary.

+../_images/add-new-vlan-equinix.png +

Request Interconnection

+

Now we are ready to request the Equinix Interconnection on the Equinix Metal side.

+

Go to Equinix Console -> IP&Networks -> Interconnections.

+

Click “Request New Interconnection”.

+

From here, under Fabric VC select “Fabric Billed”.

+

Choose the Location where your Netris Softgates are located.

+

Add an Interconnection name. This can be any name.

+

Choose “Redundant interconnection”.

+

Under “Primary Metal VLAN” select the VLAN created in the previous step, in this example it is VLAN 1000.

+

Under “Secondary Metal VLAN” select the VLAN created in the previous step, in this example it is VLAN 1001.

+

Then, finalize the request by clicking “Submit Interconnection Request”.

+../_images/add-interconnection-request.png +

Once this is accomplished, use the token generated to connect your virtual Network Edge device to Equinix Metal. Setting up a connection on the Fabric side is outside the scope of this document. Once that is complete, the Interconnection will show as Active, and you may move onto creating BGP connections with the Network Edge Virtual Device.

+

Primary BGP Connection to Fabric Network Edge Device

+

Up to this point, a redundant pair of layer 2 connections have been created and connected back to an Equinix Fabric Network Edge device (or some other port with a router on the other end). To be able to utilize the redundancy of both connections, a BGP peering must be established over the two layer 2 VLANs.

+

On the Netris Controller, go to Net -> E-BGP, then click Add.

+

Fill in the following information:

+
Name: Choose any name
+Site: Choose the site of your Equinix Project
+V-Net: None
+BGP Router: Choose the primary Softgate
+VLAN ID: Choose the Primary VLAN ID created when creating the Interconnection in the previous steps
+Neighbor AS: Choose the ASN configured on the Network Edge router
+IP Family: IPv4
+Local IP: Any /30 can be used.  Using a Private IP is recommended.
+Remote IP: Use the IP of the remote side.
+
+
+../_images/add-equinix-bgp-primary.png +

Secondary BGP Connection to Fabric Network Edge Device

+

Now to create the backup/secondary connection.

+

On the Netris Controller, go to Net -> E-BGP, then click Add.

+

Fill in the following information:

+
Name: Choose any name
+Site: Choose the site of your Equinix Project
+V-Net: None
+BGP Router: Choose the secondary Softgate
+VLAN ID: Choose the Secondary VLAN ID created when creating the Interconnection in the previous steps
+Neighbor AS: Choose the ASN configured on the Network Edge router
+IP Family: IPv4
+Local IP: Any /30 can be used.  Using a Private IP is recommended.
+Remote IP: Use the IP of the remote side.
+
+
+

Once completed, the EBGP status shoudl look something like this:

+../_images/equinix-ebgp-links-up.png +
+

Note

+

At this point, you should have both BGP sessions up, but the link may show yellow if the Network Edge device is not advertising any routes back to the Netris Softgates. Once the Network Edge router begins sending routes from other connections, the status should turn green.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/enable-services-on-equinix-metal-project.html b/en/3.4/tutorials/enable-services-on-equinix-metal-project.html new file mode 100644 index 0000000000..9d8c1016a9 --- /dev/null +++ b/en/3.4/tutorials/enable-services-on-equinix-metal-project.html @@ -0,0 +1,748 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling services (NAT, V-Net, Load Balancer, IP pools) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Enabling services (NAT, V-Net, Load Balancer, IP pools)

+

Although bare metal servers in Equinix Metal Project get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities.

+

Both NAT and on-demand Load Balancer services need public IP addresses.

+
+

1) Requesting new Public IP address block

+

Go to Equinix Metal web console and click on IPs & Networks → IPs (see the screenshot below)

+

In this example, I’m requesting two IP address blocks, one /30 (4 IPs) for NAT and one /28 (16 IPs) for Load Balancer.

+

It’s important to tag IP blocks as “netris”. This is a signal for Netris Controller that this block is intended for Netris VPC.

+

You can always request more IP address blocks in the future. Also it is possible to request a large block and then use Netris IPAM for crushing it into smaller blocks. You can read more about Netris IPAM in Netris docs.

+../_images/equinix-metal-request-ip-block.png +

Once IP address blocks are provisioned on Equinix Metal Project you should be able to find them automatically replicated in Netris web console under Net → IPAM

+../_images/equinix-metal-netris-ipam-synced.png +

You don’t need to worry about advertising them over BGP, Netris will handle that automatically when that makes sense (associated with any service).

+
+
+

2) Enable on-demand (elastic) Load Balancer

+

To Enable on-demand (elastic) Load Balancer you only need to change the “purpose” field of appropriate IP address block from “common” into “load-balancer”

+

Click on the 3 dots menu (in this example of /28 IP address block), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field.

+../_images/netris-enable-elb.png +

Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform.

+

Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it.

+
+
+

3) Enable V-Net

+

V-Net is a service for virtual private networks. You need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources.

+../_images/netris-create-common-subnets.png +
+
+

4) Enable NAT

+

To enable NAT, you need to repurpose a block of IP addresses for NAT. In the below example I’m repurposing the newly requested /30 subnet for NAT.

+../_images/netris-ipam-nat.png +

Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. In this example I’m enabling SNAT for the entire 10.0.0.0/8 private network, so basically I just want to ensure that VMs that will get IPs from private networks will get outbound Internet access through NAT. You can always have more granular control either through NAT rule or using Services → ACLs.

+../_images/netris-create-nat-rule.png +

At this point the minimal configuration of Netris VPC networking is done, next chapters will describe how to consume the VPC, how to request resources and services.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/equinix-metal-api-integration-enablement.html b/en/3.4/tutorials/equinix-metal-api-integration-enablement.html new file mode 100644 index 0000000000..b4ad17c158 --- /dev/null +++ b/en/3.4/tutorials/equinix-metal-api-integration-enablement.html @@ -0,0 +1,752 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Enable Equinix Metal API integration — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Enable Equinix Metal API integration

+

For each Equinix Metal Project+location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “Equinix Metal” from the dropdown menu.

Name

Type a descriptive name for your Equinix Metal Project+location.

Equinix Project ID

Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID.

Equinix Project API key

Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here.

Equinix Location

Select your equinix location from the dropdown menu.

+

Equinix Metal Project ID

+../_images/equinix-metal-project-id.png +

Equinix Metal Project API key

+../_images/equinix-metal-project-api-keys.png +

Netris Create New Site

+../_images/netris-create-equinix-metal-site.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/gcp-concept.html b/en/3.4/tutorials/gcp-concept.html new file mode 100644 index 0000000000..68349a9745 --- /dev/null +++ b/en/3.4/tutorials/gcp-concept.html @@ -0,0 +1,724 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Site Mesh with GCP Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Site Mesh with GCP Overview

+
+

Introduction

+

This guide provides a step-by-step process to set up and configure Netris Softgate in GCP for establishing a site mesh network between the user’s on-premises, GCP, and other cloud environments.

+
+
+

Concept

+

Netris Softgate in GCP is an VM instance that runs the Netris software. Therefore, you’ll first need to create an VM instance for Netris Softgate and install the Netris software on it. Once that’s done, you’ll need to configure the routes in your GCP VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your GCP VPC to access those destinations through the Netris Softgate VM instance.

+../_images/gcp-concept-traffic-flows.png +

Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/gcp-deploy-softgate.html b/en/3.4/tutorials/gcp-deploy-softgate.html new file mode 100644 index 0000000000..a4558f8927 --- /dev/null +++ b/en/3.4/tutorials/gcp-deploy-softgate.html @@ -0,0 +1,862 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Deploy a Softgate in GCP — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Deploy a Softgate in GCP

+

As stated in the previous section, the following sequence of actions must be taken in order to proceed: create a VM instance, add Softgate into the Netris Controller, install Netris Softgate software on the VM instance, and configure routes in GCP VPC. Let’s commence with these steps in the specified order.

+
+

Create a VM instance

+

Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated VM instance. To achieve this, create a VPC firewall rule with the following parameters:

+
    +
  • Name: Any name

  • +
  • Direction of traffic: “Ingress”

  • +
  • Action on match: “Allow”

  • +
  • Targets: “Specified target tags”

  • +
  • Target tags: allow-all

  • +
  • Source filter: “IPv4 ranges”

  • +
  • Source IPv4 ranges: 0.0.0.0/0

  • +
  • Protocols and ports: “Allow all”

  • +
+

Afterward, a VM instance can be created using the network tag above.

+../_images/gcp-firewall-rule.png +

To enable connectivity with other Netris sites, creating the VM instance in the desired VPC is essential. Therefore, deploy a new VM instance with the Ubuntu 22.04 operating system installed, utilizing an instance type that meets the minimum hardware requirements of 2 virtual CPUs, 4 GB of RAM, and 30 GB of drive space, such as e2-medium or any other type that satisfies these specifications. From the Advanced Networking options enable IP forwarding and specify the previously created allow-all network tag.

+../_images/gcp-softgate-deployed.png +

After successfully deploying the VM instance, it is crucial to take note of its External IP. This address will be required in the upcoming step.

+
+
+

Configure Netris Controller

+

Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed.

+
+

Pre-Requisite Steps

+

In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below:

+
    +
  1. Open the Netris Web Console.

  2. +
  3. Navigate to “Net” and select “Sites”.

  4. +
  5. Click on the “+ Add” button.

  6. +
  7. Select “Dot1q Trunk” as the “Switch Fabric”.

  8. +
  9. Input a descriptive name for the site.

  10. +
  11. Specify 65500 in the “Public ASN” field.

  12. +
  13. Click “Add” to create the new site.

  14. +
+../_images/gcp-netris-site-create.png +

Subsequently, it is necessary to create a private subnet for Softgate management. To achieve this, follow the steps below:

+
    +
  1. Go to the “IPAM” section under the “Net” tab.

  2. +
  3. Click on the “+ Add” button located at the top-right corner.

  4. +
  5. Enter a unique “Prefix” for the new subnet, such as 10.254.254.0/24.

  6. +
  7. Type a descriptive “Name” for the subnet.

  8. +
  9. Select the desired tenant name from the “Tenant” dropdown menu.

  10. +
  11. From the “Type” dropdown menu, select “Subnet”.

  12. +
  13. Select “management” from the “Purpose” dropdown menu.

  14. +
  15. Choose the appropriate site from the “Sites” dropdown menu.

  16. +
  17. Click on the “Add” button to create the subnet.

  18. +
+../_images/gcp-netris-ipam-mgmt.png +

Following the creation of the private subnet for Softgate management, it is necessary to create another subnet with the “Purpose” of loopback. The “Prefix” for this subnet will be the External IP of the GCP VM instance with a netmask length of /32. For instance, if the VM instance’s External IP address is 34.85.167.128, then the “Prefix” for the loopback subnet will be 34.85.167.128/32. However, unlike the subnet for Softgate management, an allocation for that “Prefix” must first be created before creating the subnet for loopback.

+../_images/gcp-netris-ipam-lo.png +
+
+

Create the Softgate in the Netris Controller

+

After completing the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide:

+
    +
  1. Ensure that you have completed all the pre-required steps.

  2. +
  3. Navigate to the “Net” tab in the Netris Controller and select the “Inventory” section.

  4. +
  5. Click on the “+Add” button to create a new Softgate.

  6. +
  7. Provide a descriptive name for the Softgate in the “Name” field.

  8. +
  9. From the “Tenant” dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets.

  10. +
  11. From the “Type” dropdown menu, select “SoftGate”.

  12. +
  13. Choose the appropriate site from the “Site” dropdown menu.

  14. +
  15. For the “Main IP address” and “Management IP Address” fields, select “Assign automatically”.

  16. +
  17. In the “Description” field, add int=ens4 to specify that Netris should use the softgate’s ens4 interface instead of the default bond0 interface that Netris Softgate Agent looks for.

  18. +
  19. Finally, click on the “Add” button to create the Softgate.

  20. +
+../_images/gcp-netris-create-sg.png +
+
+

Add GCP VPC Subnet(s) into the Netris Controller

+

The next step involves registering into Netris IPAM the subnet of the region where the SoftGate node has been created. Follow the steps below to accomplish this:

+
    +
  1. Access the GCP Console and navigate to your VPC Subnets.

  2. +
  3. Take note of the Internal IP range associated with the relevant region.

  4. +
  5. In Netris Controller, go to the “IPAM” section under the “Net” tab.

  6. +
  7. Click the “+ Add” button located at the top-right corner.

  8. +
  9. Enter the region’s Internal IP range into the “Prefix” field for the new subnet. For example, if your Internal IP range is “10.150.0.0/20”, enter that value.

  10. +
  11. Type a descriptive name for the subnet.

  12. +
  13. From the “Tenant” dropdown menu, select the desired tenant name.

  14. +
  15. From the “Type” dropdown menu, select “Subnet”.

  16. +
  17. Select “inactive” from the “Purpose” dropdown menu.

  18. +
  19. Choose the appropriate site from the “Sites” dropdown menu.

  20. +
  21. Click the “Add” button.

  22. +
+

Once the subnet of the region where the SoftGate node was created has been successfully registered in Netris IPAM, you can proceed to create subnets for other necessary regions in a similar manner.

+../_images/gcp-vpc-subnet-to-netris.png +
+
+

Static route for GCP VPC Subnet(s) in Netris Controller

+

We need to create route entries in Netris. The prefix for the route will be the GCP VPC Subnet(s), and the next-hop will be the default gateway of Netris Softgate VM Instance.

+

Here are the steps to create the static route:

+
    +
  1. Securely log in to the Netris Softgate’s VM instance using SSH.

  2. +
  3. Retrieve the default gateway address by typing the command ip route show default.

  4. +
  5. In Netris Controller, go to the “Routes” section under the “Net” tab.

  6. +
  7. Click on the “+ Add” button located at the top-right corner of the screen.

  8. +
  9. Enter the subnet of the GCP region in the “Prefix” field.

  10. +
  11. Enter the default gateway of the Netris Softgate VM instance in the “Next-Hop” field.

  12. +
  13. Choose the appropriate site from the “Site” dropdown menu.

  14. +
  15. From the “Apply to” dropdown menu, select the SoftGate.

  16. +
  17. Finally, click on the “Add” button to create the static route.

  18. +
+

Repeat these steps for each GCP VPC Subnet that has been registered in Netris IPAM.

+

Note: Regardless of the GCP region’s subnet, the “Next-Hop” field should always contain the default gateway of the Netris SoftGate VM instance.

+../_images/gcp-netris-static-route.png +
+
+

Netris SoftGate node provisioning

+

After creating a softgate resource in Netris Controller and defining all necessary routes, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting “Install Agent”. Copy the one-line installer command to your clipboard and connect to your VM instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it’s done, reboot the server.

+../_images/gcp-netris-provision-sg.png +
+
+
+

Configure routing in GCP VPC

+

To ensure that specific traffic is directed to the Netris Softgate VM instance within your GCP VPC, it is necessary to configure new route entries in VPC routes. Follow the step-by-step guide below to add GCP VPC routes:

+
    +
  1. Access the Google Cloud Platform (GCP) Console.

  2. +
  3. Navigate to the VPC Network page.

  4. +
  5. Click on the “Routes” then switch to the “ROUTE MANAGEMENT” tab to view the existing routes.

  6. +
  7. Click the “+ Create Route” button to create a new route.

  8. +
  9. Provide the following details for the new route:

  10. +
+
+
    +
  • Name: Assign a descriptive name to the route.

  • +
  • Network: Select the appropriate network for the route.

  • +
  • Destination IP Range: Specify the subnets of other Netris sites that you wish to access from this VPC.

  • +
  • Next hop: From the dropdown menu, select the “Specify an instance”

  • +
  • Next hop instance: From the dropdown menu, select the SoftGate VM Instance

  • +
+
+
    +
  1. Review the route configuration and ensure all the details are accurate.

  2. +
  3. Click the “Create” button to add the route to the GCP VPC network.

  4. +
  5. Verify that the new route appears in the list of routes for the selected VPC network.

  6. +
+

Repeat these steps for any additional routes you need to add.

+../_images/gcp-vpc-routes-created.png +
+
+

Enabling Site Mesh

+

To enable Site Mesh, in Netris Controller, navigate to the “Net” tab and select “Sites”. Next, click on the three vertical dots (⋮) on the right side of the GCP site and select “Edit”, and then from the “Site Mesh” dropdown menu, select “Hub”. Save the changes. Repeat these steps for all sites that need to have meshed.

+../_images/gcp-netris-enable-site-mesh.png +

The Site Mesh status can be viewed by navigating to the “Site Mesh” section under the “Net” tab. This will display the current status of Site Mesh for all Sites.

+../_images/gcp-netris-site-mesh-status.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/getting-started-for-equinix-metal.html b/en/3.4/tutorials/getting-started-for-equinix-metal.html new file mode 100644 index 0000000000..16d8a26a16 --- /dev/null +++ b/en/3.4/tutorials/getting-started-for-equinix-metal.html @@ -0,0 +1,762 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Equinix Metal API integration enablement — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Equinix Metal API integration enablement
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Equinix Metal API integration enablement

+

For each Equinix Metal Project+location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “Equinix Metal” from the dropdown menu.

Name

Type a descriptive name for your Equinix Metal Project+location.

Equinix Project ID

Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID.

Equinix Project API key

Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here.

Equinix Location

Select your equinix location from the dropdown menu.

+

Equinix Metal Project ID

+../_images/equinix-metal-project-id.png +

Equinix Metal Project API key

+../_images/equinix-metal-project-api-keys.png +

Netris Create New Site

+../_images/netris-create-equinix-metal-site.png +
+

Adding Netris SoftGate nodes

+

For SoftGate nodes you can start with two servers of the smallest flavor. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one.

+

Request two servers from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned.

+
    +
  1. At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory

  2. +
+../_images/softgate-nodes-created-in-equinix.png +
    +
  1. When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”.

    +
    +

    Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location).

    +
    +
  2. +
+

Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+../_images/softgate-nodes-recognized-in-netris.png +
    +
  1. Provision SoftGate nodes.

  2. +
+

Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there.

+../_images/softgate-one-liner-provisioning.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/softgate-green.png +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/index-equinix.html b/en/3.4/tutorials/index-equinix.html new file mode 100644 index 0000000000..43baedaad0 --- /dev/null +++ b/en/3.4/tutorials/index-equinix.html @@ -0,0 +1,727 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris VPC for Equinix Metal Getting Started Guide — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris VPC for Equinix Metal Getting Started Guide
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+ +
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/index-vpc.html b/en/3.4/tutorials/index-vpc.html new file mode 100644 index 0000000000..391608acbf --- /dev/null +++ b/en/3.4/tutorials/index-vpc.html @@ -0,0 +1,712 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + VPC for Anywhere Getting Started Guide — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • VPC for Anywhere Getting Started Guide
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

VPC for Anywhere Getting Started Guide

+

You are welcome to join our Slack channel to get additional support from our engineers and community.

+ +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/index.html b/en/3.4/tutorials/index.html new file mode 100644 index 0000000000..6a958f364b --- /dev/null +++ b/en/3.4/tutorials/index.html @@ -0,0 +1,726 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Generic Tutorials — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris Generic Tutorials
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Netris Generic Tutorials

+

You are welcome to join our Slack channel to get additional support from our engineers and community.

+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/install-netris-controller-in-equinix-metal.html b/en/3.4/tutorials/install-netris-controller-in-equinix-metal.html new file mode 100644 index 0000000000..36d9f95e6e --- /dev/null +++ b/en/3.4/tutorials/install-netris-controller-in-equinix-metal.html @@ -0,0 +1,757 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller on Equinix Metal on-demand server — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Installing a Netris Controller on Equinix Metal on-demand server

+
+

Installation Steps

+

You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions).

+

Request a server

+

The smallest flavor, c3.small.x86, is enough for most users.

+../_images/equinix-request-c3-small-server.png +

if you decide to use a VM, please see below the minimal VM requirements.

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

DNS record

+

In my example my host got a public IP address 139.178.89.255. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is example using Cloudflare DNS service. (same idea with any DNS software or service)

+../_images/dns-cloudflare-equinix-ip.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 139.178.89.255
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/installing-netris-controller.html b/en/3.4/tutorials/installing-netris-controller.html new file mode 100644 index 0000000000..93c57f356e --- /dev/null +++ b/en/3.4/tutorials/installing-netris-controller.html @@ -0,0 +1,784 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Installing a Netris Controller

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where you host the Netris controller. What matters is that 1) the Netris controller needs to be accessible over the Internet. 2) You can access the web console. 3) Nodes that are going to be managed by Netris have access to the Netris controller through their management network interface.

+

Linux host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

DNS record

+

In the example below, the host has a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, it is recommended to use a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is an example using Cloudflare DNS service (may use any DNS service).

+../_images/dns-record-netrisctl.png +

Ensure that newly created domain name resolves to the right IP address of the machine that will host the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 54.183.23.201
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

The Netris Controller installer will create a K3s cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once the installation process is finished, you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +

Restrict incoming TCP requests to the list below:

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + +

TCP Port

Service

22

SSH

80

HTTP

443

Netris Web Console

2003

Streaming Telemetry (Collectd)

3033

Netris Monitoring (Telescope)

50051

Netris Agent (gRPC)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/netris-vpc-for-aws.html b/en/3.4/tutorials/netris-vpc-for-aws.html new file mode 100644 index 0000000000..b6826c401b --- /dev/null +++ b/en/3.4/tutorials/netris-vpc-for-aws.html @@ -0,0 +1,733 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling Site Mesh with AWS — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Enabling Site Mesh with AWS
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/netris-vpc-for-equinix-metal.html b/en/3.4/tutorials/netris-vpc-for-equinix-metal.html new file mode 100644 index 0000000000..6d5635ae04 --- /dev/null +++ b/en/3.4/tutorials/netris-vpc-for-equinix-metal.html @@ -0,0 +1,739 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris VPC for Equinix Metal — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/netris-vpc-for-gcp.html b/en/3.4/tutorials/netris-vpc-for-gcp.html new file mode 100644 index 0000000000..48314674bc --- /dev/null +++ b/en/3.4/tutorials/netris-vpc-for-gcp.html @@ -0,0 +1,733 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling Site Mesh with GCP — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Enabling Site Mesh with GCP
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/netris-vpc-for-phoenixnap-bmc.html b/en/3.4/tutorials/netris-vpc-for-phoenixnap-bmc.html new file mode 100644 index 0000000000..57c182a2f9 --- /dev/null +++ b/en/3.4/tutorials/netris-vpc-for-phoenixnap-bmc.html @@ -0,0 +1,750 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris VPC for phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris VPC for phoenixNAP BMC
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.html b/en/3.4/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.html new file mode 100644 index 0000000000..168c9d7df2 --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.html @@ -0,0 +1,714 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Provisioning Netris SoftGate nodes in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provisioning Netris SoftGate nodes in phoenixNAP BMC
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provisioning Netris SoftGate nodes in phoenixNAP BMC

+

For SoftGate nodes you can start with two s2.c1.small or larger servers. In the future, if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration), you can upgrade the servers one-by-one.

+

1) Request two servers (s2.c1.small) from phoenixNAP BMC with Ubuntu Jammy OS, type netris-softgate in the “Server Description” field, +choose a /31 Public IP Allocation and wait until provisioned.

+
+

Note

+

It’s required to type the netris-softgate in the description. This signals Netris Controller that those are not regular bare-metal servers, and they should be synced with the type of SoftGate.

+
+../_images/phoenixnap-softgate-nodes-creation.png +

2) At this point you should see Netris Controller listing newly created servers as “Softgate Softgate1(2)” under Netris Web Console → Net → Inventory. +You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+

3) Provision SoftGate nodes. Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server with the ubuntu user and paste the one-liner there.

+../_images/phoenixnap-softgate-nodes-created.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/phoenixnap-softgate-nodes-green.png +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-api-integration-enablement.html b/en/3.4/tutorials/phoenixnap-bmc-api-integration-enablement.html new file mode 100644 index 0000000000..3c8f55ad29 --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-api-integration-enablement.html @@ -0,0 +1,733 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Enable phoenixNAP BMC API integration — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Enable phoenixNAP BMC API integration
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Enable phoenixNAP BMC API integration

+

For each phoenixNAP BMC location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “PhoenixNAP BMC” from the dropdown menu.

Name

Type a descriptive name for your phoenixNAP BMC location.

PhoenixNAP Client ID

Create a new API Credential with “bmc” scope in phoenixNAP BMC portal under API Credentials → + Create Credentials. Then copy/paste Client ID.

PhoenixNAP Client Secret

Copy/Paste the Client Secret from the already created API Credential.

PhoenixNAP Location

Select your phoenixNAP BMC location from the dropdown menu.

+

phoenixNAP BMC API Credential

+../_images/phoenixnap-api-credential.png +

Netris Create New Site

+../_images/phoenixnap-site-create.png +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-concept.html b/en/3.4/tutorials/phoenixnap-bmc-concept.html new file mode 100644 index 0000000000..f1159b0fa8 --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-concept.html @@ -0,0 +1,738 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris VPC for phoenixNAP BMC Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris VPC for phoenixNAP BMC Overview

+
+

Introduction

+

Netris VPC for phoenixNAP BMC is a solution that enables VPC network functionality in phoenixNAP’s Bare Metal Cloud. Netris VPC can be used for all or part of the servers/network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures.

+
+
+

Concept

+

There is an agent inside the Netris VPC controller for phoenixNAP BMC that communicates to phoenixNAP API once the integration has been enabled. That agent is responsible for metadata synchronization between phoenixNAP BMC and Netris VPC controller. Also, it configures the BMC network services based on the services defined in the Netris VPC controller.

+

SoftGate is a highly optimized automatic Linux gateway, which in turn communicates with the Netris VPC controller via an encrypted protocol. A SoftGate node (typically two of them for redundancy) is a regular BMC server that should be deployed on desired phoenixNAP location. Once the server has been deployed, it starts consuming the BMC public and private networks with various VLANS.

+../_images/phoenixnap-concept-solution-traffic-flows.png +

Due to phoenixNAP BMC creating a new VLAN for each server’s Public IP Allocation and Public Network/Private Network, we decided to use the upper VLANS Range - 3000-4094 (if you need to change the default range, that can be done in the Netris Site settings). Netris will consume VLANS only from that range. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the BMC network.

+

SoftGate node becomes the default gateway for the workloads consuming the VPC network. PhoenixNAP BMC Private Network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers. Netris also creates a Public Network in the phoenixNAP BMC when the IP Addresses from Public IP Allocations (passed to Netris VPC) have been used in any Netris VPC’s services. Besides Public/Private Networks creation Netris also attaches all necessary servers to that networks. Therefore, you don’t need to worry about assigning servers to the networks on the phoenixNAP side, Netris will handle that automatically when that makes sense.

+../_images/phoenixnap-concept-public-network.png +

What’s next

+ +

Usage

+ +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-install-netris-controller.html b/en/3.4/tutorials/phoenixnap-bmc-install-netris-controller.html new file mode 100644 index 0000000000..7f9b371d4e --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-install-netris-controller.html @@ -0,0 +1,740 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller on phoenixNAP BMC server — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Installing a Netris Controller on phoenixNAP BMC server
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Installing a Netris Controller on phoenixNAP BMC server

+
+

Installation Steps

+

You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions).

+

Minimum Hardware Requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

Deploy New Server

+

Deploy a new server in phoenixNAP BMC: select the Location, choose the server flavor (the smallest flavor, s0.d1.small, is enough for most users), type some name for Controller Server, request a new /31 public allocation for this instance.

+../_images/phoenixnap-request-ctl-server.png +

DNS record

+

In my example my host got a public IP address 131.153.154.61. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is example using Cloudflare DNS service. (same idea with any DNS software or service)

+../_images/phoenixnap-dns-cloudflare.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 131.153.154.61
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netrisctl.netris.dev --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-ipam-setup.html b/en/3.4/tutorials/phoenixnap-bmc-ipam-setup.html new file mode 100644 index 0000000000..a467a67a88 --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-ipam-setup.html @@ -0,0 +1,772 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + IPAM Setup for Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

IPAM Setup for Services

+

Although bare metal servers in phoenixNAP BMC may get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization, something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities.

+

Both NAT and on-demand Load Balancer services need public IP addresses.

+

Go to phoenixNAP BMC web console and click on Networking → Public IP Allocations (see the screenshot below)

+
+

Option 1 - Separated allocation for each purpose

+

In this example, I’m requesting two /29 (5 assignable IPs) IP Allocations, one for NAT and one for Load Balancer.

+

It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC.

+

You can always request more IP address allocations in the future.

+../_images/phoenixnap-request-ip-allocation.png +

Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM

+../_images/phoenixnap-netris-ipam-synced.png +

You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service).

+
+

1. Enable on-demand (elastic) Load Balancer

+

To Enable an on-demand (elastic) Load Balancer, you only need to change the “purpose” field of appropriate IP subnet from “inactive” into “load-balancer”

+

Click on the 3 dots menu (in this example of first /29 subnet), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field.

+../_images/phoenixnap-netris-ipam-lb-purpose.png +

Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform.

+

Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it.

+
+
+

2. Enable NAT

+

To enable NAT, you need to repurpose a subnet for NAT. In the below example, I’m repurposing the second of the newly requested /29 subnets for NAT.

+../_images/phoenixnap-netris-ipam-nat-purpose.png +

Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT.

+
+
+
+

Option 2 - Splitting single allocation into different subnets

+

In this example, I’m requesting a single /28 (13 assignable IPs) Public IP Allocation, then splitting it into two /29 subnets, one for NAT and one for Load Balancer.

+

It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC.

+

You can always request more IP address allocations in the future.

+../_images/phoenixnap-request-ip-allocation-slash-28.png +

Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM

+../_images/phoenixnap-netris-ipam-synced-slash-28.png +

You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service).

+
+

1. Enable on-demand (elastic) Load Balancer

+

In this scenario, to Enable an on-demand (elastic) Load Balancer, you need to create a smaller subnet with the purpose of “load-balancer”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png +

Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform.

+

Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it.

+
+
+

2. Enable NAT

+

To enable NAT, you need to create a smaller subnet with the purpose of “nat”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png +

Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT.

+
+
+
+

Note*

+

It doesn’t matter with what Option (1,2) you will go. Once the Public IP Allocation has been replicated in the Netris IPAM, Netris will automatically reserve the network, first usable, and broadcast IP addresses because they are unusable in this (phoenixNAP BMC) scenario.

+../_images/phoenixnap-reserved-ips.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-link-to-installation.html b/en/3.4/tutorials/phoenixnap-bmc-link-to-installation.html new file mode 100644 index 0000000000..82676bf64b --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-link-to-installation.html @@ -0,0 +1,717 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing Netris on phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Installing Netris on phoenixNAP BMC

+

To deploy Netris on the phoenixNAP Bare Metal Cloud, please follow the official guide provided by phoenixNAP. This guide will walk you through the step-by-step installation process.

+

Netris for Bare Metal Cloud

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-using-l4lb.html b/en/3.4/tutorials/phoenixnap-bmc-using-l4lb.html new file mode 100644 index 0000000000..bdf69c9524 --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-using-l4lb.html @@ -0,0 +1,719 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC

+

Services –> L4 Load Balancer is an on-demand (elastic) load balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console.

+../_images/phoenixnap-l4lb.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-using-nat.html b/en/3.4/tutorials/phoenixnap-bmc-using-nat.html new file mode 100644 index 0000000000..cd7fc79558 --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-using-nat.html @@ -0,0 +1,733 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using NAT services in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using NAT services in phoenixNAP BMC

+

NAT, under Net –> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 10.0.0.0/8 to have access to services outside their VPC. The goal is to provide bare-metal servers/VMs with the capability to connect to the Internet through NAT for outbound access.

+
+

Option 1 - MASQUERADE

+

The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn’t require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT).

+../_images/phoenixnap-nat-masquerade.png +
+
+

Option 2 - SNAT

+

Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a NAT purpose. SNAT replaces the source IP address of the instances with the IP address of the “SNAT to IP”. Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity.

+../_images/phoenixnap-nat-snat.png +

You can always have more granular control either through NAT rule or using Services → ACLs.

+
+
+

DNAT

+

A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a “DNAT to IP/Port”. This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database.

+

I’m creating a DNAT rule for the ssh port in the example below. It forwards the public IP’s 55022 port to the local IP’s 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server.

+../_images/phoenixnap-nat-dnat.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/phoenixnap-bmc-using-vnet.html b/en/3.4/tutorials/phoenixnap-bmc-using-vnet.html new file mode 100644 index 0000000000..af4b64bc3c --- /dev/null +++ b/en/3.4/tutorials/phoenixnap-bmc-using-vnet.html @@ -0,0 +1,761 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using V-Net (isolated virtual network) services in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using V-Net (isolated virtual network) services in phoenixNAP BMC

+

V-Net, under Services –> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from phoenixNAP BMC into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network.

+

Netris V-Net (in phoenixNAP BMC scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Private Network using phoenixNAP BMC API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Private Network. SoftGate nodes are the default gateway for the V-Net services.

+
+

Adding Subnets for V-Net

+

Before starting to use the V-NETs, you need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources.

+../_images/phoenixnap-netris-create-common-subnets.png +
+
+

Creating a V-Net

+

You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, phoenixNAP BMC, and your Compute.

+../_images/phoenixnap-netris-creating-vnet.png +

In this example, the new V-NET has VLAN ID 3000, subnet 10.128.1.0/24, and gateway 10.128.1.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 3000, and they should use IP addresses from 10.128.1.2-254 pointing to 10.128.1.1 as the default gateway or use DHCP. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 3000 will have Internet access over the NAT.

+../_images/phoenixnap-netris-vnet-ready.png +
+

Deploy a new server into an existing V-Net

+

Netris creates a private network in phoenixNAP BMC based on declared V-Nets. Besides creation, Netris continuously monitors that private networks. As a result of this continuous monitoring, you can’t edit private networks created by Netris from the phoenixNAP BMC console. However, if any modifications are made, Netris will automatically roll everything back to its state.

+

The only exception is a case with newly deployed servers. When you request a server from phoenixNAP BMC without any public IP address, use some Netris V-Net as a private network and mark the DHCP checkbox “Use your own privately managed DHCP server (Obtain IP address automatically)” - Netris will automatically add that server into the V-Net.

+../_images/phoenixnap-vnet-import-a-new-server.png +

Thanks to that functionality, you can request a bare-metal server directly into Netris V-Net. As a result, you will have bare-metal servers in a private subnet that can connect to services outside your VPC (since we have enabled NAT globally in previous chapters), but external services cannot initiate a connection with those servers. Once the servers have been provisioned, they will get a private IP from Netris DHCP, and you can find those IPs by pressing the “IP/MAC Info” button on the V-NET.

+../_images/phoenixnap-vnet-imported-new-server.png +

In order to connect via SSH to the newly deployed server, you can either create a DNAT rule and connect via Public IP, or if you don’t need permanent ssh access to that server, you can simply connect using Softgate as a JumpHost.

+
ssh -o ProxyCommand="ssh -W %h:%p ubuntu@<SoftGate Main IP>" ubuntu@<Server Private IP>
+
+
+../_images/phoenixnap-vnet-ssh-to-server.png +
+
+

Tags

+

Tags are used to associate bare-metal servers with V-NET dynamically. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers from the Private Network based on that tag. Thus, you can make flexible V-NETs, and there is no need to include every new server in the V-NET.

+../_images/phoenixnap-vnet-with-tag.png +

This feature is even more efficient when you build your infrastructure via Terraform. For example, let’s say you’ve created a V-NET with a tag using Netris Terraform Provider, then order several servers with the same tag using phoenixNAP Terraform Provider. And that’s it, when the servers are ready, Netris will detect them and make them part of the V-NET.

+../_images/phoenixnap-vnet-with-tag-terraform.png +
+
+

Unmanaged

+

Another option is turning the existing private network into Netris V-Net. All private networks from the allowed VLAN IDs range and in the proper location that Netris has not created are visible as “unmanaged” in the V-Net section.

+../_images/phoenixnap-vnet-unmanaged-vnet.png +

The “manage” button will open a dialogue window where it’s also possible to add a default gateway for the appropriate VLAN.

+
+

Warning

+

Once the private network is being converted into V-Net, it will be managed by Netris and no longer manageable through phoenixNAP BMC console.

+
+../_images/phoenixnap-vnet-managed-vnet.png +

Note that you can use Services –> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites)

+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/upgrading-netris.html b/en/3.4/tutorials/upgrading-netris.html new file mode 100644 index 0000000000..13267caaaa --- /dev/null +++ b/en/3.4/tutorials/upgrading-netris.html @@ -0,0 +1,858 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Netris Upgrade and Rollback Procedures — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ + +
+

Netris Upgrade and Rollback Procedures

+
+

Upgrade Procedure

+

Due to potential database structural changes between Netris versions, it’s highly recommended to take a backup of the database before upgrading. The backup will be used in the unlikely event of the need to perform a rollback.

+
    +
  1. To create a database backup, run the following command by first SSHing the Controller:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot.sql
+
+
+

Ensure that SQL file db-snapshot.sql is generated and present in the current directory.

+
+

Note

+

An SQL dump is enough for this scenario, but that’s not Netris-Controller’s entire backup procedure. Get familiar with the Backup/Restore documentation here.

+
+
    +
  1. Stop all Netris agents on devices managed by the controller (switch & SoftGate).

  2. +
+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl stop netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl stop netris-sg
+
+
+

Ensure that all devices in the Net → Inventory section are “red” with the “check_agent” status being “Agent is unavailable”.

+
+

Note

+

A stopped Netris agent has no impact on production traffic through the device.

+
+
    +
  1. Before upgrading the Netris Controller, take a note of the “Netris Version” by navigating to Setting → General in the Controller web interface. This version number will be used in case of the need to perform a rollback procedure.

  2. +
+Netris Version Example +
    +
  1. Start the upgrade of the Netris Controller using the one-liner after SSHing to the Controller.

  2. +
+
curl -sfL https://get.netris.io | sh -
+
+
+
+

Note

+

This process can take up to 5 minutes

+
+

Afterwards, make sure that all pods have either “Running” or “Completed” status by executing the following command:

+
kubectl -n netris-controller get pods
+
+
+

The output is similar to this:

+
NAME                                                      READY   STATUS      RESTARTS    AGE
+svclb-netris-controller-haproxy-6tkgj                     4/4     Running     0           38d
+netris-controller-haproxy-bcb944b7c-qcbf8                 1/1     Running     0           13d
+netris-controller-squid-7f6fdc6cf9-7fdx8                  1/1     Running     0           38d
+svclb-netris-controller-squid-58rnp                       1/1     Running     0           38d
+netris-controller-graphite-0                              1/1     Running     0           38d
+netris-controller-mongodb-0                               1/1     Running     0           38d
+netris-controller-redis-master-0                          1/1     Running     0           38d
+netris-controller-smtp-76778cf85f-lw5v5                   1/1     Running     0           10d
+netris-controller-mariadb-0                               1/1     Running     0           10d
+netris-controller-web-session-generator-8b9dbbcd8-8snhd   1/1     Running     0           10d
+netris-controller-telescope-notifier-647975848f-fs5dn     1/1     Running     0           10d
+netris-controller-app-b9b8d8f8d-4ssqb                     1/1     Running     0           10d
+netris-controller-grpc-987669fb9-jjskp                    1/1     Running     0           10d
+netris-controller-telescope-777c98c5d9-mqwl6              1/1     Running     0           10d
+helm-install-netris-controller-lqmq7                      0/1     Completed   0           20h
+
+
+
+

Warning

+

If, after 5 minutes, you see pods with a status other than “Running” or “Completed”, please reach out to us via Slack.

+
+

Then verify that the “Netris Version” reflects the version change by navigating to Setting → General in the Controller web interface.

+
    +
  1. Once you have verified that the Netris controller is up-to-date, it is time to update the switch and SoftGate agents.

  2. +
+

Upgrade the switch & SoftGate agents by copying the one-liner from the “Install Agent” option of the device’s 3-dot menu found under the Net → Inventory section and pasting it after SSHing to the corresponding device.

+Install Agent +

After all the agents have finished the upgrade process, make sure all devices in the Net → Inventory section have a “green” status and the Netris version for each device reflects the version change.

+

In the event the “check_agent” status is “Agent is unavailable” after the agent upgrade has finished, perform agent restart on the affected device(s).

+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl restart netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl restart netris-sg
+
+
+
+
+

Rollback Procedure

+

A rollback procedure can be executed in the event the upgrade introduced any adverse impact on the production traffic.

+
    +
  1. Stop all Netris agents on the devices managed by the controller (switch & SoftGate).

  2. +
+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl stop netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl stop netris-sg
+
+
+
    +
  1. Restore the database from the previously taken snapshot.

  2. +
+

Drop the current database and create a new one by running the following command after SSHing to the Controller:

+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"'
+kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"'
+
+
+

While still connected to the Controller, copy the backup file from the controller host system to the MariaDB container and restore the database:

+
kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql
+kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql'
+
+
+
    +
  1. Downgrade Netris Controller application with the following command.

  2. +
+
+

Note

+

For the version number, use the number collected from step #3 during the upgrade procedure.

+
+

Example:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-version 3.0.10-031
+
+
+

Afterwards, verify that the version of the “Netris Version” reflects the downgraded version by navigating to Setting → General in the Netris Controller.

+
    +
  1. Once you have verified that the Netris controller has been downgraded to the correct version, it is time to downgrade the switch and SoftGate agents.

  2. +
+

Install the correct and appropriate versions of the switch & SoftGate agents simply by copying the one-liner from the “Install Agent” option of the device’s 3-dot menu found under the Net → Inventory section and pasting it after SSHing to the corresponding device.

+

After all the switches and SoftGates have been successfully downgraded, make sure all the devices in the Net → Inventory section have a “green” status and the Netris version for each device reflects the version downgrade.

+

In case the “check_agent” status is “Agent is unavailable” after agent downgrade, perform agent restart.

+

For the switch agent, first SSH to the switch and run the following command:

+
sudo systemctl restart netris-sw
+
+
+

For the SoftGate agent, first SSH to the SoftGate and run the following command:

+
sudo systemctl restart netris-sg
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/upgrading-sonic-os.html b/en/3.4/tutorials/upgrading-sonic-os.html new file mode 100644 index 0000000000..45ece5dcaa --- /dev/null +++ b/en/3.4/tutorials/upgrading-sonic-os.html @@ -0,0 +1,745 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Upgrading SONiC Operating System — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Upgrading SONiC Operating System
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ + +
+

Upgrading SONiC Operating System

+
+

Upgrade Procedure

+

The SONiC OS image is being distributed by the Vendor of the switch, you need to obtain one via the support page of the vendor. +The image should be placed on a web server to which switch(es) have access. +In this example we will use a server with an IP address of 10.0.0.36, the image is located in the /var/www/html/ folder, which is the root of the web server.

+../_images/upgrading_sonic_folder_listing.png +
+

Hint

+

To make sure that the image is available as expected, please try to download the image with browser.

+
+

The upgrade process consists of the following steps:

+
    +
  1. Login to the switch and execute

  2. +
+
sudo sonic-installer install -y http://10.0.0.36/Edgecore-SONiC_20220929_052156_ec202012_420.bin
+
+
+

The process will take some time, after installation complete, reboot is mandatory.

+
+

Warning

+

The installation will wipe all the data on the switch, including the admin user password, the authorized keys and ssh identity. After the reboot, a new ssh identity is generated. Because of that, you will be prompted with an identity mismatch message on the first login attempt. Please use your OS-specific procedure to remove the old key.

+
+
    +
  1. Login to the switch and verify that the OS version is updated

  2. +
+
admin@switch15:~$ show version
+
+SONiC Software Version: SONiC.Edgecore-SONiC_20220929_052156_ec202012_420
+Distribution: Debian 10.13
+Kernel: 4.19.0-12-2-amd64
+Build commit: 895d178f6
+Build date: Thu Sep 29 05:49:16 UTC 2022
+Built by: ubuntu@ip-10-5-1-225
+
+Platform: x86_64-accton_as7326_56x-r0
+HwSKU: Accton-AS7326-56X
+ASIC: broadcom
+ASIC Count: 1
+Serial Number: REDACTED
+Uptime: 00:06:00,  1 user,  load average: 1.39, 2.08, 2.29
+
+
+
    +
  1. Perform netris agent installation steps 2 to 4 from “Install the Netris Agent” tutorial.

  2. +
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/using-l4-load-balancer.html b/en/3.4/tutorials/using-l4-load-balancer.html new file mode 100644 index 0000000000..c7f770126c --- /dev/null +++ b/en/3.4/tutorials/using-l4-load-balancer.html @@ -0,0 +1,718 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Using on-demand (elastic) L4 Load Balancer service — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using on-demand (elastic) L4 Load Balancer service

+

Services –> L4 Load Balancer is an on-demand (elastic) server load balancer. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console.

+../_images/netris-l4-load-balancer.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/using-vnet-in-equinix-metal-project.html b/en/3.4/tutorials/using-vnet-in-equinix-metal-project.html new file mode 100644 index 0000000000..8c74c96f94 --- /dev/null +++ b/en/3.4/tutorials/using-vnet-in-equinix-metal-project.html @@ -0,0 +1,740 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Using V-Net (isolated virtual network) services in Equinix Metal Project — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using V-Net (isolated virtual network) services in Equinix Metal Project

+

V-Net, under Services –> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from Equinix Metal Project into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network.

+

Netris V-Net (in Equinix Metal scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Layer-2 network using Equinix Metal API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Layer-2 network. SoftGate nodes are the default gateway for the V-Net services.

+

You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, Equinix, and your Compute.

+../_images/netris-creating-vnet-for-equinix-metal.png +

In this example, the new V-NET has VLAN ID 2, subnet 10.0.0.0/24, and gateway 10.0.0.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 2, and they should use IP addresses from 10.0.0.2-254 pointing to 10.0.0.1 as the default gateway. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 2 will have Internet access over the NAT.

+../_images/netris-vnet-ready-in-equinix-metal.png +
+

Tags

+

Starting from Netris v3.3.0, it is possible to dynamically associate bare-metal servers with V-NET. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers to the Layer-2 network based on that tag. Due to this, you can make flexible V-NETs, and there is no need to include every new server in the V-NET.

+../_images/equinix-metal-vnet-with-tag.png +

This feature is even more efficient when you build your infrastructure via Terraform. For example, let’s say you’ve created a V-NET with a tag using Netris Terraform Provider, then order several metal servers with the same tag using Equinix Metal Terraform Provider. And that’s it, when the servers are ready, Netris will detect them and make them part of the V-NET.

+../_images/equinix-metal-vnet-with-tag-terraform.png +
+
+

Unmanaged

+

Another option is turning the existing Layer-2 network (VLAN) into Netris V-Net. All VLANs in the particular project that aren’t used in other services, like an E-BGP, are visible as “unmanaged” in the V-Net section.

+../_images/unmanaged-vlan-equinix.png +../_images/unmanaged-vnet.png +

The “manage” button will open a dialogue window where it’s also possible to add a default gateway for the appropriate VLAN.

+
+

Warning

+

Once the VLAN is being converted into V-Net, it will be managed by Netris and no longer manageable through Equinix Metal console.

+
+../_images/manage-vnet.gif +

Note that you can use Services –> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-check-default-site.html b/en/3.4/tutorials/vpc-anywhere-check-default-site.html new file mode 100644 index 0000000000..7459824e76 --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-check-default-site.html @@ -0,0 +1,718 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Check Default Site Settings — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Check Default Site Settings

+

The VLAN ID range (under Net–>Site–>Default) defines what VLAN IDs Netris can use. You may want to ensure that this range does not overlap with other VLAN IDs you may have in your network.

+../_images/vpc-anywhere-check-site-default.png +

The default value is 700-900. Feel free to change to any range your network administrators are comfortable.

+../_images/vpc-anywhere-edit-vlan-range-default-site.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-concept.html b/en/3.4/tutorials/vpc-anywhere-concept.html new file mode 100644 index 0000000000..06f6e4fc7e --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-concept.html @@ -0,0 +1,726 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + VPC Anywhere Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

VPC Anywhere Overview

+
+

Introduction

+

VPC Anywhere is a solution designed by Netris that allows the integration of VPC network functionality into any network. Netris VPC can be used for all or part of the network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures.

+
+
+

Concept

+

SoftGate is a highly optimized automatic Linux gateway that communicates with Netris VPC controller through an encrypted management link.

+

A SoftGate node (typically two of them for redundancy) connects to the physical network on an 802.1q trunk port (similar to your Vmware or KVM hypervisors).

+

Netris needs to know what VLAN IDs it is allowed to utilize. We recommend reserving a range of VLAN IDs dedicated to Netris VPC. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the network.

+

SoftGate node becomes the default gateway for the workloads consuming the VPC network. Your existing switch network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers.

+../_images/vpc-anywhere-solution-traffic-flows.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-controller-installation.html b/en/3.4/tutorials/vpc-anywhere-controller-installation.html new file mode 100644 index 0000000000..f0c43f8764 --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-controller-installation.html @@ -0,0 +1,787 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Install a Netris Controller — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Install a Netris Controller

+
+

Requirements and Installation steps

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact, if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where to host the Netris controller. What does matter is that the Netris controller needs to be accessible over the Internet. 1) So you can access the web console. 2) Nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface.

+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

DNS record

+

In my example, my host got a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is an example using Cloudflare DNS service. (same idea with any DNS software or service)

+../_images/dns-record-netrisctl.png +

Ensure that the newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 54.183.23.201
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once the installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +

Restrict incoming TCP requests to the list below:

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + +

TCP Port

Service

22

SSH

80

HTTP

443

Netris Web Console

2003

Streaming Telemetry (Collectd)

3033

Netris Monitoring (Telescope)

50051

Netris Agent (gRPC)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-ipam-setup.html b/en/3.4/tutorials/vpc-anywhere-ipam-setup.html new file mode 100644 index 0000000000..aa6e83b588 --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-ipam-setup.html @@ -0,0 +1,736 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + IPAM Setup for Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

IPAM Setup for Services

+

To take advantage of the services provided by Netris, it is imperative to define Allocations/Subnets within the Netris IPAM. The Netris Controller comes with predefined subnets that serve the purpose of “common” and these subnets can be used to create V-Nets. However, to leverage services like NAT/L4 (elastic) Load Balancer, it is essential to create new Allocations and Subnets that align with the appropriate purpose. Alternatively, if an existing predefined subnet satisfies the requirements of your network architecture, it is possible to reconfigure it with the desired purpose.

+
+

Create an allocation

+

In addition to the predefined subnets, the Netris Controller also includes predefined allocations, consisting of private IP addresses defined in RFC 1918 - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. If you intend to create subnets that fall outside of these predefined allocations, you must first create allocations that encompass those subnets.

+../_images/vpc-anywhere-ipam-allocation.png +
+
+

Enable on-demand L4 (elastic) Load Balancer

+

Once the allocation is created, you can proceed with creating a subnet with the purpose of “load-balancer”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/vpc-anywhere-ipam-l4lb-subnet.png +

The on-demand L4 (elastic) Load Balancer service has been successfully activated and can be accessed through either the web console or Kubernetes using a service of the type load-balancer, or with Terraform.

+

It is worth noting that in the example provided, the Tenant field was left at its default setting of “Admin.” Tenancy is a feature that enables role-based access control and resource delegation. To illustrate, you may want to establish a user role and tenant for your colleagues who are meant to consume Netris VPC services but are not authorized to administer them.

+
+
+

Enable Network Address Translation (NAT)

+

To enable NAT, you need to create a subnet with the purpose of “nat”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/vpc-anywhere-ipam-nat-subnet.png +

The NAT service has been successfully activated. To create a NAT rule, navigate to the Net → NAT section of the Netris web console. Netris supports most standard SNAT and DNAT rules.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-softgate-installation.html b/en/3.4/tutorials/vpc-anywhere-softgate-installation.html new file mode 100644 index 0000000000..867c817c0d --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-softgate-installation.html @@ -0,0 +1,795 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

SoftGate Installation

+
+

Minimum Hardware Requirements

+
    +
  • 8 CPU cores

  • +
  • 16 GB RAM

  • +
  • 300 GB HDD

  • +
+
+
+

SoftGate software provisioning

+

The SoftGate deployment needs a freshly installed Ubuntu Linux 22.04 LTS.

+
+

Note

+

Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: “Adding SoftGates”.

+
+
    +
  1. Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the SoftGate node you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

  2. +
+../_images/softgate-install-agent.png +
    +
  1. Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node)

  2. +
+../_images/softgate-provisioning-cli-output.png +
+

Note

+

Please note that the Netris installation script replaces the default Netplan networking backend with regular ifupdown and attempts to migrate the configuration set during installation to /etc/network/interfaces.

+
+
    +
  1. Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.<xyz>). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as described in /etc/network/interfaces.

  2. +
+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node, otherwise adjust the line
+    # according to your setup.
+    up ip route add <Controller IP address> via <Management network gateway>
+
+# Physical port on SoftGate node connected to a TRUNK port of your network
+auto ens<X>
+iface ens<x> inet static
+    address 0.0.0.0/0
+
+# Optionally you can add more physical interfaces under your bond0, uncomment as needed
+#auto ens<Y>
+#iface ens<Y> inet static
+#    address 0.0.0.0/0
+
+# Bond interface
+auto bond0
+iface bond0 inet static
+    address 0.0.0.0/0
+    # Please replace/remove the ensX/Y with actual interface name(s) below to one(s) present in the OS.
+    bond-slaves ens<X>
+    # Optional, please adjust the bonding mode below according to the desired functionality.
+    bond-mode active-backup
+
+source /etc/network/interfaces.d/*
+
+
+
+

Note

+

Ensure that the Management network interface IP address is as expected so the SoftGate node will maintain IP connectivity with Netris Controller after reboot.

+
+
    +
  1. Reboot the SoftGate

  2. +
+
sudo reboot
+
+
+

Once the server boots up, you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+../_images/vpc-anywhere-softgates-green.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-upstream-peering.html b/en/3.4/tutorials/vpc-anywhere-upstream-peering.html new file mode 100644 index 0000000000..0ca6c618ab --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-upstream-peering.html @@ -0,0 +1,747 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Connecting VPC to upstream networks (use one of two options) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Connecting VPC to upstream networks (use one of two options)

+
+

Using a VLAN with public IP addresses (DMZ)

+

Netris VPC can use a traditional subnet or vlan with public IP addresses (a DMZ network) for upstream network peering.

+

logical diagram

+../_images/upstream-dmz-logical.png +

physical diagram

+../_images/upstream-dmz-physical.png +
+
+

BGP Upstream

+
+

BGP Upstream: overview

+

A BGP peering between Netris VPC and upstream routers is the most recommended option, especially for production use.

+

In this scenario, SoftGate nodes form BGP peering with upstream routers. Typically each BGP session will use an individual VLAN id and /30 subnet. Upstream BGP router should advertise the default route 0.0.0.0/0 or full Internet table if necessary. Netris SoftGate nodes are designed to handle over 1M routes in the routing table and can perform as border routers for the full-view table. +SoftGate nodes will automatically advertise public IP subnets in the current site as defined under the Net->IPAM section. Alternatively, you can optionally alter the default settings for full granular control over sent and received prefixes.

+

Logical diagram

+../_images/vpc-anywhere-upstream-bgp-router-logical.png +

Physical diagram

+../_images/vpc-anywhere-upstream-bgp-router-physical.png +
+
+

BGP Upstream: configuration

+

Your local public AS number is a site attribute located under the Net->Sites section. You can use the default private AS number or change to a real AS number depending on local network architecture needs.

+../_images/local-public-asn.png +

BGP peers can be defined under the Net->E-BGP section.

+../_images/add-new-ebgp-form.png +

Check BGP neighbor statuses under the Net->E-BGP

+../_images/bgp-listing.png +

To check/trobleshoot BGP use Net->Looking Glass

+../_images/bgp-looking-glass.png +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-using-l4lb.html b/en/3.4/tutorials/vpc-anywhere-using-l4lb.html new file mode 100644 index 0000000000..b891bb3bf2 --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-using-l4lb.html @@ -0,0 +1,719 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using on-demand (elastic) L4 Load Balancer service — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using on-demand (elastic) L4 Load Balancer service

+

Services –> L4 Load Balancer is an on-demand (elastic) L4 Load Balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console.

+../_images/vpc-anywhere-l4lb.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-using-multi-interface-softgate.html b/en/3.4/tutorials/vpc-anywhere-using-multi-interface-softgate.html new file mode 100644 index 0000000000..e124f054cb --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-using-multi-interface-softgate.html @@ -0,0 +1,730 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using Multi-interface SoftGate (experimental) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using Multi-interface SoftGate (experimental)

+

By default, Netris uses the bond0 interface of SoftGates exclusively. For each V-NET, Netris creates a new subinterface (e.g., bond0.700) with the next available VLAN ID from the Site’s defined VLAN ID range. However, a proposed experimental solution aims to modify this behavior.

+
+

Creating a V-Net

+

If you want to create a V-Net but prefer Netris to use existing interfaces instead of creating a new subinterface on the bond0 interface, you can achieve that by using the “int=<interface_name>” tag.

+

example - int=eth1

+../_images/vpc-anywhere-vnet-experimental.png +
+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-using-nat.html b/en/3.4/tutorials/vpc-anywhere-using-nat.html new file mode 100644 index 0000000000..b11e548f1d --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-using-nat.html @@ -0,0 +1,734 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using NAT services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using NAT services

+

NAT, under Net –> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 172.24.0.0/16 to have access to services outside their VPC. The goal is to provide instances with the capability to connect to the Internet through NAT for outbound access.

+
+

MASQUERADE

+

The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn’t require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT).

+

The Netris Controller includes a preconfigured MASQUERADE rule that facilitates instances within a private network with a subnet of 172.24.0.0/16 to have immediate access to services outside their VPC environment.

+../_images/vpc-anywhere-nat-masquerade.png +
+
+

SNAT

+

Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a NAT purpose. SNAT replaces the source IP address of the instances with the IP address of the “SNAT to IP”. Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity.

+../_images/vpc-anywhere-nat-snat.png +

You can always have more granular control either through NAT rule or using Services → ACLs.

+
+
+

DNAT

+

A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a “DNAT to IP/Port”. This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database.

+

I’m creating a DNAT rule for the ssh port in the example below. It forwards the public IP’s 55022 port to the local IP’s 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server.

+../_images/vpc-anywhere-nat-dnat.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere-using-vnet.html b/en/3.4/tutorials/vpc-anywhere-using-vnet.html new file mode 100644 index 0000000000..76effa975b --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere-using-vnet.html @@ -0,0 +1,723 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using V-Net (isolated virtual network) services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using V-Net (isolated virtual network) services

+

V-Net, under Services –> The V-Net service is a virtual network that operates in an isolated environment. In Netris, each V-Net takes the next available VLAN ID from the Site’s defined VLAN ID range and uses it as a global VLAN ID. The SoftGate nodes serve as the default gateway for the V-Net services.

+
+

Creating a V-Net

+

To ensure proper connectivity, it is recommended to create a corresponding V-Net for each virtual network used in Vmware, KVM, or any other server virtualization platform. The same applies to an isolated group of physical servers. VLAN ID serves as the unique identifier between Netris and the computing platform used.

+../_images/vpc-anywhere-vnet.png +

In this example, the new V-NET is assigned VLAN ID 700, subnet 172.24.0.0/20, and gateway 172.24.0.1. As a result, on your Compute side, you can launch VMs or subinterfaces on physical servers with a VLAN ID of 700 and obtain all necessary IP configurations from DHCP. Alternatively, if you prefer not to use DHCP, instances should use IP addresses from the 172.24.0.0/20 subnet and set the default gateway to 172.24.0.1. Netris SoftGate will manage this traffic, and the preconfigured MASQUERADE rule for the 172.24.0.0/16 subnet included in the Netris Controller will enable hosts in VLAN 700 to have internet access through NAT.

+

Note that you can use Services –> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/tutorials/vpc-anywhere.html b/en/3.4/tutorials/vpc-anywhere.html new file mode 100644 index 0000000000..749b5e1e5e --- /dev/null +++ b/en/3.4/tutorials/vpc-anywhere.html @@ -0,0 +1,769 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + VPC Anywhere Getting Started Guide — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/visibility.html b/en/3.4/visibility.html new file mode 100644 index 0000000000..763a69e0c5 --- /dev/null +++ b/en/3.4/visibility.html @@ -0,0 +1,770 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Visibility (Telescope) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Visibility (Telescope)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Visibility (Telescope)

+
+

Graph Boards

+

You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view.

+

To start with Graph Boards, first, you need to add a new Graph Board.

+
    +
  1. Navigate to Telescope→Graph Boards, open the dropdown menu in the top left corner, then click +Add board.

  2. +
+_images/telescope.png +
    +
  1. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants.

  2. +
+_images/createboard.png +

Now you can add graphs by clicking +Add graph.

+

Description of +Add graph fields:

+
    +
  • Title - Title for the new graph.

  • +
  • Type - Type of data source.

    +
      +
    • Bps - Traffic bits per second.

    • +
    • Pps - Traffic packets per second.

    • +
    • Errors - Errors per second.

    • +
    • Optical - Optical signal statistics/history.

    • +
    • MAC Count - History of the number of MAC addresses on the port.

    • +
    +
  • +
  • Function - Currently, only summing is supported.

  • +
  • +Member - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port.

  • +
+

Example: Sum of traffic on two ISP(Iris1 + Iris2) links.

+_images/ISP_Iris.png +

Example: Sum of the traffic on all ports under the service called “my V-NET”

+_images/V_NET.png +

Screenshot: Listing of a Graph Board with the explanation of the controls.

+_images/graphboard.png +
+
+

API Logs

+

Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type.

+
+
+

Dashboard

+

Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems.

+

Telescope→Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner.

+

Description of the pie charts.

+
    +
  • Hardware Health - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions.

  • +
  • E-BGP - Statuses of external BGP sessions.

  • +
  • LB VIP - Statuses of Load Balancer frontend / VIP availability.

  • +
  • LB Members - Statuses of Load Balancer backend members.

  • +
+

By clicking on each title you can see the details of the checks on the right side.

+

Screenshot: Dashboard showing details of “Hardware Health.”

+_images/hardware_health.png +

Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state.

+

Screenshot: “Save as normal” on selected ports.

+_images/saveasnormal.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/3.4/vnet.html b/en/3.4/vnet.html new file mode 100644 index 0000000000..b3b1938a42 --- /dev/null +++ b/en/3.4/vnet.html @@ -0,0 +1,746 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + V-Net — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

V-Net

+

V-Net is a virtual networking service that provides a Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or they can be created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization). +Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes.

+
+

V-Net Fields

+
    +
  • Name - Unique name for the V-Net

  • +
  • Owner - Tenant, who can make any changes to current V-Net

  • +
  • V-Net state - Active/Disable state for entire V-Net

  • +
  • VLAN aware - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible

  • +
  • Guest tenants - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters

  • +
  • Sites - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites).

  • +
  • IPv4 Gateway - IPv4 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site where V-Net is intended to span.

  • +
  • IPv6 Gateway - IPv6 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site or sites where V-Net is intended to span.

  • +
  • Port - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports.

    +
      +
    • Enabled - Enable or disable individual Switch Port under current V-Net

    • +
    • Port Name - Switch Port format: <alias>(swp<number>)@<switch name>

    • +
    • VLAN ID / Untag - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port’s ingress/egress unless VLAN aware mode is used.

    • +
    • LAG Mode - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes).

    • +
    +
  • +
+
+

Tip

+

Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gbps switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+
+_images/add-vnet.png +

+Example: Adding a new V-Net.

_images/list-vnet.png +

+Example: Listing of V-Nets.

_images/list-vnet-expanded.png +

+Example: Expanded view of a V-Net listing.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: 3.4 + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+
+ + +
latest
+ + +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/.buildinfo b/en/latest/.buildinfo new file mode 100644 index 0000000000..fdbad13e48 --- /dev/null +++ b/en/latest/.buildinfo @@ -0,0 +1,4 @@ +# Sphinx build info version 1 +# This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. +config: b95338b6024454eb231d9caee28b8d6f +tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/en/latest/.doctrees/Controller-local-repository.doctree b/en/latest/.doctrees/Controller-local-repository.doctree new file mode 100644 index 0000000000..f2e6362350 Binary files /dev/null and b/en/latest/.doctrees/Controller-local-repository.doctree differ diff --git a/en/latest/.doctrees/Dell-SONiC-Switch-initial-setup.doctree b/en/latest/.doctrees/Dell-SONiC-Switch-initial-setup.doctree new file mode 100644 index 0000000000..d00dac4548 Binary files /dev/null and b/en/latest/.doctrees/Dell-SONiC-Switch-initial-setup.doctree differ diff --git a/en/latest/.doctrees/EdgeCore-SONiC-Switch-initial-setup.doctree b/en/latest/.doctrees/EdgeCore-SONiC-Switch-initial-setup.doctree new file mode 100644 index 0000000000..c96b9c817d Binary files /dev/null and b/en/latest/.doctrees/EdgeCore-SONiC-Switch-initial-setup.doctree differ diff --git a/en/latest/.doctrees/Nvidia-Cumulus-v3.7-Switch-initial-setup.doctree b/en/latest/.doctrees/Nvidia-Cumulus-v3.7-Switch-initial-setup.doctree new file mode 100644 index 0000000000..97b7d67405 Binary files /dev/null and b/en/latest/.doctrees/Nvidia-Cumulus-v3.7-Switch-initial-setup.doctree differ diff --git a/en/latest/.doctrees/Nvidia-Cumulus-v5-Switch-initial-setup.doctree b/en/latest/.doctrees/Nvidia-Cumulus-v5-Switch-initial-setup.doctree new file mode 100644 index 0000000000..4b8fb3c910 Binary files /dev/null and b/en/latest/.doctrees/Nvidia-Cumulus-v5-Switch-initial-setup.doctree differ diff --git a/en/latest/.doctrees/Nvidia-Cumulus-v5.9+-Switch-initial-setup.doctree b/en/latest/.doctrees/Nvidia-Cumulus-v5.9+-Switch-initial-setup.doctree new file mode 100644 index 0000000000..21186c5332 Binary files /dev/null and b/en/latest/.doctrees/Nvidia-Cumulus-v5.9+-Switch-initial-setup.doctree differ diff --git a/en/latest/.doctrees/SoftGate-PRO-installation.doctree b/en/latest/.doctrees/SoftGate-PRO-installation.doctree new file mode 100644 index 0000000000..1cd5276e13 Binary files /dev/null and b/en/latest/.doctrees/SoftGate-PRO-installation.doctree differ diff --git a/en/latest/.doctrees/SoftGate-installation.doctree b/en/latest/.doctrees/SoftGate-installation.doctree new file mode 100644 index 0000000000..85500bd662 Binary files /dev/null and b/en/latest/.doctrees/SoftGate-installation.doctree differ diff --git a/en/latest/.doctrees/Ubuntu-SwitchDev-Switch-initial-setup.doctree b/en/latest/.doctrees/Ubuntu-SwitchDev-Switch-initial-setup.doctree new file mode 100644 index 0000000000..49620549dc Binary files /dev/null and b/en/latest/.doctrees/Ubuntu-SwitchDev-Switch-initial-setup.doctree differ diff --git a/en/latest/.doctrees/accounts.doctree b/en/latest/.doctrees/accounts.doctree new file mode 100644 index 0000000000..e6797fe2af Binary files /dev/null and b/en/latest/.doctrees/accounts.doctree differ diff --git a/en/latest/.doctrees/acls.doctree b/en/latest/.doctrees/acls.doctree new file mode 100644 index 0000000000..a542b65b01 Binary files /dev/null and b/en/latest/.doctrees/acls.doctree differ diff --git a/en/latest/.doctrees/controller-k3s-installation.doctree b/en/latest/.doctrees/controller-k3s-installation.doctree new file mode 100644 index 0000000000..4539a768d0 Binary files /dev/null and b/en/latest/.doctrees/controller-k3s-installation.doctree differ diff --git a/en/latest/.doctrees/controller-k8s-installation.doctree b/en/latest/.doctrees/controller-k8s-installation.doctree new file mode 100644 index 0000000000..216acfd9a8 Binary files /dev/null and b/en/latest/.doctrees/controller-k8s-installation.doctree differ diff --git a/en/latest/.doctrees/controller-k8s-quickstart.doctree b/en/latest/.doctrees/controller-k8s-quickstart.doctree new file mode 100644 index 0000000000..2052fba9be Binary files /dev/null and b/en/latest/.doctrees/controller-k8s-quickstart.doctree differ diff --git a/en/latest/.doctrees/controller-vm-installation.doctree b/en/latest/.doctrees/controller-vm-installation.doctree new file mode 100644 index 0000000000..03d887a76a Binary files /dev/null and b/en/latest/.doctrees/controller-vm-installation.doctree differ diff --git a/en/latest/.doctrees/definitions.doctree b/en/latest/.doctrees/definitions.doctree new file mode 100644 index 0000000000..13e1a19d5f Binary files /dev/null and b/en/latest/.doctrees/definitions.doctree differ diff --git a/en/latest/.doctrees/environment.pickle b/en/latest/.doctrees/environment.pickle new file mode 100644 index 0000000000..5d7e22c314 Binary files /dev/null and b/en/latest/.doctrees/environment.pickle differ diff --git a/en/latest/.doctrees/index.doctree b/en/latest/.doctrees/index.doctree new file mode 100644 index 0000000000..ea6d57a61d Binary files /dev/null and b/en/latest/.doctrees/index.doctree differ diff --git a/en/latest/.doctrees/installation.doctree b/en/latest/.doctrees/installation.doctree new file mode 100644 index 0000000000..21421277bb Binary files /dev/null and b/en/latest/.doctrees/installation.doctree differ diff --git a/en/latest/.doctrees/installing-netris-controller.doctree b/en/latest/.doctrees/installing-netris-controller.doctree new file mode 100644 index 0000000000..b670343cc6 Binary files /dev/null and b/en/latest/.doctrees/installing-netris-controller.doctree differ diff --git a/en/latest/.doctrees/introduction.doctree b/en/latest/.doctrees/introduction.doctree new file mode 100644 index 0000000000..af885d787e Binary files /dev/null and b/en/latest/.doctrees/introduction.doctree differ diff --git a/en/latest/.doctrees/inventory-profiles.doctree b/en/latest/.doctrees/inventory-profiles.doctree new file mode 100644 index 0000000000..75abb37c6f Binary files /dev/null and b/en/latest/.doctrees/inventory-profiles.doctree differ diff --git a/en/latest/.doctrees/ipam.doctree b/en/latest/.doctrees/ipam.doctree new file mode 100644 index 0000000000..e02d39882e Binary files /dev/null and b/en/latest/.doctrees/ipam.doctree differ diff --git a/en/latest/.doctrees/kubernetes-integration.doctree b/en/latest/.doctrees/kubernetes-integration.doctree new file mode 100644 index 0000000000..a254b0f8a4 Binary files /dev/null and b/en/latest/.doctrees/kubernetes-integration.doctree differ diff --git a/en/latest/.doctrees/l3-load-balancer.doctree b/en/latest/.doctrees/l3-load-balancer.doctree new file mode 100644 index 0000000000..9cc353f6f5 Binary files /dev/null and b/en/latest/.doctrees/l3-load-balancer.doctree differ diff --git a/en/latest/.doctrees/l4-load-balancer.doctree b/en/latest/.doctrees/l4-load-balancer.doctree new file mode 100644 index 0000000000..5f3038dc64 Binary files /dev/null and b/en/latest/.doctrees/l4-load-balancer.doctree differ diff --git a/en/latest/.doctrees/lag.doctree b/en/latest/.doctrees/lag.doctree new file mode 100644 index 0000000000..26bfe5633d Binary files /dev/null and b/en/latest/.doctrees/lag.doctree differ diff --git a/en/latest/.doctrees/maintenance-mode.doctree b/en/latest/.doctrees/maintenance-mode.doctree new file mode 100644 index 0000000000..290d6b1fe9 Binary files /dev/null and b/en/latest/.doctrees/maintenance-mode.doctree differ diff --git a/en/latest/.doctrees/netris-architecture.doctree b/en/latest/.doctrees/netris-architecture.doctree new file mode 100644 index 0000000000..7dd47e41d1 Binary files /dev/null and b/en/latest/.doctrees/netris-architecture.doctree differ diff --git a/en/latest/.doctrees/network-policies.doctree b/en/latest/.doctrees/network-policies.doctree new file mode 100644 index 0000000000..96284a5e10 Binary files /dev/null and b/en/latest/.doctrees/network-policies.doctree differ diff --git a/en/latest/.doctrees/reference-designs.doctree b/en/latest/.doctrees/reference-designs.doctree new file mode 100644 index 0000000000..7242a6ba41 Binary files /dev/null and b/en/latest/.doctrees/reference-designs.doctree differ diff --git a/en/latest/.doctrees/release-notes.doctree b/en/latest/.doctrees/release-notes.doctree new file mode 100644 index 0000000000..0111fa4795 Binary files /dev/null and b/en/latest/.doctrees/release-notes.doctree differ diff --git a/en/latest/.doctrees/roh.doctree b/en/latest/.doctrees/roh.doctree new file mode 100644 index 0000000000..3664e2b3fa Binary files /dev/null and b/en/latest/.doctrees/roh.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox1/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox1/configurations.doctree new file mode 100644 index 0000000000..a53615b345 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox1/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox1/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox1/creating-services.doctree new file mode 100644 index 0000000000..c7b16fa450 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox1/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox1/index.doctree b/en/latest/.doctrees/sandbox/Sandbox1/index.doctree new file mode 100644 index 0000000000..7d6f9a7881 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox1/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree new file mode 100644 index 0000000000..dc90e1f03e Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox1/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox1/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox1/sandbox-info.doctree new file mode 100644 index 0000000000..eea12e2e9a Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox1/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox10/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox10/configurations.doctree new file mode 100644 index 0000000000..9a4c33a811 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox10/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox10/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox10/creating-services.doctree new file mode 100644 index 0000000000..2a4e7e890b Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox10/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox10/index.doctree b/en/latest/.doctrees/sandbox/Sandbox10/index.doctree new file mode 100644 index 0000000000..fb2e77f217 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox10/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree new file mode 100644 index 0000000000..d3e214a7d2 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox10/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox10/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox10/sandbox-info.doctree new file mode 100644 index 0000000000..c992ee55c7 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox10/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox11/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox11/configurations.doctree new file mode 100644 index 0000000000..13262beebf Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox11/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox11/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox11/creating-services.doctree new file mode 100644 index 0000000000..a208da5cc5 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox11/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox11/index.doctree b/en/latest/.doctrees/sandbox/Sandbox11/index.doctree new file mode 100644 index 0000000000..8ca9ccada4 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox11/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree new file mode 100644 index 0000000000..346538a487 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox11/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox11/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox11/sandbox-info.doctree new file mode 100644 index 0000000000..a3b730913e Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox11/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox12/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox12/configurations.doctree new file mode 100644 index 0000000000..c8962fcce1 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox12/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox12/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox12/creating-services.doctree new file mode 100644 index 0000000000..ea3a394bc1 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox12/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox12/index.doctree b/en/latest/.doctrees/sandbox/Sandbox12/index.doctree new file mode 100644 index 0000000000..aab142020f Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox12/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree new file mode 100644 index 0000000000..65608d3c96 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox12/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox12/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox12/sandbox-info.doctree new file mode 100644 index 0000000000..ba6e0e5fc6 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox12/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox13/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox13/configurations.doctree new file mode 100644 index 0000000000..761a3b9fc6 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox13/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox13/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox13/creating-services.doctree new file mode 100644 index 0000000000..4727738855 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox13/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox13/index.doctree b/en/latest/.doctrees/sandbox/Sandbox13/index.doctree new file mode 100644 index 0000000000..c3591fcfcd Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox13/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree new file mode 100644 index 0000000000..f7eabf05fb Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox13/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox13/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox13/sandbox-info.doctree new file mode 100644 index 0000000000..05bbc1ae3b Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox13/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox14/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox14/configurations.doctree new file mode 100644 index 0000000000..0125f5e815 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox14/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox14/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox14/creating-services.doctree new file mode 100644 index 0000000000..95467ca2b4 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox14/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox14/index.doctree b/en/latest/.doctrees/sandbox/Sandbox14/index.doctree new file mode 100644 index 0000000000..b84a28db81 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox14/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree new file mode 100644 index 0000000000..a6b938b8fe Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox14/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox14/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox14/sandbox-info.doctree new file mode 100644 index 0000000000..11ded6e955 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox14/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox15/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox15/configurations.doctree new file mode 100644 index 0000000000..c316baedee Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox15/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox15/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox15/creating-services.doctree new file mode 100644 index 0000000000..b6248e0fcc Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox15/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox15/index.doctree b/en/latest/.doctrees/sandbox/Sandbox15/index.doctree new file mode 100644 index 0000000000..dd2520119f Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox15/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree new file mode 100644 index 0000000000..60bb2026f4 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox15/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox15/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox15/sandbox-info.doctree new file mode 100644 index 0000000000..7961480f9e Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox15/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox2/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox2/configurations.doctree new file mode 100644 index 0000000000..304dd80600 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox2/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox2/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox2/creating-services.doctree new file mode 100644 index 0000000000..4a38ad79ec Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox2/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox2/index.doctree b/en/latest/.doctrees/sandbox/Sandbox2/index.doctree new file mode 100644 index 0000000000..3bccb750ba Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox2/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree new file mode 100644 index 0000000000..5f1ea8db56 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox2/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox2/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox2/sandbox-info.doctree new file mode 100644 index 0000000000..2274fbb29c Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox2/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox3/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox3/configurations.doctree new file mode 100644 index 0000000000..cff6ea50cf Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox3/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox3/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox3/creating-services.doctree new file mode 100644 index 0000000000..75ebf1ae5b Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox3/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox3/index.doctree b/en/latest/.doctrees/sandbox/Sandbox3/index.doctree new file mode 100644 index 0000000000..0b8edf7c91 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox3/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree new file mode 100644 index 0000000000..db729e72e1 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox3/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox3/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox3/sandbox-info.doctree new file mode 100644 index 0000000000..71b16d19b0 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox3/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox4/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox4/configurations.doctree new file mode 100644 index 0000000000..1c014fa56d Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox4/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox4/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox4/creating-services.doctree new file mode 100644 index 0000000000..6f654595df Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox4/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox4/index.doctree b/en/latest/.doctrees/sandbox/Sandbox4/index.doctree new file mode 100644 index 0000000000..aaf55d6d3a Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox4/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree new file mode 100644 index 0000000000..f07a856a96 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox4/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox4/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox4/sandbox-info.doctree new file mode 100644 index 0000000000..b565c73616 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox4/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox5/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox5/configurations.doctree new file mode 100644 index 0000000000..ed03c69213 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox5/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox5/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox5/creating-services.doctree new file mode 100644 index 0000000000..5294688673 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox5/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox5/index.doctree b/en/latest/.doctrees/sandbox/Sandbox5/index.doctree new file mode 100644 index 0000000000..4a8b7df57b Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox5/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree new file mode 100644 index 0000000000..edbd015a31 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox5/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox5/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox5/sandbox-info.doctree new file mode 100644 index 0000000000..816d1b1fea Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox5/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox6/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox6/configurations.doctree new file mode 100644 index 0000000000..e8cdd41749 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox6/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox6/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox6/creating-services.doctree new file mode 100644 index 0000000000..b342f3c1bd Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox6/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox6/index.doctree b/en/latest/.doctrees/sandbox/Sandbox6/index.doctree new file mode 100644 index 0000000000..11e256b5bc Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox6/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree new file mode 100644 index 0000000000..050750e1ce Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox6/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox6/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox6/sandbox-info.doctree new file mode 100644 index 0000000000..806c333075 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox6/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox7/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox7/configurations.doctree new file mode 100644 index 0000000000..28ec5d7a9b Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox7/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox7/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox7/creating-services.doctree new file mode 100644 index 0000000000..caf72df176 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox7/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox7/index.doctree b/en/latest/.doctrees/sandbox/Sandbox7/index.doctree new file mode 100644 index 0000000000..c96a4a97c4 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox7/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree new file mode 100644 index 0000000000..cd0345a7dc Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox7/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox7/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox7/sandbox-info.doctree new file mode 100644 index 0000000000..ef8b8a7326 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox7/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox8/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox8/configurations.doctree new file mode 100644 index 0000000000..080dccbdf0 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox8/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox8/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox8/creating-services.doctree new file mode 100644 index 0000000000..c915e70b24 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox8/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox8/index.doctree b/en/latest/.doctrees/sandbox/Sandbox8/index.doctree new file mode 100644 index 0000000000..44b996be57 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox8/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree new file mode 100644 index 0000000000..c05aea7d7b Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox8/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox8/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox8/sandbox-info.doctree new file mode 100644 index 0000000000..5055f6d8c3 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox8/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox9/configurations.doctree b/en/latest/.doctrees/sandbox/Sandbox9/configurations.doctree new file mode 100644 index 0000000000..b9caadb2c1 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox9/configurations.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox9/creating-services.doctree b/en/latest/.doctrees/sandbox/Sandbox9/creating-services.doctree new file mode 100644 index 0000000000..d2744cdd41 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox9/creating-services.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox9/index.doctree b/en/latest/.doctrees/sandbox/Sandbox9/index.doctree new file mode 100644 index 0000000000..e9d9bf9e9f Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox9/index.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree b/en/latest/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree new file mode 100644 index 0000000000..b9abf6c84e Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox9/onprem-k8s.doctree differ diff --git a/en/latest/.doctrees/sandbox/Sandbox9/sandbox-info.doctree b/en/latest/.doctrees/sandbox/Sandbox9/sandbox-info.doctree new file mode 100644 index 0000000000..8834783484 Binary files /dev/null and b/en/latest/.doctrees/sandbox/Sandbox9/sandbox-info.doctree differ diff --git a/en/latest/.doctrees/softgate-performance.doctree b/en/latest/.doctrees/softgate-performance.doctree new file mode 100644 index 0000000000..eebf6702f3 Binary files /dev/null and b/en/latest/.doctrees/softgate-performance.doctree differ diff --git a/en/latest/.doctrees/supported-networks.doctree b/en/latest/.doctrees/supported-networks.doctree new file mode 100644 index 0000000000..bf7cfd412a Binary files /dev/null and b/en/latest/.doctrees/supported-networks.doctree differ diff --git a/en/latest/.doctrees/supported-platform-matrix.doctree b/en/latest/.doctrees/supported-platform-matrix.doctree new file mode 100644 index 0000000000..905d79a5d4 Binary files /dev/null and b/en/latest/.doctrees/supported-platform-matrix.doctree differ diff --git a/en/latest/.doctrees/supported-switch-hardware.doctree b/en/latest/.doctrees/supported-switch-hardware.doctree new file mode 100644 index 0000000000..ad57286e59 Binary files /dev/null and b/en/latest/.doctrees/supported-switch-hardware.doctree differ diff --git a/en/latest/.doctrees/switch-agent-installation.doctree b/en/latest/.doctrees/switch-agent-installation.doctree new file mode 100644 index 0000000000..b198effefa Binary files /dev/null and b/en/latest/.doctrees/switch-agent-installation.doctree differ diff --git a/en/latest/.doctrees/switch-ports.doctree b/en/latest/.doctrees/switch-ports.doctree new file mode 100644 index 0000000000..2ce7d5b0d2 Binary files /dev/null and b/en/latest/.doctrees/switch-ports.doctree differ diff --git a/en/latest/.doctrees/terraform-integration.doctree b/en/latest/.doctrees/terraform-integration.doctree new file mode 100644 index 0000000000..1e9b325223 Binary files /dev/null and b/en/latest/.doctrees/terraform-integration.doctree differ diff --git a/en/latest/.doctrees/topology-management.doctree b/en/latest/.doctrees/topology-management.doctree new file mode 100644 index 0000000000..d757f2b0ee Binary files /dev/null and b/en/latest/.doctrees/topology-management.doctree differ diff --git a/en/latest/.doctrees/try-learn/index.doctree b/en/latest/.doctrees/try-learn/index.doctree new file mode 100644 index 0000000000..838c126591 Binary files /dev/null and b/en/latest/.doctrees/try-learn/index.doctree differ diff --git a/en/latest/.doctrees/try-learn/netris-cloudsim.doctree b/en/latest/.doctrees/try-learn/netris-cloudsim.doctree new file mode 100644 index 0000000000..08934b59da Binary files /dev/null and b/en/latest/.doctrees/try-learn/netris-cloudsim.doctree differ diff --git a/en/latest/.doctrees/try-learn/nvidia-spectrum-x-scenario.doctree b/en/latest/.doctrees/try-learn/nvidia-spectrum-x-scenario.doctree new file mode 100644 index 0000000000..7a35da3480 Binary files /dev/null and b/en/latest/.doctrees/try-learn/nvidia-spectrum-x-scenario.doctree differ diff --git a/en/latest/.doctrees/try-learn/using-netris-cloudsim.doctree b/en/latest/.doctrees/try-learn/using-netris-cloudsim.doctree new file mode 100644 index 0000000000..3a5e1abbb9 Binary files /dev/null and b/en/latest/.doctrees/try-learn/using-netris-cloudsim.doctree differ diff --git a/en/latest/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree b/en/latest/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree new file mode 100644 index 0000000000..937fb11c37 Binary files /dev/null and b/en/latest/.doctrees/tutorials/activating-bgp-on-equinix-metal-project.doctree differ diff --git a/en/latest/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree b/en/latest/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree new file mode 100644 index 0000000000..46bcbcf882 Binary files /dev/null and b/en/latest/.doctrees/tutorials/adding-netris-softgate-nodes-in-equinix-metal.doctree differ diff --git a/en/latest/.doctrees/tutorials/aws-concept.doctree b/en/latest/.doctrees/tutorials/aws-concept.doctree new file mode 100644 index 0000000000..bc569a08d9 Binary files /dev/null and b/en/latest/.doctrees/tutorials/aws-concept.doctree differ diff --git a/en/latest/.doctrees/tutorials/aws-deploy-softgate.doctree b/en/latest/.doctrees/tutorials/aws-deploy-softgate.doctree new file mode 100644 index 0000000000..672a92e778 Binary files /dev/null and b/en/latest/.doctrees/tutorials/aws-deploy-softgate.doctree differ diff --git a/en/latest/.doctrees/tutorials/connecting-fabric-to-isp.doctree b/en/latest/.doctrees/tutorials/connecting-fabric-to-isp.doctree new file mode 100644 index 0000000000..0bf274ef78 Binary files /dev/null and b/en/latest/.doctrees/tutorials/connecting-fabric-to-isp.doctree differ diff --git a/en/latest/.doctrees/tutorials/connecting-servers-fabric.doctree b/en/latest/.doctrees/tutorials/connecting-servers-fabric.doctree new file mode 100644 index 0000000000..e2a98960db Binary files /dev/null and b/en/latest/.doctrees/tutorials/connecting-servers-fabric.doctree differ diff --git a/en/latest/.doctrees/tutorials/create-interconnection-to-fabric.doctree b/en/latest/.doctrees/tutorials/create-interconnection-to-fabric.doctree new file mode 100644 index 0000000000..d2c7d629c3 Binary files /dev/null and b/en/latest/.doctrees/tutorials/create-interconnection-to-fabric.doctree differ diff --git a/en/latest/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree b/en/latest/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree new file mode 100644 index 0000000000..f6b9065b7b Binary files /dev/null and b/en/latest/.doctrees/tutorials/enable-services-on-equinix-metal-project.doctree differ diff --git a/en/latest/.doctrees/tutorials/enabling-load-balancing-services.doctree b/en/latest/.doctrees/tutorials/enabling-load-balancing-services.doctree new file mode 100644 index 0000000000..c2b03f3dc5 Binary files /dev/null and b/en/latest/.doctrees/tutorials/enabling-load-balancing-services.doctree differ diff --git a/en/latest/.doctrees/tutorials/enabling-nat-services.doctree b/en/latest/.doctrees/tutorials/enabling-nat-services.doctree new file mode 100644 index 0000000000..3121b9c735 Binary files /dev/null and b/en/latest/.doctrees/tutorials/enabling-nat-services.doctree differ diff --git a/en/latest/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree b/en/latest/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree new file mode 100644 index 0000000000..0d93059afb Binary files /dev/null and b/en/latest/.doctrees/tutorials/equinix-metal-api-integration-enablement.doctree differ diff --git a/en/latest/.doctrees/tutorials/gcp-concept.doctree b/en/latest/.doctrees/tutorials/gcp-concept.doctree new file mode 100644 index 0000000000..66f1515ff9 Binary files /dev/null and b/en/latest/.doctrees/tutorials/gcp-concept.doctree differ diff --git a/en/latest/.doctrees/tutorials/gcp-deploy-softgate.doctree b/en/latest/.doctrees/tutorials/gcp-deploy-softgate.doctree new file mode 100644 index 0000000000..3d14824ee3 Binary files /dev/null and b/en/latest/.doctrees/tutorials/gcp-deploy-softgate.doctree differ diff --git a/en/latest/.doctrees/tutorials/getting-started-for-equinix-metal.doctree b/en/latest/.doctrees/tutorials/getting-started-for-equinix-metal.doctree new file mode 100644 index 0000000000..9ba3cb5bbc Binary files /dev/null and b/en/latest/.doctrees/tutorials/getting-started-for-equinix-metal.doctree differ diff --git a/en/latest/.doctrees/tutorials/index-equinix.doctree b/en/latest/.doctrees/tutorials/index-equinix.doctree new file mode 100644 index 0000000000..81861d983f Binary files /dev/null and b/en/latest/.doctrees/tutorials/index-equinix.doctree differ diff --git a/en/latest/.doctrees/tutorials/index-vpc.doctree b/en/latest/.doctrees/tutorials/index-vpc.doctree new file mode 100644 index 0000000000..85bdf43362 Binary files /dev/null and b/en/latest/.doctrees/tutorials/index-vpc.doctree differ diff --git a/en/latest/.doctrees/tutorials/index.doctree b/en/latest/.doctrees/tutorials/index.doctree new file mode 100644 index 0000000000..a0cff748d5 Binary files /dev/null and b/en/latest/.doctrees/tutorials/index.doctree differ diff --git a/en/latest/.doctrees/tutorials/install-netris-controller-in-equinix-metal.doctree b/en/latest/.doctrees/tutorials/install-netris-controller-in-equinix-metal.doctree new file mode 100644 index 0000000000..b8f2b1bd21 Binary files /dev/null and b/en/latest/.doctrees/tutorials/install-netris-controller-in-equinix-metal.doctree differ diff --git a/en/latest/.doctrees/tutorials/installing-netris-controller.doctree b/en/latest/.doctrees/tutorials/installing-netris-controller.doctree new file mode 100644 index 0000000000..628f395ac9 Binary files /dev/null and b/en/latest/.doctrees/tutorials/installing-netris-controller.doctree differ diff --git a/en/latest/.doctrees/tutorials/inventory-setup.doctree b/en/latest/.doctrees/tutorials/inventory-setup.doctree new file mode 100644 index 0000000000..c9d1f71533 Binary files /dev/null and b/en/latest/.doctrees/tutorials/inventory-setup.doctree differ diff --git a/en/latest/.doctrees/tutorials/ipam-setup.doctree b/en/latest/.doctrees/tutorials/ipam-setup.doctree new file mode 100644 index 0000000000..1b3b892d32 Binary files /dev/null and b/en/latest/.doctrees/tutorials/ipam-setup.doctree differ diff --git a/en/latest/.doctrees/tutorials/more-features.doctree b/en/latest/.doctrees/tutorials/more-features.doctree new file mode 100644 index 0000000000..aaadfe17bd Binary files /dev/null and b/en/latest/.doctrees/tutorials/more-features.doctree differ diff --git a/en/latest/.doctrees/tutorials/netris-controller-installation.doctree b/en/latest/.doctrees/tutorials/netris-controller-installation.doctree new file mode 100644 index 0000000000..50f8c80e29 Binary files /dev/null and b/en/latest/.doctrees/tutorials/netris-controller-installation.doctree differ diff --git a/en/latest/.doctrees/tutorials/netris-managed-fabric-overview.doctree b/en/latest/.doctrees/tutorials/netris-managed-fabric-overview.doctree new file mode 100644 index 0000000000..760a035024 Binary files /dev/null and b/en/latest/.doctrees/tutorials/netris-managed-fabric-overview.doctree differ diff --git a/en/latest/.doctrees/tutorials/netris-switch-agent-installation.doctree b/en/latest/.doctrees/tutorials/netris-switch-agent-installation.doctree new file mode 100644 index 0000000000..fb8b5ceeb9 Binary files /dev/null and b/en/latest/.doctrees/tutorials/netris-switch-agent-installation.doctree differ diff --git a/en/latest/.doctrees/tutorials/netris-vpc-for-aws.doctree b/en/latest/.doctrees/tutorials/netris-vpc-for-aws.doctree new file mode 100644 index 0000000000..f512ba0bde Binary files /dev/null and b/en/latest/.doctrees/tutorials/netris-vpc-for-aws.doctree differ diff --git a/en/latest/.doctrees/tutorials/netris-vpc-for-equinix-metal.doctree b/en/latest/.doctrees/tutorials/netris-vpc-for-equinix-metal.doctree new file mode 100644 index 0000000000..832980c448 Binary files /dev/null and b/en/latest/.doctrees/tutorials/netris-vpc-for-equinix-metal.doctree differ diff --git a/en/latest/.doctrees/tutorials/netris-vpc-for-gcp.doctree b/en/latest/.doctrees/tutorials/netris-vpc-for-gcp.doctree new file mode 100644 index 0000000000..5d5c87e4b5 Binary files /dev/null and b/en/latest/.doctrees/tutorials/netris-vpc-for-gcp.doctree differ diff --git a/en/latest/.doctrees/tutorials/netris-vpc-for-phoenixnap-bmc.doctree b/en/latest/.doctrees/tutorials/netris-vpc-for-phoenixnap-bmc.doctree new file mode 100644 index 0000000000..f57d229b7f Binary files /dev/null and b/en/latest/.doctrees/tutorials/netris-vpc-for-phoenixnap-bmc.doctree differ diff --git a/en/latest/.doctrees/tutorials/new-site-setup.doctree b/en/latest/.doctrees/tutorials/new-site-setup.doctree new file mode 100644 index 0000000000..0005630dd0 Binary files /dev/null and b/en/latest/.doctrees/tutorials/new-site-setup.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.doctree new file mode 100644 index 0000000000..ed1869712f Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-api-integration-enablement.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-api-integration-enablement.doctree new file mode 100644 index 0000000000..4686b17c8d Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-api-integration-enablement.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-concept.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-concept.doctree new file mode 100644 index 0000000000..5844bcb0e4 Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-concept.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-install-netris-controller.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-install-netris-controller.doctree new file mode 100644 index 0000000000..7dfc63763b Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-install-netris-controller.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-ipam-setup.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-ipam-setup.doctree new file mode 100644 index 0000000000..119f6aec72 Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-ipam-setup.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-link-to-installation.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-link-to-installation.doctree new file mode 100644 index 0000000000..2cfcbd1694 Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-link-to-installation.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-l4lb.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-l4lb.doctree new file mode 100644 index 0000000000..9090cdd09f Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-l4lb.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-nat.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-nat.doctree new file mode 100644 index 0000000000..6a89b46f0b Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-nat.doctree differ diff --git a/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-vnet.doctree b/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-vnet.doctree new file mode 100644 index 0000000000..8c49996559 Binary files /dev/null and b/en/latest/.doctrees/tutorials/phoenixnap-bmc-using-vnet.doctree differ diff --git a/en/latest/.doctrees/tutorials/softgate-software-provisioning.doctree b/en/latest/.doctrees/tutorials/softgate-software-provisioning.doctree new file mode 100644 index 0000000000..445010dab9 Binary files /dev/null and b/en/latest/.doctrees/tutorials/softgate-software-provisioning.doctree differ diff --git a/en/latest/.doctrees/tutorials/topology-setup.doctree b/en/latest/.doctrees/tutorials/topology-setup.doctree new file mode 100644 index 0000000000..8953d02338 Binary files /dev/null and b/en/latest/.doctrees/tutorials/topology-setup.doctree differ diff --git a/en/latest/.doctrees/tutorials/upgrading-netris.doctree b/en/latest/.doctrees/tutorials/upgrading-netris.doctree new file mode 100644 index 0000000000..926d767778 Binary files /dev/null and b/en/latest/.doctrees/tutorials/upgrading-netris.doctree differ diff --git a/en/latest/.doctrees/tutorials/upgrading-sonic-os.doctree b/en/latest/.doctrees/tutorials/upgrading-sonic-os.doctree new file mode 100644 index 0000000000..2997dfe789 Binary files /dev/null and b/en/latest/.doctrees/tutorials/upgrading-sonic-os.doctree differ diff --git a/en/latest/.doctrees/tutorials/using-l4-load-balancer.doctree b/en/latest/.doctrees/tutorials/using-l4-load-balancer.doctree new file mode 100644 index 0000000000..06fb4c0b67 Binary files /dev/null and b/en/latest/.doctrees/tutorials/using-l4-load-balancer.doctree differ diff --git a/en/latest/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree b/en/latest/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree new file mode 100644 index 0000000000..21ef790cd5 Binary files /dev/null and b/en/latest/.doctrees/tutorials/using-vnet-in-equinix-metal-project.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-check-default-site.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-check-default-site.doctree new file mode 100644 index 0000000000..f2430e5d6e Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-check-default-site.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-concept.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-concept.doctree new file mode 100644 index 0000000000..3a5275a491 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-concept.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-controller-installation.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-controller-installation.doctree new file mode 100644 index 0000000000..2e0b08425f Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-controller-installation.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-ipam-setup.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-ipam-setup.doctree new file mode 100644 index 0000000000..197a1d7c21 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-ipam-setup.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-softgate-installation.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-softgate-installation.doctree new file mode 100644 index 0000000000..90d41982be Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-softgate-installation.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-upstream-peering.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-upstream-peering.doctree new file mode 100644 index 0000000000..6aa275f9b0 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-upstream-peering.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-using-l4lb.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-using-l4lb.doctree new file mode 100644 index 0000000000..e08bf3e098 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-using-l4lb.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-using-multi-interface-softgate.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-using-multi-interface-softgate.doctree new file mode 100644 index 0000000000..e9cd1bc83d Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-using-multi-interface-softgate.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-using-nat.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-using-nat.doctree new file mode 100644 index 0000000000..d8b2f53b84 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-using-nat.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere-using-vnet.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere-using-vnet.doctree new file mode 100644 index 0000000000..6240d44665 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere-using-vnet.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-anywhere.doctree b/en/latest/.doctrees/tutorials/vpc-anywhere.doctree new file mode 100644 index 0000000000..3a96e234d3 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-anywhere.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-gateways-with-managed-fabric.doctree b/en/latest/.doctrees/tutorials/vpc-gateways-with-managed-fabric.doctree new file mode 100644 index 0000000000..2a475db888 Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-gateways-with-managed-fabric.doctree differ diff --git a/en/latest/.doctrees/tutorials/vpc-setup.doctree b/en/latest/.doctrees/tutorials/vpc-setup.doctree new file mode 100644 index 0000000000..87500d5e2d Binary files /dev/null and b/en/latest/.doctrees/tutorials/vpc-setup.doctree differ diff --git a/en/latest/.doctrees/visibility.doctree b/en/latest/.doctrees/visibility.doctree new file mode 100644 index 0000000000..d4d9f20de1 Binary files /dev/null and b/en/latest/.doctrees/visibility.doctree differ diff --git a/en/latest/.doctrees/vnet.doctree b/en/latest/.doctrees/vnet.doctree new file mode 100644 index 0000000000..22849a3f0e Binary files /dev/null and b/en/latest/.doctrees/vnet.doctree differ diff --git a/en/latest/.doctrees/vpc.doctree b/en/latest/.doctrees/vpc.doctree new file mode 100644 index 0000000000..e95f05f377 Binary files /dev/null and b/en/latest/.doctrees/vpc.doctree differ diff --git a/en/latest/.nojekyll b/en/latest/.nojekyll new file mode 100644 index 0000000000..e69de29bb2 diff --git a/en/latest/Controller-local-repository.html b/en/latest/Controller-local-repository.html new file mode 100644 index 0000000000..1bd909a218 --- /dev/null +++ b/en/latest/Controller-local-repository.html @@ -0,0 +1,923 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Local Repository Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Local Repository Setup

+
+

When & Why to Use the Local Repository?

+

The Netris Local Repository is essential for environments where switches, softgates, or other infrastructure devices do not have direct access to the internet. By setting up a local repository, you ensure that these devices can still download necessary packages and updates through a local APT repository. This setup is particularly useful in air-gapped or restricted network environments, where relying on external repositories is not an option. +In addition to serving as an APT repository, the local repository can also host custom ISO files. This feature allows you to store and manage ISOs for specific use cases, such as deploying operating systems or firmware to your servers, softgates, or switches. While ISO hosting is an optional feature, it provides flexibility and control over the resources you use in your infrastructure, especially when working in isolated networks. +By utilizing the Netris Local Repository, you ensure that your infrastructure can always access necessary files and packages, even when internet connectivity is limited or unavailable.

+
+
+

How to Enable the Local Repository on the Netris Controller?

+

To install the Netris Local Repository on the controller host, run the following command: +Note: The controller must be set up using the `–ctl-hostname` argument; otherwise, the Netris Local Repository setup script will exit with an error.

+
curl -sS https://get.netris.io/local-repo.sh | sh -
+
+
+

The script may take a while, depending on the controller’s connectivity and available resources.

+

The output of the script will look similar to this:

+
ubuntu@netris:~$ curl -sS https://get.netris.io/local-repo.sh | sh -
+Checking if the controller is installed
+Checking controller's FQDN
+Creating local repo service under URL: http://netris.example.com/repo/
+Waiting for netris-local-repo pod to be ready...
+No pods found. Waiting for pod creation...
+Current pod status: Pending. Waiting...
+Current pod status: Pending. Waiting...
+Current pod status: Pending. Waiting...
+Current pod status: Pending. Waiting...
+Current pod status: Pending. Waiting...
+Pod is running.
+
+Netris local repository pod is now running.
+
+Path to upload custom ISOs: /var/lib/rancher/k3s/storage/pvc-ea0dd3ef-ded1-49d3-bbd8-b797c91d76b5_netris-controller_netris-local-repo-pvc/repo/isos
+Local repository URL: http://netris.example.com/repo/
+URL to ISOs directory: http://netris.example.com/repo/isos/
+
+
+

The script will output the local repository URL and URL to ISOs directory. Copy and paste this URL of the Local repository into the Netris Controller Web UI under Settings section (as shown in the screenshots below).

+

Additionally, it provides the host system path, which you may want to use to host your custom ISOs for your servers, softgates, or switches.

+_images/Global-settings-edit.png +_images/Global-settings-save.png +
+
+

How to consume local repository

+

Once the local repository function is enabled in the Netris Controller Settings, the Netris agent installation oneliner will automatically point to the local repository (as shown in the screenshots below).

+
+

Note

+

The local repository includes all the necessary scripts and dependency packages for the Netris NVUE (Cumulus 5.9 and higher) and Netris SoftGate HS (Ubuntu 24.04) agents.

+
+_images/oneliner-from-local-repo.png +
+
+

Upgrade Local-Repo Cache

+

To update the local repository, restart the local-repo deployment in the controller. Then, verify the deployment status to ensure the rollout succeeds:

+
kubectl -nnetris-controller rollout restart deploy/netris-local-repo-nginx
+kubectl -nnetris-controller rollout status deploy/netris-local-repo-nginx
+
+
+

Expected output:

+
ubuntu@netris:~$ kubectl -nnetris-controller rollout restart deploy/netris-local-repo-nginx
+deployment.apps/netris-local-repo-nginx restarted
+
+ubuntu@netris:~$ kubectl -nnetris-controller rollout status deploy/netris-local-repo-nginx
+Waiting for deployment "netris-local-repo-nginx" rollout to finish: 1 old replicas are pending termination...
+Waiting for deployment "netris-local-repo-nginx" rollout to finish: 1 old replicas are pending termination...
+deployment "netris-local-repo-nginx" successfully rolled out
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/Dell-SONiC-Switch-initial-setup.html b/en/latest/Dell-SONiC-Switch-initial-setup.html new file mode 100644 index 0000000000..00e36f2bdc --- /dev/null +++ b/en/latest/Dell-SONiC-Switch-initial-setup.html @@ -0,0 +1,931 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Dell SONiC Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Dell SONiC Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first.

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of Dell SONiC.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Dell SONiC image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/Enterprise_SONiC_OS_4.1.1_Enterprise_Premium.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password:

+

admin/YourPaSsWoRd

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Upon the initial boot of the newly installed NOS, please wait until the message “System is ready” appears, typically resembling the following:

+
May 03 15:02:35.430469 System is ready
+
+
+

Once this message is displayed, proceed to disable ZTP using the following commands:

+
admin@sonic:~$ sonic-cli
+sonic# configure
+sonic(config)# no ztp enable
+
+
+

Wait once more until the “System is ready” message reappears. This may take approximately 1-2 minutes. +If there isn’t a DHCP server available in the Out-of-Band (OOB) management network to ensure the switch’s connection to the internet, you’ll need to configure the Management IP address, gateway, and nameservers using the following commands:

+
admin@sonic:~$ sonic-cli
+sonic# configure
+sonic(config-if-Management0)# interface Management 0
+sonic(config-if-Management0)# ip address <MGMT-IPv4>/<MGMT-SUBNET-MASK> gwaddr <MGMT-SUBNET-GATEWAY>
+sonic(config)# exit
+sonic(config)# ip name-server <DNS-SERVER1>
+sonic(config)# ip name-server <DNS-SERVER2>
+sonic(config)# end
+sonic# write memory
+sonic# exit
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent, copy the one-line installer command to your clipboard and paste in the switch.

+_images/Dell-Switch-agent-installation-Inventory.png +_images/Dell-Switch-agent-installation-oneliner.png +_images/Dell-Switch-agent-installation-cli.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/EdgeCore-SONiC-Switch-initial-setup.html b/en/latest/EdgeCore-SONiC-Switch-initial-setup.html new file mode 100644 index 0000000000..9efa55bb21 --- /dev/null +++ b/en/latest/EdgeCore-SONiC-Switch-initial-setup.html @@ -0,0 +1,921 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + EdgeCore SONiC Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

EdgeCore SONiC Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first.

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of EC SONiC.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/Edgecore-SONiC_20211125_074752_ec202012_227.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password:

+

admin/YourPaSsWoRd

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Disable ztp:

+
ztp disable -y
+
+
+

Configure the IP address, default gateway, and DNS to establish Internet connectivity via the management port.

+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of management network>
+
+
+
echo "nameserver <dns server>" > /etc/resolv.conf
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory.png +_images/Switch-agent-installation-oneliner.png +_images/Switch-agent-installation-cli.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/Nvidia-Cumulus-v3.7-Switch-initial-setup.html b/en/latest/Nvidia-Cumulus-v3.7-Switch-initial-setup.html new file mode 100644 index 0000000000..b910973b59 --- /dev/null +++ b/en/latest/Nvidia-Cumulus-v3.7-Switch-initial-setup.html @@ -0,0 +1,937 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Nvidia Cumulus v3.7 Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Nvidia Cumulus v3.7 Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first.

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/cumulus-linux-3.7.15-mlx-amd64.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password for Cumulus v3.7:

+

cumulus/CumulusLinux!

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Open the network interfaces file and add the IP address and other required details.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+       address <management IP address/prefix length>
+       gateway <gateway of management network>
+
+source /etc/network/interfaces.d/*
+
+
+
echo "nameserver <dns server>" | sudo tee /etc/resolv.conf
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Cumulus Linux license installation.

  2. +
+
sudo cl-license -i
+
+
+

Copy/paste the Cumulus Linux license string, then press ctrl-d.

+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory.png +_images/Switch-agent-installation-oneliner.png +_images/Switch-agent-installation-cli.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/Nvidia-Cumulus-v5-Switch-initial-setup.html b/en/latest/Nvidia-Cumulus-v5-Switch-initial-setup.html new file mode 100644 index 0000000000..57bb735f50 --- /dev/null +++ b/en/latest/Nvidia-Cumulus-v5-Switch-initial-setup.html @@ -0,0 +1,946 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Nvidia Cumulus v5 Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Nvidia Cumulus v5 Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/cumulus-linux-5.4.0-mlx-amd64.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password for Cumulus v5:

+

cumulus/cumulus

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Upon installation of Cumulus Linux v5, the default Virtual Routing and Forwarding (VRF) is set to ‘mgmt.’ To switch to the default VRF, please refer to the following instructions:

+

Disable ztp:

+
sudo ztp -d
+
+
+
sudo ip vrf exec default bash
+
+
+

Open the network interfaces file, add the IP address and other required details, and ensure that you remove the ‘mgmt’ VRF configuration:

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <management IP address/prefix length>
+        gateway <gateway of management network>
+
+source /etc/network/interfaces.d/*
+
+
+
echo "nameserver <dns server>" | sudo tee /etc/resolv.conf
+
+
+
sudo ifreload -a
+
+
+
+

Note

+

You might see a one-time warning in the output of ifreload, which you can ignore:

+
+
warning: mgmt: cmd '/usr/lib/vrf/vrf-helper delete mgmt 1001' failed: returned 1 (Failed to delete cgroup for vrf mgmt)
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory.png +_images/Switch-agent-installation-oneliner.png +_images/Switch-agent-installation-cli.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/Nvidia-Cumulus-v5.9+-Switch-initial-setup.html b/en/latest/Nvidia-Cumulus-v5.9+-Switch-initial-setup.html new file mode 100644 index 0000000000..29fcf134ed --- /dev/null +++ b/en/latest/Nvidia-Cumulus-v5.9+-Switch-initial-setup.html @@ -0,0 +1,928 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Nvidia Cumulus v5.9+ Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Nvidia Cumulus v5.9+ Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via the management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet.

+

Example:

+
onie-nos-install http://192.168.100.10/cumulus-linux-5.9.1-mlx-amd64.bin
+
+
+

After completion of the installation, the switch will automatically reboot.

+

To login use the default username and password for Cumulus v5.9:

+

cumulus/cumulus

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Upon installation of Cumulus Linux v5.9 or later ZTP must be disabled and internet connectivity must be provided to the switch via OOB management network:

+

Disable ztp:

+
sudo ztp -d
+
+
+

If there is no DHCP server in the OOB management network, then IP/Gateway/DNS information must be configured manually:

+
nv set interface eth0 ip address <IP-ADDRESS/MASK>
+nv set interface eth0 ip gateway <DEFAULT-GATEWAY>
+nv set service dns mgmt server <DNS-SERVER1>
+nv set service dns mgmt server <DNS-SERVER2>
+nv unset interface eth0 ip address dhcp
+nv config apply -y
+nv config save
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory-cl5.9.png +_images/Switch-agent-installation-oneliner-cl5.9.png +_images/Switch-agent-installation-cli-cl5.9.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
    +
  1. Netris agent connected to the controller.

  2. +
+_images/Switch-agent-installation-Inventory-online-cl5.9.png +

Once the switch is back, the Heartbeat status should be “OK” in the Netris controller.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/SoftGate-PRO-installation.html b/en/latest/SoftGate-PRO-installation.html new file mode 100644 index 0000000000..b0c77aded3 --- /dev/null +++ b/en/latest/SoftGate-PRO-installation.html @@ -0,0 +1,923 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate PRO Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • SoftGate PRO Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

SoftGate PRO Installation

+
+

Minimum Hardware Requirements

+
    +
  • 2 x Intel® Xeon® Silver Processor with 10 physical cores per socket (20 cores total)

  • +
  • 128 GB (64 GB RAM per socket) in multichannel configuration

  • +
  • 300 GB HDD

  • +
  • Nvidia Mellanox Connect-X 5/6 SmartNIC card

  • +
+
+
+

BIOS Configuration

+

The following are some recommendations for BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference:

+
    +
  • Before starting consider resetting all BIOS settings to their defaults

  • +
  • Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report

  • +
  • Select Performance as the CPU Power and Performance policy

  • +
  • Enable Turbo Boost

  • +
  • Set memory frequency to the highest available number, NOT auto

  • +
  • Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d

  • +
  • Disable Hyper-Threading

  • +
+
+
+

Install the Netris Agent

+

Requires freshly installed Ubuntu Linux 18.04 LTS and internet connectivity configured from netplan via management port.

+
    +
  1. Add the SoftGate in the controller Inventory or Topology section. Detailed configuration documentation is available here: “Adding SoftGates”.

  2. +
  3. Once the SoftGate is created, navigate to the Inventory section, click the three vertical dots (⋮) on the right side of the newly created SoftGate and select the Install Agent option.

  4. +
  5. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user.

  6. +
  7. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration).

  8. +
+
+

Note

+

If the Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured.

+
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node.
+    up ip route add <Controller address> via <Management network gateway>
+    gateway <Gateway IP address>
+
+ source /etc/network/interfaces.d/*
+
+
+
    +
  1. If everything seems ok, please remove/comment the Gateway line and save the file.

  2. +
+
+

Note

+

Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent.

+
+
    +
  1. Reboot the SoftGate

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/SoftGate-installation.html b/en/latest/SoftGate-installation.html new file mode 100644 index 0000000000..78b4563beb --- /dev/null +++ b/en/latest/SoftGate-installation.html @@ -0,0 +1,933 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • SoftGate Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

SoftGate Installation

+
+

Minimum Hardware Requirements

+
    +
  • 8 CPU cores

  • +
  • 16 GB RAM

  • +
  • 300 GB HDD

  • +
+
+
+

Provision Netris SoftGate software

+

Requires freshly installed Ubuntu Linux 22.04 LTS and internet connectivity.

+
    +
  1. Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: “Adding SoftGates”.

  2. +
  3. Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the SoftGate node you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

  4. +
+_images/softgate-install-agent.png +
    +
  1. Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node)

  2. +
+_images/softgate-provisioning-cli-output.png +
+

Note

+

Please note that Netris replaces Netplan with regular ifupdown and attempts to migrate any prior configuration to /etc/network/interfaces.

+
+
    +
  1. Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.<xyz>). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as you describe in /etc/network/interfaces.

  2. +
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# Physical port on SoftGate node connected to a TRUNK port of your network
+auto ens<X>
+iface ens<x> inet static
+    address 0.0.0.0/0
+
+# Optionally you can add more physical interfaces under your bond0
+auto ens<Y>
+iface ens<Y> inet static
+    address 0.0.0.0/0
+
+# Bond interface
+auto bond0
+iface bond0 inet static
+    address 0.0.0.0/0
+    # Please replace the ensX/Y with actual interface name(s) below to one(s) present in the OS.
+    bond-slaves ens<X> ens<Y>
+    # Optional, please adjust the bonding mode below according to the desired functionality.
+    bond-mode active-backup
+
+source /etc/network/interfaces.d/*
+
+
+
    +
  1. Ensure that SoftGate node will maintain IP connectivity with Netris Controller after reboot.

  2. +
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    # Please delete or comment the line below if Netris Controller is located in the same network with the SoftGate node.
+    up ip route add <Controller address> via <Management network gateway>
+
+
+
    +
  1. Reboot the SoftGate

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up, you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/Ubuntu-SwitchDev-Switch-initial-setup.html b/en/latest/Ubuntu-SwitchDev-Switch-initial-setup.html new file mode 100644 index 0000000000..0fc0468647 --- /dev/null +++ b/en/latest/Ubuntu-SwitchDev-Switch-initial-setup.html @@ -0,0 +1,926 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Ubuntu SwitchDev Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Ubuntu SwitchDev Switch Initial Setup

+
+

Note

+

Further installation requires a Console and Internet connectivity via management port!

+
+

If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first.

+
    +
  1. NOS Uninstall (if pre-installed)

  2. +
+

To uninstall the current NOS, access ONIE from the GRUB menu and select the Uninstall OS option.

+_images/uninstallOS.png +

Once it’s done, the switch will automatically reboot and get ready for the installation of the Ubuntu SwitchDev.

+
    +
  1. NOS Install

  2. +
+

If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually.

+
onie-discovery-stop
+
+
+
ip addr add <management IP address/prefix> dev eth0
+
+
+
ip route add default via <gateway of the management network>
+
+
+
echo "nameserver <DNS server address>" > /etc/resolv.conf
+
+
+

Install Ubuntu SwitchDev using the Netris customized image:

+
onie-nos-install http://downloads.netris.ai/netris-ubuntu-18.04.1.bin
+
+
+

Default username/password

+

netris/newNet0ps

+
    +
  1. Set up the Out-of-Band (OOB) Management.

  2. +
+

Open the network interfaces file and add the IP address and other required details.

+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <management IP address/prefix length>
+        gateway <gateway of management network>
+        dns-nameserver <dns server>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Netris agent installation.

  2. +
+

Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

+_images/Switch-agent-installation-Inventory-ubuntu.png +_images/Switch-agent-installation-oneliner-ubuntu.png +_images/Switch-agent-installation-cli-ubuntu.png +
    +
  1. Reboot the switch

  2. +
+
sudo reboot
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/_images/ACL_active.png b/en/latest/_images/ACL_active.png new file mode 100644 index 0000000000..6ba98a50b8 Binary files /dev/null and b/en/latest/_images/ACL_active.png differ diff --git a/en/latest/_images/ACL_approval.png b/en/latest/_images/ACL_approval.png new file mode 100644 index 0000000000..6345dea408 Binary files /dev/null and b/en/latest/_images/ACL_approval.png differ diff --git a/en/latest/_images/ACL_rule.png b/en/latest/_images/ACL_rule.png new file mode 100644 index 0000000000..81c08c11db Binary files /dev/null and b/en/latest/_images/ACL_rule.png differ diff --git a/en/latest/_images/Dell-Switch-agent-installation-Inventory.png b/en/latest/_images/Dell-Switch-agent-installation-Inventory.png new file mode 100644 index 0000000000..5fc7ae06f2 Binary files /dev/null and b/en/latest/_images/Dell-Switch-agent-installation-Inventory.png differ diff --git a/en/latest/_images/Dell-Switch-agent-installation-cli.png b/en/latest/_images/Dell-Switch-agent-installation-cli.png new file mode 100644 index 0000000000..4226c7a74d Binary files /dev/null and b/en/latest/_images/Dell-Switch-agent-installation-cli.png differ diff --git a/en/latest/_images/Dell-Switch-agent-installation-oneliner.png b/en/latest/_images/Dell-Switch-agent-installation-oneliner.png new file mode 100644 index 0000000000..23838d0a19 Binary files /dev/null and b/en/latest/_images/Dell-Switch-agent-installation-oneliner.png differ diff --git a/en/latest/_images/Global-settings-edit.png b/en/latest/_images/Global-settings-edit.png new file mode 100644 index 0000000000..705fdac1a3 Binary files /dev/null and b/en/latest/_images/Global-settings-edit.png differ diff --git a/en/latest/_images/Global-settings-save.png b/en/latest/_images/Global-settings-save.png new file mode 100644 index 0000000000..8456554cfe Binary files /dev/null and b/en/latest/_images/Global-settings-save.png differ diff --git a/en/latest/_images/ROH-instance.png b/en/latest/_images/ROH-instance.png new file mode 100644 index 0000000000..ca21d16b10 Binary files /dev/null and b/en/latest/_images/ROH-instance.png differ diff --git a/en/latest/_images/ROH-listing.png b/en/latest/_images/ROH-listing.png new file mode 100644 index 0000000000..0e3c74214c Binary files /dev/null and b/en/latest/_images/ROH-listing.png differ diff --git a/en/latest/_images/SiteMesh_listing.png b/en/latest/_images/SiteMesh_listing.png new file mode 100644 index 0000000000..7142f87e52 Binary files /dev/null and b/en/latest/_images/SiteMesh_listing.png differ diff --git a/en/latest/_images/SiteMesh_modes.png b/en/latest/_images/SiteMesh_modes.png new file mode 100644 index 0000000000..5edfe756f0 Binary files /dev/null and b/en/latest/_images/SiteMesh_modes.png differ diff --git a/en/latest/_images/Switch-agent-installation-Inventory-cl5.9.png b/en/latest/_images/Switch-agent-installation-Inventory-cl5.9.png new file mode 100644 index 0000000000..c9d9980dd0 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-Inventory-cl5.9.png differ diff --git a/en/latest/_images/Switch-agent-installation-Inventory-online-cl5.9.png b/en/latest/_images/Switch-agent-installation-Inventory-online-cl5.9.png new file mode 100644 index 0000000000..920a96c954 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-Inventory-online-cl5.9.png differ diff --git a/en/latest/_images/Switch-agent-installation-Inventory-ubuntu.png b/en/latest/_images/Switch-agent-installation-Inventory-ubuntu.png new file mode 100644 index 0000000000..17818408a8 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-Inventory-ubuntu.png differ diff --git a/en/latest/_images/Switch-agent-installation-Inventory.png b/en/latest/_images/Switch-agent-installation-Inventory.png new file mode 100644 index 0000000000..cf99dd239a Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-Inventory.png differ diff --git a/en/latest/_images/Switch-agent-installation-cli-cl5.9.png b/en/latest/_images/Switch-agent-installation-cli-cl5.9.png new file mode 100644 index 0000000000..68e903dcb1 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-cli-cl5.9.png differ diff --git a/en/latest/_images/Switch-agent-installation-cli-ubuntu.png b/en/latest/_images/Switch-agent-installation-cli-ubuntu.png new file mode 100644 index 0000000000..366ddaa85e Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-cli-ubuntu.png differ diff --git a/en/latest/_images/Switch-agent-installation-cli.png b/en/latest/_images/Switch-agent-installation-cli.png new file mode 100644 index 0000000000..da9b8b4ff1 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-cli.png differ diff --git a/en/latest/_images/Switch-agent-installation-oneliner-cl5.9.png b/en/latest/_images/Switch-agent-installation-oneliner-cl5.9.png new file mode 100644 index 0000000000..e6b160bec9 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-oneliner-cl5.9.png differ diff --git a/en/latest/_images/Switch-agent-installation-oneliner-ubuntu.png b/en/latest/_images/Switch-agent-installation-oneliner-ubuntu.png new file mode 100644 index 0000000000..ce08901584 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-oneliner-ubuntu.png differ diff --git a/en/latest/_images/Switch-agent-installation-oneliner.png b/en/latest/_images/Switch-agent-installation-oneliner.png new file mode 100644 index 0000000000..aae208da96 Binary files /dev/null and b/en/latest/_images/Switch-agent-installation-oneliner.png differ diff --git a/en/latest/_images/TCAM.png b/en/latest/_images/TCAM.png new file mode 100644 index 0000000000..0bba84f117 Binary files /dev/null and b/en/latest/_images/TCAM.png differ diff --git a/en/latest/_images/action_permit.png b/en/latest/_images/action_permit.png new file mode 100644 index 0000000000..c478300133 Binary files /dev/null and b/en/latest/_images/action_permit.png differ diff --git a/en/latest/_images/add-allocation.png b/en/latest/_images/add-allocation.png new file mode 100644 index 0000000000..0d4784990e Binary files /dev/null and b/en/latest/_images/add-allocation.png differ diff --git a/en/latest/_images/add-equinix-bgp-primary.png b/en/latest/_images/add-equinix-bgp-primary.png new file mode 100644 index 0000000000..2b343ada9b Binary files /dev/null and b/en/latest/_images/add-equinix-bgp-primary.png differ diff --git a/en/latest/_images/add-interconnection-request.png b/en/latest/_images/add-interconnection-request.png new file mode 100644 index 0000000000..de0be61993 Binary files /dev/null and b/en/latest/_images/add-interconnection-request.png differ diff --git a/en/latest/_images/add-l3-lb.png b/en/latest/_images/add-l3-lb.png new file mode 100644 index 0000000000..6760093ee5 Binary files /dev/null and b/en/latest/_images/add-l3-lb.png differ diff --git a/en/latest/_images/add-new-ebgp-form.png b/en/latest/_images/add-new-ebgp-form.png new file mode 100644 index 0000000000..8b771a90c8 Binary files /dev/null and b/en/latest/_images/add-new-ebgp-form.png differ diff --git a/en/latest/_images/add-new-hardware.png b/en/latest/_images/add-new-hardware.png new file mode 100644 index 0000000000..5e00df64a3 Binary files /dev/null and b/en/latest/_images/add-new-hardware.png differ diff --git a/en/latest/_images/add-new-vlan-equinix.png b/en/latest/_images/add-new-vlan-equinix.png new file mode 100644 index 0000000000..241878136e Binary files /dev/null and b/en/latest/_images/add-new-vlan-equinix.png differ diff --git a/en/latest/_images/add-softgate.png b/en/latest/_images/add-softgate.png new file mode 100644 index 0000000000..0ea2b859ac Binary files /dev/null and b/en/latest/_images/add-softgate.png differ diff --git a/en/latest/_images/add-subnet.png b/en/latest/_images/add-subnet.png new file mode 100644 index 0000000000..54586a7099 Binary files /dev/null and b/en/latest/_images/add-subnet.png differ diff --git a/en/latest/_images/add-to-lag-port.png b/en/latest/_images/add-to-lag-port.png new file mode 100644 index 0000000000..479e36897e Binary files /dev/null and b/en/latest/_images/add-to-lag-port.png differ diff --git a/en/latest/_images/add-vnet.png b/en/latest/_images/add-vnet.png new file mode 100644 index 0000000000..4321573d7c Binary files /dev/null and b/en/latest/_images/add-vnet.png differ diff --git a/en/latest/_images/add_board.png b/en/latest/_images/add_board.png new file mode 100644 index 0000000000..377c7bba93 Binary files /dev/null and b/en/latest/_images/add_board.png differ diff --git a/en/latest/_images/add_graph.png b/en/latest/_images/add_graph.png new file mode 100644 index 0000000000..8a0907edc3 Binary files /dev/null and b/en/latest/_images/add_graph.png differ diff --git a/en/latest/_images/add_graph2.png b/en/latest/_images/add_graph2.png new file mode 100644 index 0000000000..2cbeb00c24 Binary files /dev/null and b/en/latest/_images/add_graph2.png differ diff --git a/en/latest/_images/add_perm_group.png b/en/latest/_images/add_perm_group.png new file mode 100644 index 0000000000..deac4fc1c0 Binary files /dev/null and b/en/latest/_images/add_perm_group.png differ diff --git a/en/latest/_images/add_tenant.png b/en/latest/_images/add_tenant.png new file mode 100644 index 0000000000..e2af3665db Binary files /dev/null and b/en/latest/_images/add_tenant.png differ diff --git a/en/latest/_images/add_user.png b/en/latest/_images/add_user.png new file mode 100644 index 0000000000..2c7c916c3f Binary files /dev/null and b/en/latest/_images/add_user.png differ diff --git a/en/latest/_images/add_user_role.png b/en/latest/_images/add_user_role.png new file mode 100644 index 0000000000..cd3245890c Binary files /dev/null and b/en/latest/_images/add_user_role.png differ diff --git a/en/latest/_images/allocation_empty.png b/en/latest/_images/allocation_empty.png new file mode 100644 index 0000000000..2115fc59bd Binary files /dev/null and b/en/latest/_images/allocation_empty.png differ diff --git a/en/latest/_images/approve_reject.png b/en/latest/_images/approve_reject.png new file mode 100644 index 0000000000..a3054f9912 Binary files /dev/null and b/en/latest/_images/approve_reject.png differ diff --git a/en/latest/_images/aws-concept-traffic-flows.png b/en/latest/_images/aws-concept-traffic-flows.png new file mode 100644 index 0000000000..6f29b595ca Binary files /dev/null and b/en/latest/_images/aws-concept-traffic-flows.png differ diff --git a/en/latest/_images/aws-ec2-stop-fwd-check.png b/en/latest/_images/aws-ec2-stop-fwd-check.png new file mode 100644 index 0000000000..773981a438 Binary files /dev/null and b/en/latest/_images/aws-ec2-stop-fwd-check.png differ diff --git a/en/latest/_images/aws-netris-create-sg.png b/en/latest/_images/aws-netris-create-sg.png new file mode 100644 index 0000000000..aa8a2bf59e Binary files /dev/null and b/en/latest/_images/aws-netris-create-sg.png differ diff --git a/en/latest/_images/aws-netris-enable-site-mesh.png b/en/latest/_images/aws-netris-enable-site-mesh.png new file mode 100644 index 0000000000..7b72dc4012 Binary files /dev/null and b/en/latest/_images/aws-netris-enable-site-mesh.png differ diff --git a/en/latest/_images/aws-netris-ipam-lo.png b/en/latest/_images/aws-netris-ipam-lo.png new file mode 100644 index 0000000000..4ee3b3ed88 Binary files /dev/null and b/en/latest/_images/aws-netris-ipam-lo.png differ diff --git a/en/latest/_images/aws-netris-provision-sg.png b/en/latest/_images/aws-netris-provision-sg.png new file mode 100644 index 0000000000..71d018f61d Binary files /dev/null and b/en/latest/_images/aws-netris-provision-sg.png differ diff --git a/en/latest/_images/aws-netris-site-create.png b/en/latest/_images/aws-netris-site-create.png new file mode 100644 index 0000000000..641f1ef4e2 Binary files /dev/null and b/en/latest/_images/aws-netris-site-create.png differ diff --git a/en/latest/_images/aws-netris-site-mesh-status.png b/en/latest/_images/aws-netris-site-mesh-status.png new file mode 100644 index 0000000000..03d25a1383 Binary files /dev/null and b/en/latest/_images/aws-netris-site-mesh-status.png differ diff --git a/en/latest/_images/aws-netris-static-route.png b/en/latest/_images/aws-netris-static-route.png new file mode 100644 index 0000000000..97bee607d2 Binary files /dev/null and b/en/latest/_images/aws-netris-static-route.png differ diff --git a/en/latest/_images/aws-security-group.png b/en/latest/_images/aws-security-group.png new file mode 100644 index 0000000000..ea71ffdef6 Binary files /dev/null and b/en/latest/_images/aws-security-group.png differ diff --git a/en/latest/_images/aws-softgate-deployed.png b/en/latest/_images/aws-softgate-deployed.png new file mode 100644 index 0000000000..0ae61a94f7 Binary files /dev/null and b/en/latest/_images/aws-softgate-deployed.png differ diff --git a/en/latest/_images/aws-vpc-cidr-to-netris.png b/en/latest/_images/aws-vpc-cidr-to-netris.png new file mode 100644 index 0000000000..31849fd37b Binary files /dev/null and b/en/latest/_images/aws-vpc-cidr-to-netris.png differ diff --git a/en/latest/_images/aws-vpc-routes-created.png b/en/latest/_images/aws-vpc-routes-created.png new file mode 100644 index 0000000000..b355e602ab Binary files /dev/null and b/en/latest/_images/aws-vpc-routes-created.png differ diff --git a/en/latest/_images/bgp-listing.png b/en/latest/_images/bgp-listing.png new file mode 100644 index 0000000000..2e06af3d0d Binary files /dev/null and b/en/latest/_images/bgp-listing.png differ diff --git a/en/latest/_images/bgp-looking-glass.png b/en/latest/_images/bgp-looking-glass.png new file mode 100644 index 0000000000..b2cef4aaca Binary files /dev/null and b/en/latest/_images/bgp-looking-glass.png differ diff --git a/en/latest/_images/bgp_empty.png b/en/latest/_images/bgp_empty.png new file mode 100644 index 0000000000..9e5243f510 Binary files /dev/null and b/en/latest/_images/bgp_empty.png differ diff --git a/en/latest/_images/bgp_status.png b/en/latest/_images/bgp_status.png new file mode 100644 index 0000000000..d297a399d7 Binary files /dev/null and b/en/latest/_images/bgp_status.png differ diff --git a/en/latest/_images/change-password.png b/en/latest/_images/change-password.png new file mode 100644 index 0000000000..ad093aa331 Binary files /dev/null and b/en/latest/_images/change-password.png differ diff --git a/en/latest/_images/cloudflare-dns-record.png b/en/latest/_images/cloudflare-dns-record.png new file mode 100644 index 0000000000..a0d7586c2b Binary files /dev/null and b/en/latest/_images/cloudflare-dns-record.png differ diff --git a/en/latest/_images/community.png b/en/latest/_images/community.png new file mode 100644 index 0000000000..26cc83495c Binary files /dev/null and b/en/latest/_images/community.png differ diff --git a/en/latest/_images/create-new-admin-user.png b/en/latest/_images/create-new-admin-user.png new file mode 100644 index 0000000000..4dcde0ba0b Binary files /dev/null and b/en/latest/_images/create-new-admin-user.png differ diff --git a/en/latest/_images/create_bgp.png b/en/latest/_images/create_bgp.png new file mode 100644 index 0000000000..6a697e4bfe Binary files /dev/null and b/en/latest/_images/create_bgp.png differ diff --git a/en/latest/_images/create_board.png b/en/latest/_images/create_board.png new file mode 100644 index 0000000000..7f55a8952c Binary files /dev/null and b/en/latest/_images/create_board.png differ diff --git a/en/latest/_images/create_dnat_empty.png b/en/latest/_images/create_dnat_empty.png new file mode 100644 index 0000000000..8e4313d507 Binary files /dev/null and b/en/latest/_images/create_dnat_empty.png differ diff --git a/en/latest/_images/create_link.png b/en/latest/_images/create_link.png new file mode 100644 index 0000000000..e951e381a0 Binary files /dev/null and b/en/latest/_images/create_link.png differ diff --git a/en/latest/_images/create_snat_empty.png b/en/latest/_images/create_snat_empty.png new file mode 100644 index 0000000000..61c0fe864b Binary files /dev/null and b/en/latest/_images/create_snat_empty.png differ diff --git a/en/latest/_images/credentials.png b/en/latest/_images/credentials.png new file mode 100644 index 0000000000..edbc78148f Binary files /dev/null and b/en/latest/_images/credentials.png differ diff --git a/en/latest/_images/diagrams_terraform.png b/en/latest/_images/diagrams_terraform.png new file mode 100644 index 0000000000..33f33920f0 Binary files /dev/null and b/en/latest/_images/diagrams_terraform.png differ diff --git a/en/latest/_images/dnat_add.png b/en/latest/_images/dnat_add.png new file mode 100644 index 0000000000..45df0247f5 Binary files /dev/null and b/en/latest/_images/dnat_add.png differ diff --git a/en/latest/_images/dns-cloudflare-equinix-ip.png b/en/latest/_images/dns-cloudflare-equinix-ip.png new file mode 100644 index 0000000000..212d1af7d0 Binary files /dev/null and b/en/latest/_images/dns-cloudflare-equinix-ip.png differ diff --git a/en/latest/_images/dns-record-netrisctl.png b/en/latest/_images/dns-record-netrisctl.png new file mode 100644 index 0000000000..1d04c129b5 Binary files /dev/null and b/en/latest/_images/dns-record-netrisctl.png differ diff --git a/en/latest/_images/edit-port.png b/en/latest/_images/edit-port.png new file mode 100644 index 0000000000..eed82b6388 Binary files /dev/null and b/en/latest/_images/edit-port.png differ diff --git a/en/latest/_images/equinix-ebgp-links-up.png b/en/latest/_images/equinix-ebgp-links-up.png new file mode 100644 index 0000000000..decce718f9 Binary files /dev/null and b/en/latest/_images/equinix-ebgp-links-up.png differ diff --git a/en/latest/_images/equinix-metal-activate-bgp.png b/en/latest/_images/equinix-metal-activate-bgp.png new file mode 100644 index 0000000000..947f999389 Binary files /dev/null and b/en/latest/_images/equinix-metal-activate-bgp.png differ diff --git a/en/latest/_images/equinix-metal-bgp-diagram.png b/en/latest/_images/equinix-metal-bgp-diagram.png new file mode 100644 index 0000000000..50080c16ad Binary files /dev/null and b/en/latest/_images/equinix-metal-bgp-diagram.png differ diff --git a/en/latest/_images/equinix-metal-netris-bgp-up.png b/en/latest/_images/equinix-metal-netris-bgp-up.png new file mode 100644 index 0000000000..8c993d3a97 Binary files /dev/null and b/en/latest/_images/equinix-metal-netris-bgp-up.png differ diff --git a/en/latest/_images/equinix-metal-netris-ipam-synced.png b/en/latest/_images/equinix-metal-netris-ipam-synced.png new file mode 100644 index 0000000000..7095db4a4c Binary files /dev/null and b/en/latest/_images/equinix-metal-netris-ipam-synced.png differ diff --git a/en/latest/_images/equinix-metal-project-api-keys.png b/en/latest/_images/equinix-metal-project-api-keys.png new file mode 100644 index 0000000000..84182fb370 Binary files /dev/null and b/en/latest/_images/equinix-metal-project-api-keys.png differ diff --git a/en/latest/_images/equinix-metal-project-id.png b/en/latest/_images/equinix-metal-project-id.png new file mode 100644 index 0000000000..4a89f373a3 Binary files /dev/null and b/en/latest/_images/equinix-metal-project-id.png differ diff --git a/en/latest/_images/equinix-metal-request-ip-block.png b/en/latest/_images/equinix-metal-request-ip-block.png new file mode 100644 index 0000000000..97db2afc16 Binary files /dev/null and b/en/latest/_images/equinix-metal-request-ip-block.png differ diff --git a/en/latest/_images/equinix-metal-vnet-with-tag-terraform.png b/en/latest/_images/equinix-metal-vnet-with-tag-terraform.png new file mode 100644 index 0000000000..e40e8c93ce Binary files /dev/null and b/en/latest/_images/equinix-metal-vnet-with-tag-terraform.png differ diff --git a/en/latest/_images/equinix-metal-vnet-with-tag.png b/en/latest/_images/equinix-metal-vnet-with-tag.png new file mode 100644 index 0000000000..d2b053476c Binary files /dev/null and b/en/latest/_images/equinix-metal-vnet-with-tag.png differ diff --git a/en/latest/_images/equinix-request-c3-small-server.png b/en/latest/_images/equinix-request-c3-small-server.png new file mode 100644 index 0000000000..3b42bced8a Binary files /dev/null and b/en/latest/_images/equinix-request-c3-small-server.png differ diff --git a/en/latest/_images/gcp-concept-traffic-flows.png b/en/latest/_images/gcp-concept-traffic-flows.png new file mode 100644 index 0000000000..cc8e5a03a0 Binary files /dev/null and b/en/latest/_images/gcp-concept-traffic-flows.png differ diff --git a/en/latest/_images/gcp-firewall-rule.png b/en/latest/_images/gcp-firewall-rule.png new file mode 100644 index 0000000000..8b7c9585a0 Binary files /dev/null and b/en/latest/_images/gcp-firewall-rule.png differ diff --git a/en/latest/_images/gcp-netris-create-sg.png b/en/latest/_images/gcp-netris-create-sg.png new file mode 100644 index 0000000000..e9b8e65736 Binary files /dev/null and b/en/latest/_images/gcp-netris-create-sg.png differ diff --git a/en/latest/_images/gcp-netris-enable-site-mesh.png b/en/latest/_images/gcp-netris-enable-site-mesh.png new file mode 100644 index 0000000000..9b7c2b6086 Binary files /dev/null and b/en/latest/_images/gcp-netris-enable-site-mesh.png differ diff --git a/en/latest/_images/gcp-netris-ipam-lo.png b/en/latest/_images/gcp-netris-ipam-lo.png new file mode 100644 index 0000000000..06bfeddbb2 Binary files /dev/null and b/en/latest/_images/gcp-netris-ipam-lo.png differ diff --git a/en/latest/_images/gcp-netris-provision-sg.png b/en/latest/_images/gcp-netris-provision-sg.png new file mode 100644 index 0000000000..8c73863859 Binary files /dev/null and b/en/latest/_images/gcp-netris-provision-sg.png differ diff --git a/en/latest/_images/gcp-netris-site-create.png b/en/latest/_images/gcp-netris-site-create.png new file mode 100644 index 0000000000..19b2f73242 Binary files /dev/null and b/en/latest/_images/gcp-netris-site-create.png differ diff --git a/en/latest/_images/gcp-netris-site-mesh-status.png b/en/latest/_images/gcp-netris-site-mesh-status.png new file mode 100644 index 0000000000..6f23791ec7 Binary files /dev/null and b/en/latest/_images/gcp-netris-site-mesh-status.png differ diff --git a/en/latest/_images/gcp-netris-static-route.png b/en/latest/_images/gcp-netris-static-route.png new file mode 100644 index 0000000000..e2ec8aa8b4 Binary files /dev/null and b/en/latest/_images/gcp-netris-static-route.png differ diff --git a/en/latest/_images/gcp-softgate-deployed.png b/en/latest/_images/gcp-softgate-deployed.png new file mode 100644 index 0000000000..1db913283e Binary files /dev/null and b/en/latest/_images/gcp-softgate-deployed.png differ diff --git a/en/latest/_images/gcp-vpc-routes-created.png b/en/latest/_images/gcp-vpc-routes-created.png new file mode 100644 index 0000000000..f574abbf27 Binary files /dev/null and b/en/latest/_images/gcp-vpc-routes-created.png differ diff --git a/en/latest/_images/gcp-vpc-subnet-to-netris.png b/en/latest/_images/gcp-vpc-subnet-to-netris.png new file mode 100644 index 0000000000..db551ea3f1 Binary files /dev/null and b/en/latest/_images/gcp-vpc-subnet-to-netris.png differ diff --git a/en/latest/_images/graph_board.png b/en/latest/_images/graph_board.png new file mode 100644 index 0000000000..0d24f70a37 Binary files /dev/null and b/en/latest/_images/graph_board.png differ diff --git a/en/latest/_images/hardware_health.png b/en/latest/_images/hardware_health.png new file mode 100644 index 0000000000..df70fc2ecc Binary files /dev/null and b/en/latest/_images/hardware_health.png differ diff --git a/en/latest/_images/install_agent.gif b/en/latest/_images/install_agent.gif new file mode 100644 index 0000000000..0928829c40 Binary files /dev/null and b/en/latest/_images/install_agent.gif differ diff --git a/en/latest/_images/inventory-listing.png b/en/latest/_images/inventory-listing.png new file mode 100644 index 0000000000..caa35c8f66 Binary files /dev/null and b/en/latest/_images/inventory-listing.png differ diff --git a/en/latest/_images/inventory-profile.png b/en/latest/_images/inventory-profile.png new file mode 100644 index 0000000000..e58ef725b7 Binary files /dev/null and b/en/latest/_images/inventory-profile.png differ diff --git a/en/latest/_images/inventory_profile_custom.png b/en/latest/_images/inventory_profile_custom.png new file mode 100644 index 0000000000..a948d9bf71 Binary files /dev/null and b/en/latest/_images/inventory_profile_custom.png differ diff --git a/en/latest/_images/inventory_softgate.png b/en/latest/_images/inventory_softgate.png new file mode 100644 index 0000000000..9700b0da5e Binary files /dev/null and b/en/latest/_images/inventory_softgate.png differ diff --git a/en/latest/_images/inventory_switch.png b/en/latest/_images/inventory_switch.png new file mode 100644 index 0000000000..6ff3b733bb Binary files /dev/null and b/en/latest/_images/inventory_switch.png differ diff --git a/en/latest/_images/ipam_allocation.png b/en/latest/_images/ipam_allocation.png new file mode 100644 index 0000000000..b43bf76cac Binary files /dev/null and b/en/latest/_images/ipam_allocation.png differ diff --git a/en/latest/_images/ipam_common_subnet.png b/en/latest/_images/ipam_common_subnet.png new file mode 100644 index 0000000000..039074c1c2 Binary files /dev/null and b/en/latest/_images/ipam_common_subnet.png differ diff --git a/en/latest/_images/ipam_l4lb_subnet.png b/en/latest/_images/ipam_l4lb_subnet.png new file mode 100644 index 0000000000..190b62ac70 Binary files /dev/null and b/en/latest/_images/ipam_l4lb_subnet.png differ diff --git a/en/latest/_images/ipam_loopback_subnet.png b/en/latest/_images/ipam_loopback_subnet.png new file mode 100644 index 0000000000..b7ad1730dd Binary files /dev/null and b/en/latest/_images/ipam_loopback_subnet.png differ diff --git a/en/latest/_images/ipam_mgmt_subnet.png b/en/latest/_images/ipam_mgmt_subnet.png new file mode 100644 index 0000000000..b3a83f00af Binary files /dev/null and b/en/latest/_images/ipam_mgmt_subnet.png differ diff --git a/en/latest/_images/ipam_nat_subnet.png b/en/latest/_images/ipam_nat_subnet.png new file mode 100644 index 0000000000..501198a966 Binary files /dev/null and b/en/latest/_images/ipam_nat_subnet.png differ diff --git a/en/latest/_images/ipam_tree_new.png b/en/latest/_images/ipam_tree_new.png new file mode 100644 index 0000000000..6f86d4da00 Binary files /dev/null and b/en/latest/_images/ipam_tree_new.png differ diff --git a/en/latest/_images/ipv4_prefix.png b/en/latest/_images/ipv4_prefix.png new file mode 100644 index 0000000000..f8f1e56fb7 Binary files /dev/null and b/en/latest/_images/ipv4_prefix.png differ diff --git a/en/latest/_images/ipv6_prefix.png b/en/latest/_images/ipv6_prefix.png new file mode 100644 index 0000000000..bf1d73881c Binary files /dev/null and b/en/latest/_images/ipv6_prefix.png differ diff --git a/en/latest/_images/l3lb_srv01.png b/en/latest/_images/l3lb_srv01.png new file mode 100644 index 0000000000..f2fe0bf3bb Binary files /dev/null and b/en/latest/_images/l3lb_srv01.png differ diff --git a/en/latest/_images/l3lb_srv02.png b/en/latest/_images/l3lb_srv02.png new file mode 100644 index 0000000000..6d3a5bd423 Binary files /dev/null and b/en/latest/_images/l3lb_srv02.png differ diff --git a/en/latest/_images/l4lb_create.png b/en/latest/_images/l4lb_create.png new file mode 100644 index 0000000000..693e6eb92e Binary files /dev/null and b/en/latest/_images/l4lb_create.png differ diff --git a/en/latest/_images/lag_add_lag.png b/en/latest/_images/lag_add_lag.png new file mode 100644 index 0000000000..f574821122 Binary files /dev/null and b/en/latest/_images/lag_add_lag.png differ diff --git a/en/latest/_images/lag_add_lag2.png b/en/latest/_images/lag_add_lag2.png new file mode 100644 index 0000000000..e6893eff67 Binary files /dev/null and b/en/latest/_images/lag_add_lag2.png differ diff --git a/en/latest/_images/lag_add_lag3.png b/en/latest/_images/lag_add_lag3.png new file mode 100644 index 0000000000..e447e21eb3 Binary files /dev/null and b/en/latest/_images/lag_add_lag3.png differ diff --git a/en/latest/_images/lag_add_vnet.png b/en/latest/_images/lag_add_vnet.png new file mode 100644 index 0000000000..4087b036a7 Binary files /dev/null and b/en/latest/_images/lag_add_vnet.png differ diff --git a/en/latest/_images/lag_diagram.png b/en/latest/_images/lag_diagram.png new file mode 100644 index 0000000000..021004d44d Binary files /dev/null and b/en/latest/_images/lag_diagram.png differ diff --git a/en/latest/_images/lag_diagram2.png b/en/latest/_images/lag_diagram2.png new file mode 100644 index 0000000000..6714a4ba24 Binary files /dev/null and b/en/latest/_images/lag_diagram2.png differ diff --git a/en/latest/_images/lag_diagram3.png b/en/latest/_images/lag_diagram3.png new file mode 100644 index 0000000000..d6df0d3450 Binary files /dev/null and b/en/latest/_images/lag_diagram3.png differ diff --git a/en/latest/_images/lg_bgp_route.png b/en/latest/_images/lg_bgp_route.png new file mode 100644 index 0000000000..c264aabc90 Binary files /dev/null and b/en/latest/_images/lg_bgp_route.png differ diff --git a/en/latest/_images/lg_mac.png b/en/latest/_images/lg_mac.png new file mode 100644 index 0000000000..78e1b88aab Binary files /dev/null and b/en/latest/_images/lg_mac.png differ diff --git a/en/latest/_images/lg_ping.png b/en/latest/_images/lg_ping.png new file mode 100644 index 0000000000..26b68af946 Binary files /dev/null and b/en/latest/_images/lg_ping.png differ diff --git a/en/latest/_images/lg_rd.png b/en/latest/_images/lg_rd.png new file mode 100644 index 0000000000..dd3fa8046b Binary files /dev/null and b/en/latest/_images/lg_rd.png differ diff --git a/en/latest/_images/lg_summary.png b/en/latest/_images/lg_summary.png new file mode 100644 index 0000000000..443cf355ea Binary files /dev/null and b/en/latest/_images/lg_summary.png differ diff --git a/en/latest/_images/list-l3-lb-detail.png b/en/latest/_images/list-l3-lb-detail.png new file mode 100644 index 0000000000..d9a57ae5f1 Binary files /dev/null and b/en/latest/_images/list-l3-lb-detail.png differ diff --git a/en/latest/_images/list-l3-lb.png b/en/latest/_images/list-l3-lb.png new file mode 100644 index 0000000000..5f33c2ad02 Binary files /dev/null and b/en/latest/_images/list-l3-lb.png differ diff --git a/en/latest/_images/list-l4-load-balancers.png b/en/latest/_images/list-l4-load-balancers.png new file mode 100644 index 0000000000..b100218c79 Binary files /dev/null and b/en/latest/_images/list-l4-load-balancers.png differ diff --git a/en/latest/_images/list-subnets.png b/en/latest/_images/list-subnets.png new file mode 100644 index 0000000000..a3e9d60c73 Binary files /dev/null and b/en/latest/_images/list-subnets.png differ diff --git a/en/latest/_images/list-vnet-expanded.png b/en/latest/_images/list-vnet-expanded.png new file mode 100644 index 0000000000..fdc5f16998 Binary files /dev/null and b/en/latest/_images/list-vnet-expanded.png differ diff --git a/en/latest/_images/list-vnet.png b/en/latest/_images/list-vnet.png new file mode 100644 index 0000000000..2ad4b6fa49 Binary files /dev/null and b/en/latest/_images/list-vnet.png differ diff --git a/en/latest/_images/local-public-asn.png b/en/latest/_images/local-public-asn.png new file mode 100644 index 0000000000..22ffb71e17 Binary files /dev/null and b/en/latest/_images/local-public-asn.png differ diff --git a/en/latest/_images/maintenance-mode.png b/en/latest/_images/maintenance-mode.png new file mode 100644 index 0000000000..545166b753 Binary files /dev/null and b/en/latest/_images/maintenance-mode.png differ diff --git a/en/latest/_images/manage-vnet.gif b/en/latest/_images/manage-vnet.gif new file mode 100644 index 0000000000..f8e7e8620c Binary files /dev/null and b/en/latest/_images/manage-vnet.gif differ diff --git a/en/latest/_images/nat_subnet_empty.png b/en/latest/_images/nat_subnet_empty.png new file mode 100644 index 0000000000..01c3bfc8df Binary files /dev/null and b/en/latest/_images/nat_subnet_empty.png differ diff --git a/en/latest/_images/netris-controller-installed.png b/en/latest/_images/netris-controller-installed.png new file mode 100644 index 0000000000..a6851e2b71 Binary files /dev/null and b/en/latest/_images/netris-controller-installed.png differ diff --git a/en/latest/_images/netris-create-common-subnets.png b/en/latest/_images/netris-create-common-subnets.png new file mode 100644 index 0000000000..b47c8e952c Binary files /dev/null and b/en/latest/_images/netris-create-common-subnets.png differ diff --git a/en/latest/_images/netris-create-equinix-metal-site.png b/en/latest/_images/netris-create-equinix-metal-site.png new file mode 100644 index 0000000000..39734075f1 Binary files /dev/null and b/en/latest/_images/netris-create-equinix-metal-site.png differ diff --git a/en/latest/_images/netris-create-nat-rule.png b/en/latest/_images/netris-create-nat-rule.png new file mode 100644 index 0000000000..28c5913965 Binary files /dev/null and b/en/latest/_images/netris-create-nat-rule.png differ diff --git a/en/latest/_images/netris-creating-vnet-for-equinix-metal.png b/en/latest/_images/netris-creating-vnet-for-equinix-metal.png new file mode 100644 index 0000000000..98fdbd910d Binary files /dev/null and b/en/latest/_images/netris-creating-vnet-for-equinix-metal.png differ diff --git a/en/latest/_images/netris-enable-elb.png b/en/latest/_images/netris-enable-elb.png new file mode 100644 index 0000000000..36e7566443 Binary files /dev/null and b/en/latest/_images/netris-enable-elb.png differ diff --git a/en/latest/_images/netris-ipam-nat.png b/en/latest/_images/netris-ipam-nat.png new file mode 100644 index 0000000000..d7ff2449c7 Binary files /dev/null and b/en/latest/_images/netris-ipam-nat.png differ diff --git a/en/latest/_images/netris-l4-load-balancer.png b/en/latest/_images/netris-l4-load-balancer.png new file mode 100644 index 0000000000..e6e466f622 Binary files /dev/null and b/en/latest/_images/netris-l4-load-balancer.png differ diff --git a/en/latest/_images/netris-vnet-ready-in-equinix-metal.png b/en/latest/_images/netris-vnet-ready-in-equinix-metal.png new file mode 100644 index 0000000000..c537e657c6 Binary files /dev/null and b/en/latest/_images/netris-vnet-ready-in-equinix-metal.png differ diff --git a/en/latest/_images/netris_controller_diagram.png b/en/latest/_images/netris_controller_diagram.png new file mode 100644 index 0000000000..610a8ae396 Binary files /dev/null and b/en/latest/_images/netris_controller_diagram.png differ diff --git a/en/latest/_images/netris_version_example.png b/en/latest/_images/netris_version_example.png new file mode 100644 index 0000000000..55e23d8df9 Binary files /dev/null and b/en/latest/_images/netris_version_example.png differ diff --git a/en/latest/_images/oneliner-from-local-repo.png b/en/latest/_images/oneliner-from-local-repo.png new file mode 100644 index 0000000000..68bfabd684 Binary files /dev/null and b/en/latest/_images/oneliner-from-local-repo.png differ diff --git a/en/latest/_images/phoenixnap-api-credential.png b/en/latest/_images/phoenixnap-api-credential.png new file mode 100644 index 0000000000..23cdf89bfa Binary files /dev/null and b/en/latest/_images/phoenixnap-api-credential.png differ diff --git a/en/latest/_images/phoenixnap-concept-public-network.png b/en/latest/_images/phoenixnap-concept-public-network.png new file mode 100644 index 0000000000..eb2ee8c35c Binary files /dev/null and b/en/latest/_images/phoenixnap-concept-public-network.png differ diff --git a/en/latest/_images/phoenixnap-concept-solution-traffic-flows.png b/en/latest/_images/phoenixnap-concept-solution-traffic-flows.png new file mode 100644 index 0000000000..0e58af3d47 Binary files /dev/null and b/en/latest/_images/phoenixnap-concept-solution-traffic-flows.png differ diff --git a/en/latest/_images/phoenixnap-dns-cloudflare.png b/en/latest/_images/phoenixnap-dns-cloudflare.png new file mode 100644 index 0000000000..2d45337dad Binary files /dev/null and b/en/latest/_images/phoenixnap-dns-cloudflare.png differ diff --git a/en/latest/_images/phoenixnap-l4lb.png b/en/latest/_images/phoenixnap-l4lb.png new file mode 100644 index 0000000000..e445fd0e0b Binary files /dev/null and b/en/latest/_images/phoenixnap-l4lb.png differ diff --git a/en/latest/_images/phoenixnap-nat-dnat.png b/en/latest/_images/phoenixnap-nat-dnat.png new file mode 100644 index 0000000000..5c2c8678db Binary files /dev/null and b/en/latest/_images/phoenixnap-nat-dnat.png differ diff --git a/en/latest/_images/phoenixnap-nat-masquerade.png b/en/latest/_images/phoenixnap-nat-masquerade.png new file mode 100644 index 0000000000..8a882dcc30 Binary files /dev/null and b/en/latest/_images/phoenixnap-nat-masquerade.png differ diff --git a/en/latest/_images/phoenixnap-nat-snat.png b/en/latest/_images/phoenixnap-nat-snat.png new file mode 100644 index 0000000000..f83b410439 Binary files /dev/null and b/en/latest/_images/phoenixnap-nat-snat.png differ diff --git a/en/latest/_images/phoenixnap-netris-create-common-subnets.png b/en/latest/_images/phoenixnap-netris-create-common-subnets.png new file mode 100644 index 0000000000..0b25e81e28 Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-create-common-subnets.png differ diff --git a/en/latest/_images/phoenixnap-netris-creating-vnet.png b/en/latest/_images/phoenixnap-netris-creating-vnet.png new file mode 100644 index 0000000000..9554bbfc75 Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-creating-vnet.png differ diff --git a/en/latest/_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png b/en/latest/_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png new file mode 100644 index 0000000000..6eaad541b1 Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png differ diff --git a/en/latest/_images/phoenixnap-netris-ipam-lb-purpose.png b/en/latest/_images/phoenixnap-netris-ipam-lb-purpose.png new file mode 100644 index 0000000000..f024de0677 Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-ipam-lb-purpose.png differ diff --git a/en/latest/_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png b/en/latest/_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png new file mode 100644 index 0000000000..46ec2cacff Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png differ diff --git a/en/latest/_images/phoenixnap-netris-ipam-nat-purpose.png b/en/latest/_images/phoenixnap-netris-ipam-nat-purpose.png new file mode 100644 index 0000000000..e0206cf652 Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-ipam-nat-purpose.png differ diff --git a/en/latest/_images/phoenixnap-netris-ipam-synced-slash-28.png b/en/latest/_images/phoenixnap-netris-ipam-synced-slash-28.png new file mode 100644 index 0000000000..8c6fcbfbbf Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-ipam-synced-slash-28.png differ diff --git a/en/latest/_images/phoenixnap-netris-ipam-synced.png b/en/latest/_images/phoenixnap-netris-ipam-synced.png new file mode 100644 index 0000000000..a102be885d Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-ipam-synced.png differ diff --git a/en/latest/_images/phoenixnap-netris-vnet-ready.png b/en/latest/_images/phoenixnap-netris-vnet-ready.png new file mode 100644 index 0000000000..794a4a1834 Binary files /dev/null and b/en/latest/_images/phoenixnap-netris-vnet-ready.png differ diff --git a/en/latest/_images/phoenixnap-request-ctl-server.png b/en/latest/_images/phoenixnap-request-ctl-server.png new file mode 100644 index 0000000000..9d035e7784 Binary files /dev/null and b/en/latest/_images/phoenixnap-request-ctl-server.png differ diff --git a/en/latest/_images/phoenixnap-request-ip-allocation-slash-28.png b/en/latest/_images/phoenixnap-request-ip-allocation-slash-28.png new file mode 100644 index 0000000000..145e7db0d4 Binary files /dev/null and b/en/latest/_images/phoenixnap-request-ip-allocation-slash-28.png differ diff --git a/en/latest/_images/phoenixnap-request-ip-allocation.png b/en/latest/_images/phoenixnap-request-ip-allocation.png new file mode 100644 index 0000000000..1c4beebb8f Binary files /dev/null and b/en/latest/_images/phoenixnap-request-ip-allocation.png differ diff --git a/en/latest/_images/phoenixnap-reserved-ips.png b/en/latest/_images/phoenixnap-reserved-ips.png new file mode 100644 index 0000000000..815a10d859 Binary files /dev/null and b/en/latest/_images/phoenixnap-reserved-ips.png differ diff --git a/en/latest/_images/phoenixnap-site-create.png b/en/latest/_images/phoenixnap-site-create.png new file mode 100644 index 0000000000..6ce0021bbd Binary files /dev/null and b/en/latest/_images/phoenixnap-site-create.png differ diff --git a/en/latest/_images/phoenixnap-softgate-nodes-created.png b/en/latest/_images/phoenixnap-softgate-nodes-created.png new file mode 100644 index 0000000000..b3f65b8b08 Binary files /dev/null and b/en/latest/_images/phoenixnap-softgate-nodes-created.png differ diff --git a/en/latest/_images/phoenixnap-softgate-nodes-creation.png b/en/latest/_images/phoenixnap-softgate-nodes-creation.png new file mode 100644 index 0000000000..388f903bf5 Binary files /dev/null and b/en/latest/_images/phoenixnap-softgate-nodes-creation.png differ diff --git a/en/latest/_images/phoenixnap-softgate-nodes-green.png b/en/latest/_images/phoenixnap-softgate-nodes-green.png new file mode 100644 index 0000000000..570870bc91 Binary files /dev/null and b/en/latest/_images/phoenixnap-softgate-nodes-green.png differ diff --git a/en/latest/_images/phoenixnap-vnet-import-a-new-server-with-ip.png b/en/latest/_images/phoenixnap-vnet-import-a-new-server-with-ip.png new file mode 100644 index 0000000000..d11fec2196 Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-import-a-new-server-with-ip.png differ diff --git a/en/latest/_images/phoenixnap-vnet-import-a-new-server.png b/en/latest/_images/phoenixnap-vnet-import-a-new-server.png new file mode 100644 index 0000000000..4d091b6248 Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-import-a-new-server.png differ diff --git a/en/latest/_images/phoenixnap-vnet-imported-new-server.png b/en/latest/_images/phoenixnap-vnet-imported-new-server.png new file mode 100644 index 0000000000..2d706c5ecf Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-imported-new-server.png differ diff --git a/en/latest/_images/phoenixnap-vnet-managed-vnet.png b/en/latest/_images/phoenixnap-vnet-managed-vnet.png new file mode 100644 index 0000000000..d4075cf571 Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-managed-vnet.png differ diff --git a/en/latest/_images/phoenixnap-vnet-ssh-to-server.png b/en/latest/_images/phoenixnap-vnet-ssh-to-server.png new file mode 100644 index 0000000000..61dce121ac Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-ssh-to-server.png differ diff --git a/en/latest/_images/phoenixnap-vnet-unmanaged-vnet.png b/en/latest/_images/phoenixnap-vnet-unmanaged-vnet.png new file mode 100644 index 0000000000..3e4b0105d7 Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-unmanaged-vnet.png differ diff --git a/en/latest/_images/phoenixnap-vnet-with-tag-terraform.png b/en/latest/_images/phoenixnap-vnet-with-tag-terraform.png new file mode 100644 index 0000000000..d0adc87318 Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-with-tag-terraform.png differ diff --git a/en/latest/_images/phoenixnap-vnet-with-tag.png b/en/latest/_images/phoenixnap-vnet-with-tag.png new file mode 100644 index 0000000000..f8ef08dc88 Binary files /dev/null and b/en/latest/_images/phoenixnap-vnet-with-tag.png differ diff --git a/en/latest/_images/private-cloud-enterprise-dc-2.png b/en/latest/_images/private-cloud-enterprise-dc-2.png new file mode 100644 index 0000000000..3601718e43 Binary files /dev/null and b/en/latest/_images/private-cloud-enterprise-dc-2.png differ diff --git a/en/latest/_images/quick-action-ports.png b/en/latest/_images/quick-action-ports.png new file mode 100644 index 0000000000..347b7bcb66 Binary files /dev/null and b/en/latest/_images/quick-action-ports.png differ diff --git a/en/latest/_images/request-L4.png b/en/latest/_images/request-L4.png new file mode 100644 index 0000000000..8e73cf8be8 Binary files /dev/null and b/en/latest/_images/request-L4.png differ diff --git a/en/latest/_images/route-map.png b/en/latest/_images/route-map.png new file mode 100644 index 0000000000..692cfd63f3 Binary files /dev/null and b/en/latest/_images/route-map.png differ diff --git a/en/latest/_images/sandbox-l4lb-kubeapi.png b/en/latest/_images/sandbox-l4lb-kubeapi.png new file mode 100644 index 0000000000..7d916a8206 Binary files /dev/null and b/en/latest/_images/sandbox-l4lb-kubeapi.png differ diff --git a/en/latest/_images/sandbox-l4lbs.png b/en/latest/_images/sandbox-l4lbs.png new file mode 100644 index 0000000000..4c246d2d26 Binary files /dev/null and b/en/latest/_images/sandbox-l4lbs.png differ diff --git a/en/latest/_images/sandbox-podinfo-prov.png b/en/latest/_images/sandbox-podinfo-prov.png new file mode 100644 index 0000000000..ed87f72481 Binary files /dev/null and b/en/latest/_images/sandbox-podinfo-prov.png differ diff --git a/en/latest/_images/sandbox-podinfo-ready.png b/en/latest/_images/sandbox-podinfo-ready.png new file mode 100644 index 0000000000..da5bbfd821 Binary files /dev/null and b/en/latest/_images/sandbox-podinfo-ready.png differ diff --git a/en/latest/_images/sandbox_topology.png b/en/latest/_images/sandbox_topology.png new file mode 100644 index 0000000000..ccb044f313 Binary files /dev/null and b/en/latest/_images/sandbox_topology.png differ diff --git a/en/latest/_images/sandbox_topology_new.png b/en/latest/_images/sandbox_topology_new.png new file mode 100644 index 0000000000..68477316e7 Binary files /dev/null and b/en/latest/_images/sandbox_topology_new.png differ diff --git a/en/latest/_images/saveasnormal.png b/en/latest/_images/saveasnormal.png new file mode 100644 index 0000000000..6f5cd5bda3 Binary files /dev/null and b/en/latest/_images/saveasnormal.png differ diff --git a/en/latest/_images/sg_agent_install.png b/en/latest/_images/sg_agent_install.png new file mode 100644 index 0000000000..2bd7ea6827 Binary files /dev/null and b/en/latest/_images/sg_agent_install.png differ diff --git a/en/latest/_images/siteDefault.png b/en/latest/_images/siteDefault.png new file mode 100644 index 0000000000..f8b1cf2688 Binary files /dev/null and b/en/latest/_images/siteDefault.png differ diff --git a/en/latest/_images/site_setup.png b/en/latest/_images/site_setup.png new file mode 100644 index 0000000000..564d07e63b Binary files /dev/null and b/en/latest/_images/site_setup.png differ diff --git a/en/latest/_images/sitemesh_edit.png b/en/latest/_images/sitemesh_edit.png new file mode 100644 index 0000000000..45cbcf12d1 Binary files /dev/null and b/en/latest/_images/sitemesh_edit.png differ diff --git a/en/latest/_images/slide-1.png b/en/latest/_images/slide-1.png new file mode 100644 index 0000000000..b9c94ee198 Binary files /dev/null and b/en/latest/_images/slide-1.png differ diff --git a/en/latest/_images/slide-2.png b/en/latest/_images/slide-2.png new file mode 100644 index 0000000000..99b0ae67e9 Binary files /dev/null and b/en/latest/_images/slide-2.png differ diff --git a/en/latest/_images/slide-3.png b/en/latest/_images/slide-3.png new file mode 100644 index 0000000000..4f17eafc0e Binary files /dev/null and b/en/latest/_images/slide-3.png differ diff --git a/en/latest/_images/slide-4.png b/en/latest/_images/slide-4.png new file mode 100644 index 0000000000..3324e389da Binary files /dev/null and b/en/latest/_images/slide-4.png differ diff --git a/en/latest/_images/snat_add.png b/en/latest/_images/snat_add.png new file mode 100644 index 0000000000..f97ba7db6c Binary files /dev/null and b/en/latest/_images/snat_add.png differ diff --git a/en/latest/_images/softgate-green.png b/en/latest/_images/softgate-green.png new file mode 100644 index 0000000000..5043c928bd Binary files /dev/null and b/en/latest/_images/softgate-green.png differ diff --git a/en/latest/_images/softgate-install-agent.png b/en/latest/_images/softgate-install-agent.png new file mode 100644 index 0000000000..b07f8b551b Binary files /dev/null and b/en/latest/_images/softgate-install-agent.png differ diff --git a/en/latest/_images/softgate-nodes-created-in-equinix.png b/en/latest/_images/softgate-nodes-created-in-equinix.png new file mode 100644 index 0000000000..21a8cd4425 Binary files /dev/null and b/en/latest/_images/softgate-nodes-created-in-equinix.png differ diff --git a/en/latest/_images/softgate-nodes-recognized-in-netris.png b/en/latest/_images/softgate-nodes-recognized-in-netris.png new file mode 100644 index 0000000000..bfcba948e7 Binary files /dev/null and b/en/latest/_images/softgate-nodes-recognized-in-netris.png differ diff --git a/en/latest/_images/softgate-one-liner-provisioning.png b/en/latest/_images/softgate-one-liner-provisioning.png new file mode 100644 index 0000000000..f1c72da232 Binary files /dev/null and b/en/latest/_images/softgate-one-liner-provisioning.png differ diff --git a/en/latest/_images/softgate-provisioning-cli-output.png b/en/latest/_images/softgate-provisioning-cli-output.png new file mode 100644 index 0000000000..a2f2bdd9bd Binary files /dev/null and b/en/latest/_images/softgate-provisioning-cli-output.png differ diff --git a/en/latest/_images/softgate_diagram.png b/en/latest/_images/softgate_diagram.png new file mode 100644 index 0000000000..a6ade1fd5f Binary files /dev/null and b/en/latest/_images/softgate_diagram.png differ diff --git a/en/latest/_images/static_route2_empty.png b/en/latest/_images/static_route2_empty.png new file mode 100644 index 0000000000..17be7b2c53 Binary files /dev/null and b/en/latest/_images/static_route2_empty.png differ diff --git a/en/latest/_images/static_route3.png b/en/latest/_images/static_route3.png new file mode 100644 index 0000000000..6db5da6f3f Binary files /dev/null and b/en/latest/_images/static_route3.png differ diff --git a/en/latest/_images/static_route_empty.png b/en/latest/_images/static_route_empty.png new file mode 100644 index 0000000000..31bb42ac8f Binary files /dev/null and b/en/latest/_images/static_route_empty.png differ diff --git a/en/latest/_images/subnet-tree.png b/en/latest/_images/subnet-tree.png new file mode 100644 index 0000000000..22710f17e9 Binary files /dev/null and b/en/latest/_images/subnet-tree.png differ diff --git a/en/latest/_images/subnet_empty.png b/en/latest/_images/subnet_empty.png new file mode 100644 index 0000000000..0dd21f1a9c Binary files /dev/null and b/en/latest/_images/subnet_empty.png differ diff --git a/en/latest/_images/switch_fabric_vpc.png b/en/latest/_images/switch_fabric_vpc.png new file mode 100644 index 0000000000..a6068a92cb Binary files /dev/null and b/en/latest/_images/switch_fabric_vpc.png differ diff --git a/en/latest/_images/topology.png b/en/latest/_images/topology.png new file mode 100644 index 0000000000..379b61b118 Binary files /dev/null and b/en/latest/_images/topology.png differ diff --git a/en/latest/_images/topology_2.png b/en/latest/_images/topology_2.png new file mode 100644 index 0000000000..e8ef02813f Binary files /dev/null and b/en/latest/_images/topology_2.png differ diff --git a/en/latest/_images/topology_completed.png b/en/latest/_images/topology_completed.png new file mode 100644 index 0000000000..5ae28552e4 Binary files /dev/null and b/en/latest/_images/topology_completed.png differ diff --git a/en/latest/_images/topology_create_link.png b/en/latest/_images/topology_create_link.png new file mode 100644 index 0000000000..8d1cee5cd7 Binary files /dev/null and b/en/latest/_images/topology_create_link.png differ diff --git a/en/latest/_images/uninstallOS.png b/en/latest/_images/uninstallOS.png new file mode 100644 index 0000000000..eb2e1965c0 Binary files /dev/null and b/en/latest/_images/uninstallOS.png differ diff --git a/en/latest/_images/unmanaged-vlan-equinix.png b/en/latest/_images/unmanaged-vlan-equinix.png new file mode 100644 index 0000000000..26f5d2a180 Binary files /dev/null and b/en/latest/_images/unmanaged-vlan-equinix.png differ diff --git a/en/latest/_images/unmanaged-vnet.png b/en/latest/_images/unmanaged-vnet.png new file mode 100644 index 0000000000..1d9afc47f2 Binary files /dev/null and b/en/latest/_images/unmanaged-vnet.png differ diff --git a/en/latest/_images/upgrading_sonic_folder_listing.png b/en/latest/_images/upgrading_sonic_folder_listing.png new file mode 100644 index 0000000000..10825b7e34 Binary files /dev/null and b/en/latest/_images/upgrading_sonic_folder_listing.png differ diff --git a/en/latest/_images/upstream-dmz-logical.png b/en/latest/_images/upstream-dmz-logical.png new file mode 100644 index 0000000000..6a0be1d504 Binary files /dev/null and b/en/latest/_images/upstream-dmz-logical.png differ diff --git a/en/latest/_images/upstream-dmz-physical.png b/en/latest/_images/upstream-dmz-physical.png new file mode 100644 index 0000000000..a3d2637c5c Binary files /dev/null and b/en/latest/_images/upstream-dmz-physical.png differ diff --git a/en/latest/_images/user_set_password.png b/en/latest/_images/user_set_password.png new file mode 100644 index 0000000000..4fd601ddc7 Binary files /dev/null and b/en/latest/_images/user_set_password.png differ diff --git a/en/latest/_images/vnet.png b/en/latest/_images/vnet.png new file mode 100644 index 0000000000..5a8fe31874 Binary files /dev/null and b/en/latest/_images/vnet.png differ diff --git a/en/latest/_images/vpc-anywhere-check-site-default.png b/en/latest/_images/vpc-anywhere-check-site-default.png new file mode 100644 index 0000000000..5f133ebd39 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-check-site-default.png differ diff --git a/en/latest/_images/vpc-anywhere-edit-vlan-range-default-site.png b/en/latest/_images/vpc-anywhere-edit-vlan-range-default-site.png new file mode 100644 index 0000000000..50531fb072 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-edit-vlan-range-default-site.png differ diff --git a/en/latest/_images/vpc-anywhere-ipam-allocation.png b/en/latest/_images/vpc-anywhere-ipam-allocation.png new file mode 100644 index 0000000000..649c00b33a Binary files /dev/null and b/en/latest/_images/vpc-anywhere-ipam-allocation.png differ diff --git a/en/latest/_images/vpc-anywhere-ipam-l4lb-subnet.png b/en/latest/_images/vpc-anywhere-ipam-l4lb-subnet.png new file mode 100644 index 0000000000..337e63732f Binary files /dev/null and b/en/latest/_images/vpc-anywhere-ipam-l4lb-subnet.png differ diff --git a/en/latest/_images/vpc-anywhere-ipam-nat-subnet.png b/en/latest/_images/vpc-anywhere-ipam-nat-subnet.png new file mode 100644 index 0000000000..0234367484 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-ipam-nat-subnet.png differ diff --git a/en/latest/_images/vpc-anywhere-l4lb.png b/en/latest/_images/vpc-anywhere-l4lb.png new file mode 100644 index 0000000000..f5d39737d2 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-l4lb.png differ diff --git a/en/latest/_images/vpc-anywhere-nat-dnat.png b/en/latest/_images/vpc-anywhere-nat-dnat.png new file mode 100644 index 0000000000..e2c1487a6b Binary files /dev/null and b/en/latest/_images/vpc-anywhere-nat-dnat.png differ diff --git a/en/latest/_images/vpc-anywhere-nat-masquerade.png b/en/latest/_images/vpc-anywhere-nat-masquerade.png new file mode 100644 index 0000000000..d5118ed47a Binary files /dev/null and b/en/latest/_images/vpc-anywhere-nat-masquerade.png differ diff --git a/en/latest/_images/vpc-anywhere-nat-snat.png b/en/latest/_images/vpc-anywhere-nat-snat.png new file mode 100644 index 0000000000..4f2658c7db Binary files /dev/null and b/en/latest/_images/vpc-anywhere-nat-snat.png differ diff --git a/en/latest/_images/vpc-anywhere-sg-to-sg-experimental.png b/en/latest/_images/vpc-anywhere-sg-to-sg-experimental.png new file mode 100644 index 0000000000..9e7275f132 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-sg-to-sg-experimental.png differ diff --git a/en/latest/_images/vpc-anywhere-softgates-green.png b/en/latest/_images/vpc-anywhere-softgates-green.png new file mode 100644 index 0000000000..bb919d3185 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-softgates-green.png differ diff --git a/en/latest/_images/vpc-anywhere-solution-traffic-flows.png b/en/latest/_images/vpc-anywhere-solution-traffic-flows.png new file mode 100644 index 0000000000..bbc1ad06ce Binary files /dev/null and b/en/latest/_images/vpc-anywhere-solution-traffic-flows.png differ diff --git a/en/latest/_images/vpc-anywhere-upstream-bgp-router-logical.png b/en/latest/_images/vpc-anywhere-upstream-bgp-router-logical.png new file mode 100644 index 0000000000..2096aeacfa Binary files /dev/null and b/en/latest/_images/vpc-anywhere-upstream-bgp-router-logical.png differ diff --git a/en/latest/_images/vpc-anywhere-upstream-bgp-router-physical.png b/en/latest/_images/vpc-anywhere-upstream-bgp-router-physical.png new file mode 100644 index 0000000000..423d159a9e Binary files /dev/null and b/en/latest/_images/vpc-anywhere-upstream-bgp-router-physical.png differ diff --git a/en/latest/_images/vpc-anywhere-vnet-experimental.png b/en/latest/_images/vpc-anywhere-vnet-experimental.png new file mode 100644 index 0000000000..50408639d5 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-vnet-experimental.png differ diff --git a/en/latest/_images/vpc-anywhere-vnet.png b/en/latest/_images/vpc-anywhere-vnet.png new file mode 100644 index 0000000000..3249bf18b4 Binary files /dev/null and b/en/latest/_images/vpc-anywhere-vnet.png differ diff --git a/en/latest/_images/vpc_add.png b/en/latest/_images/vpc_add.png new file mode 100644 index 0000000000..1d107ff894 Binary files /dev/null and b/en/latest/_images/vpc_add.png differ diff --git a/en/latest/_images/vpc_concept.png b/en/latest/_images/vpc_concept.png new file mode 100644 index 0000000000..21aef9ef44 Binary files /dev/null and b/en/latest/_images/vpc_concept.png differ diff --git a/en/latest/_images/vpc_diagram.png b/en/latest/_images/vpc_diagram.png new file mode 100644 index 0000000000..21aef9ef44 Binary files /dev/null and b/en/latest/_images/vpc_diagram.png differ diff --git a/en/latest/_images/vpc_empty.png b/en/latest/_images/vpc_empty.png new file mode 100644 index 0000000000..b5b53ad1f4 Binary files /dev/null and b/en/latest/_images/vpc_empty.png differ diff --git a/en/latest/_images/waiting_approval.png b/en/latest/_images/waiting_approval.png new file mode 100644 index 0000000000..ac769acb93 Binary files /dev/null and b/en/latest/_images/waiting_approval.png differ diff --git a/en/latest/_sources/Controller-local-repository.rst.txt b/en/latest/_sources/Controller-local-repository.rst.txt new file mode 100644 index 0000000000..8208370363 --- /dev/null +++ b/en/latest/_sources/Controller-local-repository.rst.txt @@ -0,0 +1,93 @@ +Netris Local Repository Setup +============================= + + +When & Why to Use the Local Repository? +--------------------------------------- + +The Netris Local Repository is essential for environments where switches, softgates, or other infrastructure devices do not have direct access to the internet. By setting up a local repository, you ensure that these devices can still download necessary packages and updates through a local APT repository. This setup is particularly useful in air-gapped or restricted network environments, where relying on external repositories is not an option. +In addition to serving as an APT repository, the local repository can also host custom ISO files. This feature allows you to store and manage ISOs for specific use cases, such as deploying operating systems or firmware to your servers, softgates, or switches. While ISO hosting is an optional feature, it provides flexibility and control over the resources you use in your infrastructure, especially when working in isolated networks. +By utilizing the Netris Local Repository, you ensure that your infrastructure can always access necessary files and packages, even when internet connectivity is limited or unavailable. + +How to Enable the Local Repository on the Netris Controller? +------------------------------------------------------------ + +To install the Netris Local Repository on the controller host, run the following command: +*Note: The controller must be set up using the `--ctl-hostname` argument; otherwise, the Netris Local Repository setup script will exit with an error.* + +.. code-block:: shell-session + + curl -sS https://get.netris.io/local-repo.sh | sh - + +The script may take a while, depending on the controller's connectivity and available resources. + +The output of the script will look similar to this: + +.. code-block:: shell-session + + ubuntu@netris:~$ curl -sS https://get.netris.io/local-repo.sh | sh - + Checking if the controller is installed + Checking controller's FQDN + Creating local repo service under URL: http://netris.example.com/repo/ + Waiting for netris-local-repo pod to be ready... + No pods found. Waiting for pod creation... + Current pod status: Pending. Waiting... + Current pod status: Pending. Waiting... + Current pod status: Pending. Waiting... + Current pod status: Pending. Waiting... + Current pod status: Pending. Waiting... + Pod is running. + + Netris local repository pod is now running. + + Path to upload custom ISOs: /var/lib/rancher/k3s/storage/pvc-ea0dd3ef-ded1-49d3-bbd8-b797c91d76b5_netris-controller_netris-local-repo-pvc/repo/isos + Local repository URL: http://netris.example.com/repo/ + URL to ISOs directory: http://netris.example.com/repo/isos/ + +The script will output the local repository URL and URL to ISOs directory. Copy and paste this URL of the Local repository into the Netris Controller Web UI under **Settings** section (as shown in the screenshots below). + +Additionally, it provides the host system path, which you may want to use to host your custom ISOs for your servers, softgates, or switches. + +.. image:: images/Global-settings-edit.png + :align: center + +.. image:: images/Global-settings-save.png + :align: center + + +How to consume local repository +------------------------------- + +Once the local repository function is enabled in the Netris Controller Settings, the Netris agent installation oneliner will automatically point to the local repository (as shown in the screenshots below). + +.. note:: + + The local repository includes all the necessary scripts and dependency packages for the Netris NVUE (Cumulus 5.9 and higher) and Netris SoftGate HS (Ubuntu 24.04) agents. + + +.. image:: images/oneliner-from-local-repo.png + :align: center + + +Upgrade Local-Repo Cache +------------------------ + +To update the local repository, restart the `local-repo` deployment in the controller. Then, verify the deployment status to ensure the rollout succeeds: + +.. code-block:: shell-session + + kubectl -nnetris-controller rollout restart deploy/netris-local-repo-nginx + kubectl -nnetris-controller rollout status deploy/netris-local-repo-nginx + + +Expected output: + +.. code-block:: shell-session + + ubuntu@netris:~$ kubectl -nnetris-controller rollout restart deploy/netris-local-repo-nginx + deployment.apps/netris-local-repo-nginx restarted + + ubuntu@netris:~$ kubectl -nnetris-controller rollout status deploy/netris-local-repo-nginx + Waiting for deployment "netris-local-repo-nginx" rollout to finish: 1 old replicas are pending termination... + Waiting for deployment "netris-local-repo-nginx" rollout to finish: 1 old replicas are pending termination... + deployment "netris-local-repo-nginx" successfully rolled out diff --git a/en/latest/_sources/Dell-SONiC-Switch-initial-setup.rst.txt b/en/latest/_sources/Dell-SONiC-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..81d5eef8d2 --- /dev/null +++ b/en/latest/_sources/Dell-SONiC-Switch-initial-setup.rst.txt @@ -0,0 +1,104 @@ +=================================== +Dell SONiC Switch Initial Setup +=================================== +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first. + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of Dell SONiC. + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Dell SONiC image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/Enterprise_SONiC_OS_4.1.1_Enterprise_Premium.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password: + +``admin/YourPaSsWoRd`` + +3. Set up the Out-of-Band (OOB) Management. + +Upon the initial boot of the newly installed NOS, please wait until the message "System is ready" appears, typically resembling the following: + +.. code-block:: shell-session + + May 03 15:02:35.430469 System is ready + +Once this message is displayed, proceed to disable ZTP using the following commands: + +.. code-block:: shell-session + + admin@sonic:~$ sonic-cli + sonic# configure + sonic(config)# no ztp enable + +Wait once more until the "System is ready" message reappears. This may take approximately 1-2 minutes. +If there isn't a DHCP server available in the Out-of-Band (OOB) management network to ensure the switch's connection to the internet, you'll need to configure the Management IP address, gateway, and nameservers using the following commands: + +.. code-block:: shell-session + + admin@sonic:~$ sonic-cli + sonic# configure + sonic(config-if-Management0)# interface Management 0 + sonic(config-if-Management0)# ip address / gwaddr + sonic(config)# exit + sonic(config)# ip name-server + sonic(config)# ip name-server + sonic(config)# end + sonic# write memory + sonic# exit + +.. _sonic-switch-agent-installation: + +4. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent, copy the one-line installer command to your clipboard and paste in the switch. + +.. image:: images/Dell-Switch-agent-installation-Inventory.png + :align: center + +.. image:: images/Dell-Switch-agent-installation-oneliner.png + :align: center + +.. image:: images/Dell-Switch-agent-installation-cli.png + :align: center + +5. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/latest/_sources/EdgeCore-SONiC-Switch-initial-setup.rst.txt b/en/latest/_sources/EdgeCore-SONiC-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..232d0bfe24 --- /dev/null +++ b/en/latest/_sources/EdgeCore-SONiC-Switch-initial-setup.rst.txt @@ -0,0 +1,94 @@ +=================================== +EdgeCore SONiC Switch Initial Setup +=================================== +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first. + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of EC SONiC. + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/Edgecore-SONiC_20211125_074752_ec202012_227.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password: + +``admin/YourPaSsWoRd`` + +3. Set up the Out-of-Band (OOB) Management. + +Disable ztp: + +.. code-block:: shell-session + + ztp disable -y + +Configure the IP address, default gateway, and DNS to establish Internet connectivity via the management port. + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +.. _sonic-switch-agent-installation: + +4. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner.png + :align: center + +.. image:: images/Switch-agent-installation-cli.png + :align: center + +5. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/latest/_sources/Nvidia-Cumulus-v3.7-Switch-initial-setup.rst.txt b/en/latest/_sources/Nvidia-Cumulus-v3.7-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..ea006a1e32 --- /dev/null +++ b/en/latest/_sources/Nvidia-Cumulus-v3.7-Switch-initial-setup.rst.txt @@ -0,0 +1,108 @@ +======================================== +Nvidia Cumulus v3.7 Switch Initial Setup +======================================== +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first. + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux. + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/cumulus-linux-3.7.15-mlx-amd64.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password for Cumulus v3.7: + +``cumulus/CumulusLinux!`` + +3. Set up the Out-of-Band (OOB) Management. + +Open the network interfaces file and add the IP address and other required details. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + echo "nameserver " | sudo tee /etc/resolv.conf + +.. code-block:: shell-session + + sudo ifreload -a + +4. Cumulus Linux license installation. + +.. code-block:: shell-session + + sudo cl-license -i + +Copy/paste the Cumulus Linux license string, then press ctrl-d. + +5. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner.png + :align: center + +.. image:: images/Switch-agent-installation-cli.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/latest/_sources/Nvidia-Cumulus-v5-Switch-initial-setup.rst.txt b/en/latest/_sources/Nvidia-Cumulus-v5-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..1145d90deb --- /dev/null +++ b/en/latest/_sources/Nvidia-Cumulus-v5-Switch-initial-setup.rst.txt @@ -0,0 +1,130 @@ +.. _switch-agent-installation: +.. meta:: + :description: Network Switch Initial Setup + +====================================== +Nvidia Cumulus v5 Switch Initial Setup +====================================== + + +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first + + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux. + + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/cumulus-linux-5.4.0-mlx-amd64.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password for Cumulus v5: + +``cumulus/cumulus`` + + +3. Set up the Out-of-Band (OOB) Management. + +Upon installation of Cumulus Linux v5, the default Virtual Routing and Forwarding (VRF) is set to 'mgmt.' To switch to the default VRF, please refer to the following instructions: + +Disable ztp: + +.. code-block:: shell-session + + sudo ztp -d + +.. code-block:: shell-session + + sudo ip vrf exec default bash + +Open the network interfaces file, add the IP address and other required details, and ensure that you remove the 'mgmt' VRF configuration: + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + echo "nameserver " | sudo tee /etc/resolv.conf + +.. code-block:: shell-session + + sudo ifreload -a + +.. note:: + + You might see a one-time warning in the output of ifreload, which you can ignore: + +.. code-block:: shell-session + + warning: mgmt: cmd '/usr/lib/vrf/vrf-helper delete mgmt 1001' failed: returned 1 (Failed to delete cgroup for vrf mgmt) + + +4. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner.png + :align: center + +.. image:: images/Switch-agent-installation-cli.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/latest/_sources/Nvidia-Cumulus-v5.9+-Switch-initial-setup.rst.txt b/en/latest/_sources/Nvidia-Cumulus-v5.9+-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..3a5ba271b5 --- /dev/null +++ b/en/latest/_sources/Nvidia-Cumulus-v5.9+-Switch-initial-setup.rst.txt @@ -0,0 +1,109 @@ +.. _switch-agent-installation: +.. meta:: + :description: Network Switch Initial Setup + +====================================== +Nvidia Cumulus v5.9+ Switch Initial Setup +====================================== + + +.. note:: + + Further installation requires a Console and Internet connectivity via the management port! + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first + + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of the Cumulus Linux. + + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +The Cumulus image should be available on a web server to which the switch has access through the local network or the Internet. + +Example: + +.. code-block:: shell-session + + onie-nos-install http://192.168.100.10/cumulus-linux-5.9.1-mlx-amd64.bin + +After completion of the installation, the switch will automatically reboot. + +To login use the default username and password for Cumulus v5.9: + +``cumulus/cumulus`` + + +3. Set up the Out-of-Band (OOB) Management. + +Upon installation of Cumulus Linux v5.9 or later ZTP must be disabled and internet connectivity must be provided to the switch via OOB management network: + +Disable ztp: + +.. code-block:: shell-session + + sudo ztp -d + +If there is no DHCP server in the OOB management network, then IP/Gateway/DNS information must be configured manually: + +.. code-block:: shell-session + + nv set interface eth0 ip address + nv set interface eth0 ip gateway + nv set service dns mgmt server + nv set service dns mgmt server + nv unset interface eth0 ip address dhcp + nv config apply -y + nv config save + + +4. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory-cl5.9.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner-cl5.9.png + :align: center + +.. image:: images/Switch-agent-installation-cli-cl5.9.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot + +7. Netris agent connected to the controller. + +.. image:: images/Switch-agent-installation-Inventory-online-cl5.9.png + :align: center + +Once the switch is back, the Heartbeat status should be "OK" in the Netris controller. diff --git a/en/latest/_sources/SoftGate-PRO-installation.rst.txt b/en/latest/_sources/SoftGate-PRO-installation.rst.txt new file mode 100644 index 0000000000..26417b6c7a --- /dev/null +++ b/en/latest/_sources/SoftGate-PRO-installation.rst.txt @@ -0,0 +1,72 @@ +.. meta:: + :description: Netris SoftGate PRO Installation + +*************************** +SoftGate PRO Installation +*************************** + +Minimum Hardware Requirements +============================= +* 2 x Intel® Xeon® Silver Processor with 10 physical cores per socket (20 cores total) +* 128 GB (64 GB RAM per socket) in multichannel configuration +* 300 GB HDD +* Nvidia Mellanox Connect-X 5/6 SmartNIC card + +BIOS Configuration +================== +The following are some recommendations for BIOS settings. Different vendors will have different BIOS naming so the following is mainly for reference: + +* Before starting consider resetting all BIOS settings to their defaults +* Disable all power saving options such as: Power performance tuning, CPU P-State, CPU C3 Report and CPU C6 Report +* Select Performance as the CPU Power and Performance policy +* Enable Turbo Boost +* Set memory frequency to the highest available number, NOT auto +* Disable all virtualization options when you test the physical function of the NIC, and turn off VT-d +* Disable Hyper-Threading + +Install the Netris Agent +======================== +Requires freshly installed Ubuntu Linux 18.04 LTS and internet connectivity configured from netplan via management port. + +1. Add the SoftGate in the controller **Inventory** or **Topology** section. Detailed configuration documentation is available here: :ref:`"Adding SoftGates"`. +2. Once the SoftGate is created, navigate to the **Inventory** section, click the **three vertical dots (⋮)** on the right side of the newly created SoftGate and select the **Install Agent** option. +3. Copy the agent install line to your clipboard and run it on the SoftGate as an ordinary user. +4. When the installation is complete, review the ifupdown configuration file and verify that the presented configuration corresponds to what you configured during OS installation (the file is generated based on your initial netplan configuration). + +.. note:: + + If the Netris Controller is not in the same OOB network then add a route to Netris Controller. No default route or other IP addresses should be configured. + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The management network interface + auto ensZ + iface ensZ inet static + address + # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node. + up ip route add via + gateway + + source /etc/network/interfaces.d/* + +5. If everything seems ok, please remove/comment the **Gateway** line and save the file. + +.. note:: + + Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent. + +6. Reboot the SoftGate + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. \ No newline at end of file diff --git a/en/latest/_sources/SoftGate-installation.rst.txt b/en/latest/_sources/SoftGate-installation.rst.txt new file mode 100644 index 0000000000..fc29f41685 --- /dev/null +++ b/en/latest/_sources/SoftGate-installation.rst.txt @@ -0,0 +1,91 @@ +.. meta:: + :description: Netris SoftGate Installation + +*************************** +SoftGate Installation +*************************** + +Minimum Hardware Requirements +============================= +* 8 CPU cores +* 16 GB RAM +* 300 GB HDD + +Provision Netris SoftGate software +================================== +Requires freshly installed Ubuntu Linux 22.04 LTS and internet connectivity. + +1. Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: :ref:`"Adding SoftGates"`. + +2. Navigate to the **Net-->Inventory** section and click the **three vertical dots (⋮)** on the right side of the SoftGate node you are provisioning. Then click **Install Agent** and copy the one-line installer command to your clipboard. + +.. image:: /images/softgate-install-agent.png + :align: center + + +3. Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node) + +.. image:: /images/softgate-provisioning-cli-output.png + :align: center + +.. note:: + Please note that Netris replaces Netplan with regular ifupdown and attempts to migrate any prior configuration to /etc/network/interfaces. + +4. Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as you describe in /etc/network/interfaces. + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # Physical port on SoftGate node connected to a TRUNK port of your network + auto ens + iface ens inet static + address 0.0.0.0/0 + + # Optionally you can add more physical interfaces under your bond0 + auto ens + iface ens inet static + address 0.0.0.0/0 + + # Bond interface + auto bond0 + iface bond0 inet static + address 0.0.0.0/0 + # Please replace the ensX/Y with actual interface name(s) below to one(s) present in the OS. + bond-slaves ens ens + # Optional, please adjust the bonding mode below according to the desired functionality. + bond-mode active-backup + + source /etc/network/interfaces.d/* + + +5. Ensure that SoftGate node will maintain IP connectivity with Netris Controller after reboot. + + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The management network interface + auto ensZ + iface ensZ inet static + address + # Please delete or comment the line below if Netris Controller is located in the same network with the SoftGate node. + up ip route add via + + +6. Reboot the SoftGate + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up, you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. diff --git a/en/latest/_sources/Ubuntu-SwitchDev-Switch-initial-setup.rst.txt b/en/latest/_sources/Ubuntu-SwitchDev-Switch-initial-setup.rst.txt new file mode 100644 index 0000000000..c51a5eea76 --- /dev/null +++ b/en/latest/_sources/Ubuntu-SwitchDev-Switch-initial-setup.rst.txt @@ -0,0 +1,94 @@ +===================================== +Ubuntu SwitchDev Switch Initial Setup +===================================== +.. note:: + + Further installation requires a Console and Internet connectivity via management port! + + +If the switch has pre-installed network operating system (NOS), it needs to be uninstalled first. + +1. NOS Uninstall (if pre-installed) + +To uninstall the current NOS, access **ONIE** from the GRUB menu and select the **Uninstall OS** option. + +.. image:: images/uninstallOS.png + :align: center + +Once it's done, the switch will automatically reboot and get ready for the installation of the Ubuntu SwitchDev. + +2. NOS Install + +If there is no DHCP in the management network, stop the onie-discovery service and configure an IP address and default gateway manually. + +.. code-block:: shell-session + + onie-discovery-stop + +.. code-block:: shell-session + + ip addr add dev eth0 + +.. code-block:: shell-session + + ip route add default via + +.. code-block:: shell-session + + echo "nameserver " > /etc/resolv.conf + +Install Ubuntu SwitchDev using the Netris customized image: + +.. code-block:: shell-session + + onie-nos-install http://downloads.netris.ai/netris-ubuntu-18.04.1.bin + +Default username/password + +``netris/newNet0ps`` + +3. Set up the Out-of-Band (OOB) Management. + +Open the network interfaces file and add the IP address and other required details. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +5. Netris agent installation. + +Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the switch you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard. + +.. image:: images/Switch-agent-installation-Inventory-ubuntu.png + :align: center + +.. image:: images/Switch-agent-installation-oneliner-ubuntu.png + :align: center + +.. image:: images/Switch-agent-installation-cli-ubuntu.png + :align: center + +6. Reboot the switch + +.. code-block:: shell-session + + sudo reboot diff --git a/en/latest/_sources/accounts.rst.txt b/en/latest/_sources/accounts.rst.txt new file mode 100644 index 0000000000..d1594f9718 --- /dev/null +++ b/en/latest/_sources/accounts.rst.txt @@ -0,0 +1,69 @@ +.. meta:: + :description: Netris Controller User Account Management + +######## +Accounts +######## + +The accounts section is for the management of user accounts, access permissions, and tenants. + +Users +===== +Description of User account fields: + +* **Username** - Unique username. +* **Full Name** - Full Name of the user. +* **E-mail** - The email address of the user. Also used for system notifications and for password retrieval. +* **E-mail CC** - Send copies of email notifications to this address. +* **Phone Number** - User’s phone number. +* **Company** - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers. +* **Position** - Position within the company. +* **User Role** - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate. +* **Permission Group** - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used) +* **+Tenant** - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used) + +Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin. + +.. image:: images/add_user.png + :align: center + :alt: User Management + +**Password**: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side. + +.. image:: images/user_set_password.png + :align: center + :alt: List User Accounts + +Tenants +======= + +IP addresses and Network Interfaces are resources that can be assigned to different Tenants for their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs or Terraform) as part of DevOps/NetOps pipeline. + +A Tenant has just two fields, the unique name and custom description. + +Example: Adding a tenant. + +.. image:: images/add_tenant.png + :align: center + :alt: Adding Tenants + +Permission Groups +================= + +Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections. + +Example: Permission Group. + +.. image:: images/add_perm_group.png + :align: center + :alt: Managing Permissions + +User Roles +========== + +Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username. + +.. image:: images/add_user_role.png + :align: center + :alt: User Roles + diff --git a/en/latest/_sources/acls.rst.txt b/en/latest/_sources/acls.rst.txt new file mode 100644 index 0000000000..4ff5a7ab12 --- /dev/null +++ b/en/latest/_sources/acls.rst.txt @@ -0,0 +1,115 @@ +.. meta:: + :description: Access Control Lists (ACLs) + +.. _acl_def: + +########################## +Access Control Lists (ACL) +########################## +Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access. + +Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches. + +Screenshot: TCAM utilization can be seen under Net→Inventory + +.. image:: images/TCAM.png + :align: center + :class: with-shadow + +Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements. + +ACL Default Policy +------------------ +The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules. + +Example: Changing “ACL Default Policy” for the site “siteDefault”. + +.. image:: images/siteDefault.png + :align: center + :class: with-shadow + + +ACL Rules +--------- +ACL rules can be created, listed, edited, approved under Services→ACL. + +Description of ACL fields. +General + +* **Name** - Unique name for the ACL entry. +* **Protocol** - IP protocol to match. + + * All - Any IP protocols. + * IP - Specific IP protocol number. + * TCP - TCP. + * UDP - UDP. + * ICMP ALL - Any IPv4 ICMP protocol. + * ICMP Custom - Custom IPv4 ICMP code. + * ICMPv6 ALL - Any IPv6 ICMP protocol. + * ICMPv6 Custom - Custom IPv6 ICMP code. + +* **Active Until** - Disable this rule at the defined date/time. +* **Action** - Permit or Deny forwarding of matched packets. +* **Established/Reverse** - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination. + +Source/Destination - Source and destination addresses and ports to match. + +* **Source** IPv4/IPv6 - IPv4/IPv6 address. +* **Ports Type** + + * Port Range - Match on the port or a port range defined in this window. + * Port Group - Match on a group of ports defined under Services→ ACL Port Group. + +* **From Port** - Port range starting from. +* **To Port** - Port range ending with. + +* **Comment** - Descriptive comment, commonly used for approval workflows. + +* **Check button** - Check if Another ACL on the system already permits the described network access. + +Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established). + +.. image:: images/action_permit.png + :align: center + :class: with-shadow + +Example: “Check” shows that requested access is already provided by a broader ACL rule. + +.. image:: images/ACL_rule.png + :align: center + :class: with-shadow + +ACL Approval Workflow +--------------------- +When one tenant (one team) needs to get network access to resources under the responsibility of another tenant (another team), an ACL can be created but will activate only after approval of the tenant responsible for the destination address resources. See the below example. + +Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant. + +.. image:: images/ACL_approval.png + :align: center + :class: with-shadow + +Screenshot: ACL stays in “waiting for approval” state until approved. + +.. image:: images/waiting_approval.png + :align: center + :class: with-shadow + +Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it. + +.. image:: images/approve_reject.png + :align: center + :class: with-shadow + +Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric. + +.. image:: images/ACL_active.png + :align: center + :class: with-shadow + +ACL Processing Order +-------------------- +#. User-defined Deny Rules +#. User-defined Permit Rules +#. Deny the rest + diff --git a/en/latest/_sources/controller-k3s-installation.rst.txt b/en/latest/_sources/controller-k3s-installation.rst.txt new file mode 100644 index 0000000000..2aec4f8276 --- /dev/null +++ b/en/latest/_sources/controller-k3s-installation.rst.txt @@ -0,0 +1,220 @@ +.. meta:: + :description: Controller Generic Linux Host + +.. _ctl-k3s-def: + +###################################################### +Netris Controller installation on a generic Linux host +###################################################### + +Linux Host requirements +======================= + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +.. note:: + + K3s is expected to work on most modern Linux systems. + + Some OSs have specific requirements: + + * If you are using Raspbian Buster, follow `these steps `__ to switch to legacy iptables. + * If you are using Alpine Linux, follow `these steps `__ for additional setup. + * If you are using (Red Hat/CentOS) Enterprise Linux, follow `these steps `__ for additional setup. + + +Installation +============ + +The following command will install the Netris Controller on your Linux server: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +Once installed, you will be able to log in to Netris Controller using your host's IP address. + + +.. note:: + The installation script does the following: + + * Installs `k3s `_ + * Installs the `Cert-Manager Helm chart `_ + * Installs the `Netris Controller Helm chart `_ + + + +Installation with the specific host name +---------------------------------------- + +In order to set the specific ingress host name to the Netris Controller, use the ``--ctl-hostname`` installation argument: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com + +A self-signed SSL certificate will be generated from that host name. + +Installation with the Let's Encrypt SSL +--------------------------------------- + +The installation script supports Let's Encrypt SSL generation out-of-box. To instruct the installation script to do that use ``--ctl-ssl-issuer`` argument. + +.. note:: + | The argument ``--ctl-ssl-issuer`` is passing ``cert-manager.io/cluster-issuer`` value to the ingress resource of the Netris Controller. The installation script can create two types of ClusterIssuer resource: ``selfsigned`` or ``letsencrypt``, where ``selfsigned`` is just `Cert-Manager self-signed `_ SSL and the ``letsencrypt`` is the ACME issuer with `HTTP01 challenge validation `_. + | If the ``--ctl-ssl-issuer`` argument is not set, the installation script will proceed with ``selfsigned`` ClusterIssuer type. + + +Run the following command to install Netris Controller and use ``letsencrypt`` ClusterIssuer for SSL generation: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + + To successfully validate and complete Let's Encrypt SSL generation, a valid A/CNAME record for the domain/subdomain name should exist prior, and that name must be accessible from the Internet. + + +Installation with the Custom SSL Issuer +--------------------------------------- + +The HTTP01 challenge validation is the simplest way of issuing the Let's Encrypt SSL, but it does not work when the host behind the FQDN is not accessible from the public internet. +The common approach of validating and completing Let's Encrypt SSL generation for private deployments is `DNS01 challenge validation `_. +If the ``DNS01`` does not work for you either, Cert-Manager supports a number of certificate issuers, get familiar with all types of issuers `here `_. + +In order to install Netris Controller with the custom SSL issuer, you need to run installation script with the specified host name: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com + +Once the installation is complete, create a yaml file with the ``ClusterIssuer`` resource, suitable for your requirements, and apply it: + +.. code-block:: shell-session + + kubectl apply -f my-cluster-issuer.yaml + +Then rerun the installation script with the ``--ctl-ssl-issuer`` argument: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-ssl-issuer + + +Upgrading +========= + +To upgrade the Netris Controller to the latest version simply run the script: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +If a newer version of Netris Controller is available, it will be updated in a few minutes. + + +Uninstalling +============ + +To uninstall Netris Controller and K3s from a server node, run: + +.. code-block:: shell-session + + /usr/local/bin/k3s-uninstall.sh + + +.. _ctl-backup-restore: + +Backup and Restore +================== + +Netris Controller stores all critical data in MariaDB. It's highly recommended to create a cronjob with ``mysqldump``. + + +Backup +------ + +To take database snapshot run the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot-$(date +%Y-%m-%d-%H-%M-%S).sql + +After command execution, you can find ``db-snapshot-YYYY-MM-DD-HH-MM-SS.sql`` file in the current working directory. + +.. _ctl-secret-key-backup: + + + +Backup the Secret Key +~~~~~~~~~~~~~~~~~~~~~ + +Netris Controller generates a unique secret key at the first installation. If you're moving or reinstalling your controller, it makes sense to take note of the secret key for restoring purpose in the future. Overwise, you have to reinitiate all devices connected to the controller. + +.. code-block:: shell-session + + kubectl -n netris-controller get secret netris-controller-grpc-secret -o jsonpath='{.data.secret-key}{"\n"}' + + +Restore +------- + +In order to restore DB from a database snapshot, follow these steps: + +1. Drop the current database by running the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"' + +2. Create a new database: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"' + +3. Copy snapshot file to the MariaDB container: + +.. code-block:: shell-session + + kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql + +4. Run the restore command: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql' + +.. note:: + + In this example the snapshot file name is db-snapshot.sql and it's located in the current working directory + + +Restore the Secret Key +~~~~~~~~~~~~~~~~~~~~~~ + +If you want to restore the controller secret key too (you might want to do that if you're reinstalling or moving the controller to the other place), follow these steps: + +1. Set ``OLD_SECRET`` environment variable (the secret key taken from :ref:`the old controller`): + +.. code-block:: shell-session + + export OLD_SECRET= + +example: ``export OLD_SECRET=VUdodFFSakJCU2lFVVA4T1c0cnpuUmdiMkQxem85Y2dnS3pkajlNSg==`` + +2. Update the secret key of the new controller: + +.. code-block:: shell-session + + kubectl -n netris-controller patch secret netris-controller-grpc-secret --type='json' -p='[{"op" : "replace" ,"path" : "/data/secret-key" ,"value" : "'$OLD_SECRET'"}]' + +3. Restart Netris Controller's all microservices + +.. code-block:: shell-session + + kubectl -n netris-controller rollout restart deployments diff --git a/en/latest/_sources/controller-k8s-installation.rst.txt b/en/latest/_sources/controller-k8s-installation.rst.txt new file mode 100644 index 0000000000..b88d540a63 --- /dev/null +++ b/en/latest/_sources/controller-k8s-installation.rst.txt @@ -0,0 +1,57 @@ + +.. meta:: + :description: Controller Helm Chart Installation + +####################### +Helm Chart Installation +####################### + +Requirements +------------ + +* Kubernetes 1.12+ +* Helm 3.1+ +* PV provisioner support in the underlying infrastructure + +Get Repo Info +------------- + +Add the Netris Helm repository: + +.. code-block:: shell-session + + helm repo add netrisai https://netrisai.github.io/charts + helm repo update + +Installing the Chart +-------------------- + +In order to install the Helm chart, you must follow these steps: + +1. Create the namespace for netris-controller: + +.. code-block:: shell-session + + kubectl create namespace netris-controller + +1. Install helm chart with netris-controller: + +.. code-block:: + + helm install netris-controller netrisai/netris-controller \ + --namespace netris-controller \ + --set ingress.hosts={my.domain.com} + +Uninstalling the Chart +--------------------------- + +To uninstall/delete the ``netris-controller`` helm release: + +.. code-block:: + + helm uninstall netris-controller + +Chart Configuration +------------------- + +See the `netris-controller README `_ for details about configurable parameters and their default values. diff --git a/en/latest/_sources/controller-k8s-quickstart.rst.txt b/en/latest/_sources/controller-k8s-quickstart.rst.txt new file mode 100644 index 0000000000..09c3eeea16 --- /dev/null +++ b/en/latest/_sources/controller-k8s-quickstart.rst.txt @@ -0,0 +1,41 @@ +.. meta:: + :description: Controller Quickstart + +*********************** +Quickstart Installation +*********************** + +Netris offers a simplified deployment model for users who want to quickly install the Netris Controller in the shortest amount of time. + +**This installation process is streamlined for Linux servers that do not already have Kubernetes running.** The install does the following: + +* Installs `k3s `_ +* Installs the `Netris Controller Helm chart `_ + +If you wish to install the controller on an existing Kubernetes cluster, follow `these instructions `_ instead of this Quickstart. + +Quickstart Process +------------------ + +1. Install the Netris Controller w/ k3s by running the following command on your Linux server: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +2. When the installation completes, you will be provided with the login for the web UI. Login with the provided credentials. + +3. Navigate in the UI to Net→IPAM and add a new subnet that contains the desired management IP addresses you wish to use for your SoftGates and switches. + + For example, if you are planning on using 192.168.1.100 as the IP address of your Ubuntu server, then create a subnet in Netris UI for 192.168.1.0/24. + + Detailed configuration documentation is available here: `Netris IPAM `_. + +4. Navigate in the UI to **Topology** +5. Click the **Add** in the upper right +6. Fill out the fields for the SoftGate you wish to add +7. Select the proper **Management IP address** from the subnet selector +8. Once the SoftGate is created in the Topology, **right-click** on the SoftGate and select the **Install Agent** option +9. Copy the agent install command to your clipboard and run the command on the Ubuntu server you are using as your SoftGate +10. Congratulations. The SoftGate should now be connected to your controller. + diff --git a/en/latest/_sources/controller-vm-installation.rst.txt b/en/latest/_sources/controller-vm-installation.rst.txt new file mode 100644 index 0000000000..73b989dde7 --- /dev/null +++ b/en/latest/_sources/controller-vm-installation.rst.txt @@ -0,0 +1,191 @@ +.. meta:: + :description: Controller Virtual Machine Installation + +**************************** +Virtual Machine Installation +**************************** + +Requirements +============ + +Minimal system requirements for the VM: + +* CPU - 4 Core +* RAM - 4 Gb +* Disk - 100Gb +* Network - 1 virtual NIC + +Recommended system requirements for the VM: + +* CPU - 8 Core +* RAM - 16 Gb +* Disk - 100Gb +* Network - 1 virtual NIC + +KVM Hypervisor Installation +=========================== +If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04) + +.. code-block:: shell-session + + sudo apt-get install virt-manager + +VM Controller Installation +========================== + +1. Download the Netris Controller image. (contact Netris support for repository access permissions). + +.. code-block:: shell-session + + cd /var/lib/libvirt/images + + sudo wget http://img.netris.io/netris-controller3.qcow2 + +2. Download VM definition file. + +.. code-block:: shell-session + + cd /etc/libvirt/qemu + + sudo wget http://img.netris.io/netris-controller3.xml + +3. Define the KVM virtual machine + +.. code-block:: shell-session + + sudo virsh define netris-controller3.xml + +.. note:: + + Netris Controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below for the network interface configuration example. + +Example: Network configuration on host (hypervisor) machine. + +.. note:: + + replace , and + with the correct NIC and IP for your host machine. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + #Physical NIC connected to the management network + auto + iface inet static + address 0.0.0.0/0 + + #bridge interface + auto br-mgmt + iface br-mgmt inet static + address + gateway + bridge-ports + + source /etc/network/interfaces.d/* + +.. code-block:: shell-session + + sudo ifreload -a + +4. Set the virtual machine to autostart and start it. + +.. code-block:: shell-session + + sudo virsh autostart netris-controller + +.. code-block:: shell-session + + sudo virsh start netris-controller + +Accessing the Netris Controller +=============================== +By default, Netris Controller will obtain an IP address from a **DHCP** server. + +Below steps describe how to configure a **Static IP** address for the Netris Controller. + +1. Connecting to the VM console. + +default credentials. **login**: ``netris`` **password**: ``newNet0ps`` + +.. code-block:: shell-session + + sudo virsh console netris-controller + +.. note:: + + Do not forget to change the default password (using passwd command). + +2. Setting a static IP address. + +Edit network configuration file. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +Example: IP configuration file. + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + + # The primary network interface + auto eth0 + iface eth0 inet static + address + gateway + dns-nameserver + + source /etc/network/interfaces.d/* + +Reload the network config. + +.. code-block:: shell-session + + sudo ifreload -a + +.. note:: + + Make sure Netris Controller has Internet access. + +3. Reboot the controller + +.. code-block:: shell-session + + sudo reboot + +After reboot, the Netris Controller GUI should be accessible using a browser. Use ``netris/newNet0ps`` credentials. + +.. image:: images/credentials.png + :align: center + :class: with-shadow + :alt: Netris Credentials + +.. note::Don’t forget to change the default password by clicking your login name in the top right corner and then clicking “Change Password”. + +Replacing the SSL certificate +============================= + +1. Replace the below file with your SSL certificate file. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.cert.pem; + +2. Replace the below file with your SSL private key. + +.. code-block:: shell-session + + /etc/nginx/ssl/controller.key.pem; + +3. Restart Nginx service. + +.. code-block:: shell-session + + systemctl restart nginx.service diff --git a/en/latest/_sources/definitions.rst.txt b/en/latest/_sources/definitions.rst.txt new file mode 100644 index 0000000000..678e8be73e --- /dev/null +++ b/en/latest/_sources/definitions.rst.txt @@ -0,0 +1,28 @@ +.. meta:: + :description: Definitions + +=========== +Definitions +=========== + +When configuring and operating a Netris system, the following nomenclature is important to understand: + +* **User** - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is ``netris``, with password ``newNet0ps``. + +* **The Netris VPC** - logically segregated virtual network.The VPC acts as a VRF in traditional networking, providing the flexibility to employ overlapping IP ranges across various VPCs while maintaining secure management and operation of resources. + +* **Tenant** - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request and manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs or Terraform) via a DevOps/NetOps pipeline. + +* **Permission Group** - List of permissions on a per section basis can be attached individually to a User or a User Role. + +* **User Role** - Group of user permissions and tenants for role-based access control (RBAC). + +* **Site** - Each separate deployment (each data center) should be defined as a Site. All network units and resources are attached to a site. Site entry defines global attributes such as; AS numbers, default ACL policy, Site Mesh (site to site VPN), and other site-level parameters. + +* **Subnet** - IPv4/IPv6 address resources linked to *Sites* and *Tenants* + +* **Switch Port** - Physical ports of all switches attached to the system, or server endpoints in a Bare Metal Cloud environment. + +* **Inventory** - Inventory of all network units that are operated using Netris Agent. + +* **E-BGP** - Defines all External BGP peers (iBGP and eBGP). diff --git a/en/latest/_sources/index.rst.txt b/en/latest/_sources/index.rst.txt new file mode 100644 index 0000000000..a8b7f48b4d --- /dev/null +++ b/en/latest/_sources/index.rst.txt @@ -0,0 +1,104 @@ +.. Read the Docs Template documentation master file, created by + sphinx-quickstart on Tue Aug 26 14:19:49 2014. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +Welcome to Netris Documentation +================================================== + +Learn how to get started with Netris VPC Networking for your network environment. + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + +.. toctree:: + :maxdepth: 2 + :caption: General Information + + definitions + introduction + supported-platform-matrix + supported-switch-hardware + netris-architecture + +.. toctree:: + :maxdepth: 2 + :caption: Try & Learn Netris + + try-learn/index + +.. toctree:: + :maxdepth: 2 + :caption: Tutorials + + vpc + tutorials/index + tutorials/vpc-anywhere + tutorials/netris-vpc-for-equinix-metal + tutorials/netris-vpc-for-phoenixnap-bmc + tutorials/netris-vpc-for-aws + tutorials/netris-vpc-for-gcp + tutorials/vpc-gateways-with-managed-fabric.rst + + + +.. toctree:: + :maxdepth: 4 + :caption: Cloud Native Tools + + kubernetes-integration + terraform-integration + + +.. toctree:: + :maxdepth: 4 + :caption: Network Services + + vnet + l3-load-balancer + l4-load-balancer + acls + roh + +.. toctree:: + :maxdepth: 4 + :caption: Network Policies + + network-policies + + +.. toctree:: + :maxdepth: 4 + :caption: Operations + + accounts + maintenance-mode + tutorials/upgrading-netris + visibility + + +.. toctree:: + :maxdepth: 2 + :caption: Switch-fabric Management + + topology-management + switch-ports + lag + +.. toctree:: + :maxdepth: 2 + :caption: Detailed Installation + + installation + switch-agent-installation + SoftGate-installation + SoftGate-PRO-installation + + +.. toctree:: + :maxdepth: 4 + :caption: Miscellaneous + + softgate-performance + + + sandbox/Sandbox1/onprem-k8s diff --git a/en/latest/_sources/installation.rst.txt b/en/latest/_sources/installation.rst.txt new file mode 100644 index 0000000000..960bd2bf9f --- /dev/null +++ b/en/latest/_sources/installation.rst.txt @@ -0,0 +1,21 @@ +.. meta:: + :description: Controller Installation + +======================= +Controller Installation +======================= + +Netris Controller can be installed locally as a VM, deployed as a Kubernetes application, or hosted in the Netris Cloud. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime. + +.. note:: + + Select ONE installation option below. + + +.. toctree:: + :maxdepth: 2 + :caption: Controller Installation + + controller-k3s-installation + controller-k8s-installation + Controller-local-repository diff --git a/en/latest/_sources/installing-netris-controller.rst.txt b/en/latest/_sources/installing-netris-controller.rst.txt new file mode 100644 index 0000000000..cc7db17230 --- /dev/null +++ b/en/latest/_sources/installing-netris-controller.rst.txt @@ -0,0 +1,37 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Installing a Netris Controller +============================== + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment. + +It doesn't matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. So you can access the console, and nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface. + +Linux Host requirements + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +In this example I am running my Netris controller on an AWS hosted virtual machine (EC2) which has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +I'm using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my EC2 : 54.219.211.71. + +.. image:: images/cloudflare-dns-record.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-hostname ” will instruct the installer to generate a Let’s Encrypt SSL certificate for the provided domain name. That’s why it is important to create the DNS record before this step. + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com + +Once installation process is finished you will be able to access your newly installed Netris Controller web consonle using netris/newNet0ps credentials. + +Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller. diff --git a/en/latest/_sources/introduction.rst.txt b/en/latest/_sources/introduction.rst.txt new file mode 100644 index 0000000000..af5b6d2c4d --- /dev/null +++ b/en/latest/_sources/introduction.rst.txt @@ -0,0 +1,11 @@ +.. meta:: + :description: Introduction to Netris + +Introduction to Netris +====================== + +Netris is a network automation and abstraction software for cloud builders. Netris brings cloud-like VPC abstractions for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network’s health and either applies software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation. + +.. image:: images/private-cloud-enterprise-dc-2.png + :align: center + diff --git a/en/latest/_sources/inventory-profiles.rst.txt b/en/latest/_sources/inventory-profiles.rst.txt new file mode 100644 index 0000000000..98737f45a5 --- /dev/null +++ b/en/latest/_sources/inventory-profiles.rst.txt @@ -0,0 +1,26 @@ +.. meta:: + :description: Inventory Profiles + +================== +Inventory Profiles +================== + +Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/softgate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except netris-defined and user-defined custom flows. Generated rules include: + +* SSH from user defined subnets +* NTP from user defined ntp services +* DNS from user defined DNS servers +* Custom user defined rules + +.. csv-table:: Inventory Profile Fields + :file: tables/inventory-profile-fields.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup). + +.. image:: images/inventory-profile.png + :align: center + :class: with-shadow + :alt: Adding an inventory profile diff --git a/en/latest/_sources/ipam.rst.txt b/en/latest/_sources/ipam.rst.txt new file mode 100644 index 0000000000..0191d2a4a7 --- /dev/null +++ b/en/latest/_sources/ipam.rst.txt @@ -0,0 +1,96 @@ +.. meta:: + :description: IP Address Management + +.. _ipam_def: + +============================ +IP Address Management (IPAM) +============================ + +Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting. + +Purpose: +Users define specific roles(purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-net, NAT, etc… + + +Allocations and Subnets +----------------------- + +There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets. + + +.. image:: images/subnet-tree.png + :align: center + :alt: IPAM Tree View + :class: with-shadow + +IPAM Tree View + +-------------------------- + +Add an Allocation +----------------- + +#. Navigate to Net→IPAM +#. Click the **Add** button +#. Select **Allocation** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Allocation Fields + :widths: 25 50 + :header-rows: 0 + + * - Name + - Unique name for current allocation. + * - Prefix + - Unique prefix for allocation, must not overlap with other allocations. + * - Tenant + - Owner of the allocation. + +.. image:: images/add-allocation.png + :align: center + :class: with-shadow + :alt: Add a New IP Allocation + +Add Allocation Window + +-------------------------- + +Add a Subnet +------------ + +#. Navigate to Net→IPAM +#. Click the **Add** button +#. Select **Subnet** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Subnet fields + :widths: 25 50 + :header-rows: 0 + + * - **Name** + - Unique name for current subnet. + * - **Prefix** + - Unique prefix for subnet, ust be included in one of allocations. + * - **Tenant** + - Owner of the subnet. + * - **Purpose** + - This field describes for what kind of services the current subnet can be used. It can have the following values: + + - *common* - ordinary subnet, can be used in v-nets and ROH. + - *loopback* - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates). + - *management* - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates). + - *load-balancer* - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience. + - *nat* - hosts of this subnet or subnet itself can be used to define NAT services. + - *inactive* - can't be used in any services, useful for reserving/documenting prefixes for future use. + +.. image:: images/add-subnet.png + :align: center + :alt: Add a New Subnet + :class: with-shadow + +Add Subnet Window diff --git a/en/latest/_sources/kubernetes-integration.rst.txt b/en/latest/_sources/kubernetes-integration.rst.txt new file mode 100644 index 0000000000..94f6d2ea76 --- /dev/null +++ b/en/latest/_sources/kubernetes-integration.rst.txt @@ -0,0 +1,477 @@ +.. meta:: + :description: Kubernetes Integration + +######################## +Kubernetes Integration +######################## +Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris-Kubernetes integration is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters. + + +Install Netris Operator +======================= + +Integration between the Netris Controller and the Kubernetes API is completed by installing the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart: + +Helm Chart Method +----------------- +Instructions are available on Github: https://github.com/netrisai/netris-operator/tree/master/deploy/charts/netris-operator#installing-the-chart + +Regular Manifest Method +----------------------- + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='http://**your-netris-controller-ip-or-host**' \ + --from-literal=login='**your-netris-admin-username**' --from-literal=password='**your-netris-admin-password**' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Using Type 'LoadBalancer' +========================= + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services / L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :class: with-shadow + :alt: Sandbox pod provisioning + + +After provisioning has finished, inspect the service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.202 9898:32584/TCP,9999:30365/TCP 9m17s + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo’s service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9898 active 50.117.59.202 9898/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 32m + +You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9898 active 50.117.59.202 9898/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.202 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.203 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.203 + +You will see the servers’ hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :class: with-shadow + :alt: View L4 LB instances + +V-Net Custom Resource +--------------------- + +You can also create Netris V-Nets (L2 segments) via Kubernetes with a simple manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let’s check our VNet resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new VNet has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them. + +BGP Custom Resource +------------------- + +You can create BGP peers via Kubernetes manifests: + +1. Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1092 + localIP: 50.117.59.118/30 + remoteIP: 50.117.59.117/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.192/28 le 32 + EOF + +2. Apply the manifest file: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +3. Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.118/30 50.117.59.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.118/30 50.117.59.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the controller web interface previously. + +Return to the Netris UI and navigate to **Net / Topology** to see the new BGP neighbor you created. + +Importing existing resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them w/out "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Calico CNI Integration +====================== + +Netris Operator can integrate with Calico CNI. This annotation will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, just need to add the annotation in calico’s ``bgpconfigurations`` custom resource. Before doing that, let’s see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it’s Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let’s enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let’s check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.118/30 50.117.59.117/30 7m59s + sandbox9-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox9-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox9-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + + +You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As seen our BGP peers are established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.118/30 50.117.59.117/30 8m41s + sandbox9-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox9-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox9-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let’s check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established. + +Finally, let’s check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.202 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico’s ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. diff --git a/en/latest/_sources/l3-load-balancer.rst.txt b/en/latest/_sources/l3-load-balancer.rst.txt new file mode 100644 index 0000000000..99e9ef59a8 --- /dev/null +++ b/en/latest/_sources/l3-load-balancer.rst.txt @@ -0,0 +1,44 @@ +.. meta:: + :description: Layer-3 Load Balancer (Anycast) + +.. _l3lb_def: + +############################# +L3 Load Balancer (Anycast LB) +############################# +L3 (Anycast) load balancer leverages ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks. + +ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server. + +End-user traffic should be destined to the anycast IP address. The switch fabric uses ECMP to load balance the traffic towards every server, and will hash sessions based on IP/Protocol/Port such that TCP sessions will exist between the given end-user and server pair for the lifetime of the session. Optional health checks are used to identify application failures and reroute traffic in the case of a server outage. + +Creating an L3 Load Balancer +============================ + +To configure L3 (Anycast) load balancing: + +#. Navigate to **Services→Instances (ROH)** and locate an existing ROH instance +#. Click the ellipses and then the **Edit** button +#. Select an extra IPv4 address from the select box at the bottom, and check the **Anycast** option. +#. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances. + +.. image:: images/add-l3-lb.png + :align: center + :class: with-shadow + :alt: Add a L3 LB + +Example: Adding an Anycast IPv4 address + +.. image:: images/list-l3-lb.png + :align: center + :class: with-shadow + :alt: List L3 LBs + +Example: Under Services→Load Balancer, you can find the listing of L3 (Anycast) Load Balancers, service statuses, and you can add/remove more ROH instances and/or health checks. + +.. image:: images/list-l3-lb-detail.png + :align: center + :class: with-shadow + :alt: List L3 LB Details + +Screenshot: L3 (Anycast) Load Balancer Detail diff --git a/en/latest/_sources/l4-load-balancer.rst.txt b/en/latest/_sources/l4-load-balancer.rst.txt new file mode 100644 index 0000000000..63de30d5fc --- /dev/null +++ b/en/latest/_sources/l4-load-balancer.rst.txt @@ -0,0 +1,84 @@ +.. meta:: + :description: Netris Services and Configuration Examples + +.. _l4lb_def: + +####################### +L4 Load Balancer (L4LB) +####################### +Netris L4 Load Balancer (L4LB) leverages SoftGate(Linux router) nodes to provide Layer-4 load balancing services, including on-demand cloud load balancing with native integration with Kubernetes. + +Enabling L4LB service +--------------------- +L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer). + +The IP address pool for L4LB can be defined in the Net→IPAM section by adding an Allocation and setting the purpose field to ‘load-balancer’. You can define multiple IP pools for L4LB at any given site. See the below example. + +Example: Adding a load-balancer IP pool assignment. + +.. image:: images/add-allocation.png + :align: center + :class: with-shadow + :alt: Add an IP Allocation + + +Screenshot: Listing of Net→IPAM after adding a load-balancer assignment + +.. image:: images/list-subnets.png + :align: center + :class: with-shadow + :alt: List IP Subnets + + +Consuming L4LB service +---------------------- +This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section. + +Click +add under Services→L4 Load Balancer to request an L4LB service. + +Add new L4 Load Balancer fields are described below: + +**General fields** + +* **Name** - Unique name. +* **Protocol** - TCP or UDP. +* **Tenant** - Requestor Tenant should have access to the backend IP space. +* **Site** - Site where L4LB service is being requested for. Backends should belong on this site. +* **State** - Administrative state. + +**Frontend** + +* **Address** - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses. +* **Port** - TCP or UDP port to be exposed. + +**Health-check** + +* **Type** - Probe backends on service availability. + + * **None** - load balance unconditionally. + * **TCP** - probe backend service availability through TCP connect checks. + * **HTTP** - probe backend service availability through HTTP GET checks. + +* **Timeout(ms)** - Probe timeout in milliseconds. +* **Request path** - HTTP request path. + +**Backend** + +* **+Add** - add a backend host. +* **Address** - IP address of the backend host. +* **Port** - Service port on the backend host. +* **Enabled** - Administrative state of particular backend. + +.. image:: images/request-L4.png + :align: center + :class: with-shadow + :alt: Request an L4 Load Balancer + +Example: Requesting an L4 Load Balancer service. + +.. image:: images/list-l4-load-balancers.png + :align: center + :class: with-shadow + :alt: List L4 Load Balancers + +Example: Listing of L4 Load Balancer services \ No newline at end of file diff --git a/en/latest/_sources/lag.rst.txt b/en/latest/_sources/lag.rst.txt new file mode 100644 index 0000000000..e0d854069d --- /dev/null +++ b/en/latest/_sources/lag.rst.txt @@ -0,0 +1,86 @@ +.. meta:: + :description: Link Aggregation + +====================== +Link Aggregation (LAG) +====================== + +Link Aggregation (LAG), also known as link bundling, Ethernet/network/NIC bonding, or port teaming, is a method of combining (aggregating) multiple network connections in parallel to increase throughput beyond what a single connection could sustain and to provide redundancy in case one of the links fails. + +.. image:: images/lag_diagram.png + :align: center + :alt: LAG diagram + + +The Link Aggregation Control Protocol (LACP) is a key component of LAG. It's a protocol for the collective handling of multiple physical ports that can be seen as a single channel for network traffic. + +Netris supports Automatic and Custom modes of configuring LAGs, each described below. + + +Automatic LAG with EVPN Multi-homing +------------------------------------ + +If LACP is configured on the server side, then Netris will automatically identify the corresponding switch port pairs and form a LAG with EVPN Multihoming enabled for any switch ports participating in a V-Net service. + +EVPN Multi-Homing (EVPN-MH) offers robust support for an all-active redundancy model for servers. This means that all connections from a server to multiple switches are concurrently active and operational, ensuring high availability, load balancing, and seamless failover capabilities. As a result, EVPN-MH enhances network resilience and continuity of service. + +.. image:: images/lag_diagram2.png + :align: center + :alt: EVPN-MH diagram + + +**Note: Active Active Multi-homing is currently limited to:** + +* Switch OS Cumulus 5.3 or higher. +* Netris 4.0 or higher. +* One port per switch (can be overcome in Custom LAG). +* Only two switches in EVPN-MH domain with ASIC Spectrum A1. + +When you add switch ports to a V-Net service, Netris agents automatically configure LAG and apply LACP with EVPN-MH and LACP fallback. If Active-Active Multi Homing is not supported on your hardware, Active-Standby LACP LAG will be configured. + +Both in Active-Active and Active-Standby cases LACP should be configured on the server side. If the server does not have LACP configured, the LAG will fallback into non LACP mode and one of two links will still remain active. + + +To create a V-Net with EVPN-MH go to Service → V-Net → +Add + +.. image:: images/lag_add_vnet.png + :align: center + :alt: Add LAG to V-Net + + + +Custom LAG +---------- + +There are several use cases when you may need a Custom (manually configured) LAG. + +* Aggregate more than one port per each switch (with or without EVPN). +* LACP fallback is not sufficient and needs to be unconditionally disabled on the switch side. EVPN-MH will be deactivated in this case. + +.. image:: images/lag_diagram3.png + :align: center + :alt: Custom LAG diagram + +Please note that a Custom LAG can only be created with one switch. However, if you add two or more Custom LAGs to a V-Net, EVPN-MH will be automatically activated. + +To create a Custom LAG go to Network → Network Interfaces. + +.. image:: images/lag_add_lag.png + :align: center + :alt: Add Custom LAG + +Add a new LAG. + +.. image:: images/lag_add_lag2.png + :align: center + :alt: Add Custom LAG + :width: 543 + :height: 337 + +Set necessary options. + +.. image:: images/lag_add_lag3.png + :align: center + :alt: Add Custom LAG + :width: 543 + :height: 337 diff --git a/en/latest/_sources/maintenance-mode.rst.txt b/en/latest/_sources/maintenance-mode.rst.txt new file mode 100644 index 0000000000..90a037eada --- /dev/null +++ b/en/latest/_sources/maintenance-mode.rst.txt @@ -0,0 +1,45 @@ +.. meta:: + :description: Netris System Visibility, Monitoring & Telemetry + +********************** +Maintenance Mode +********************** + +Overview +======== +Maintenance mode is intended to assist in smoothly redirecting traffic away from a particular device for a maintenance to be carried out with minimal impact on your network. It's advisable to activate Maintenance Mode, wait a few minutes and ensure that traffic has been re-routed, prior to initiating maintenance procedures on the device. Once the maintenance is completed, you should deactivate Maintenance Mode to switchover the traffic back to normal. +To toggle Maintenance Mode on or off, navigate to the Inventory section, or Topology manager (more convenient for switch-fabric). Edit the devices, and use the Maintenance Mode checkbox to enable/disable. + +.. image:: images/maintenance-mode.png + :align: center + +.. note:: + Maintenance mode strives to redirect traffic away from the current device; however, please note that it does not guarantee the complete offloading of traffic. + +Maintenance Mode for Softgate - What’s happening behind the scenes? +=================================================================== + +When you activate Maintenance Mode for the softgate, several automatic actions are undertaken behind the scenes to redirect traffic away from the softgate: + + - The BGP local preference attribute is lowered for both external and internal peers. + - Route information is prepended tenfold for outbound direction to all external and internal peers. + - The BGP MED attribute is increased for all external and internal peers. + - The BGP origin attribute is decreased for all external and internal peers. + - Connection-oriented services such as SNAT and L4LB will be transferred to the second softgate, resulting in a reestablishment of TCP connections. + +.. note:: + Before activating the maintenance mode, make sure that the second softgate is operating correctly. + +Maintenance Mode for Switch - What’s happening behind the scenes? +================================================================= + +When you activate Maintenance Mode for the switch, several automatic actions are undertaken behind the scenes to redirect traffic away from the switch: + + - The BGP local preference attribute is lowered for both external and internal peers. + - Route information is prepended tenfold for outbound direction to all external and internal peers. + - The BGP MED attribute is increased for all external and internal peers. + - The BGP origin attribute is decreased for all external and internal peers. + - The LACP system ID will undergo modification for EVPN Multihomed hosts, resulting in a reorganization of traffic from ports that connect to these multihomed hosts. The traffic will be redirected to other switches where the hosts are also linked. + +.. note:: + If a host is connected to two switches, and the first switch is in Maintenance Mode, if the link connected to the second switch experiences instability (link flapping), the connection linked to the first switch will become active. However, this connection will not automatically revert back to the second switch once it stabilizes (no preemption). diff --git a/en/latest/_sources/netris-architecture.rst.txt b/en/latest/_sources/netris-architecture.rst.txt new file mode 100644 index 0000000000..6fdabb01a8 --- /dev/null +++ b/en/latest/_sources/netris-architecture.rst.txt @@ -0,0 +1,55 @@ +.. meta:: + :description: Netris Architecture + +.. _netris_architecture: + +################### +Netris Architecture +################### + +A Netris system is composed of 3 elements: + +* Netris Controller +* Netris Switch Agent +* Netris SoftGate + +.. _netris_controller_def: + +Netris Controller +================= + +Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. The Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes, Terraform, etc.). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud. + +Diagram: High level Netris architecture + +.. image:: images/netris_controller_diagram.png + :align: center + +* **Controller HA** We highly recommend running more than one copy of the controller for database replication. Find here :doc:`backup and restore` procedure. +* **Multiple sites** Netris is designed to operate multiple sites with just a single controller with HA +* **What if the controller is unreachable.** Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers core operations will not be affected. + +.. _netris_sw_agent: + +Netris Switch Agent +=================== + +Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet. + +.. _netris_sg_agent: + +Netris SoftGate +=============== + +Netris SoftGate is a software for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), DHCP, Firewall, and site-to-site VPN function on a regular x86 server with a SmartNIC (optionally) card. + +Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2. + +The SoftGate PRO (the 100Gbps version) server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities. + +Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels. + +Diagram: Netris SoftGate high level architecture. + +.. image:: images/softgate_diagram.png + :align: center diff --git a/en/latest/_sources/network-policies.rst.txt b/en/latest/_sources/network-policies.rst.txt new file mode 100644 index 0000000000..95ba16fa5c --- /dev/null +++ b/en/latest/_sources/network-policies.rst.txt @@ -0,0 +1,445 @@ +.. meta:: + :description: Netris Network Policies & Protocol Configuration + +### +VPC +### + +Netris Controller is preconfigured with a default system VPC-1. Use the default VPC, and create additional VPCs as needed in the future. + +The VPC acts as a VRF in traditional networking, providing the ability to use overlapping IP ranges across various VPCs while maintaining safe management and operation of services. + +VPC can be created in the Network → VPC section. + +Adding new VPC +-------------- + +1. Navigate to Network → VPC in the web UI. +2. Click Add button. + +.. image:: images/vpc_empty.png + :align: center + +.. _ipam_def_vpc: + + +##################### +IP Address Management +##################### + +Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting. + +Purpose: Users define specific roles (purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-Net, NAT, etc… + +Each VPC has its own IPAM table. + + +Allocations and Subnets +----------------------- + +There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets. + + +.. image:: images/ipam_tree_new.png + :align: center + :alt: IPAM Tree View + :class: with-shadow + +IPAM Tree View +-------------------------- + +Add an Allocation +----------------- + +#. Navigate to Network → IPAM +#. Click the **Add** button +#. Select **Allocation** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Allocation Fields + :widths: 25 50 + :header-rows: 0 + + * - Prefix + - Unique prefix for allocation, must not overlap with other allocations. + * - Name + - Unique name for current allocation. + * - VPC + - Select a VPC to which the allocation belongs. + * - Tenant + - Owner of the allocation. + +.. image:: images/allocation_empty.png + :align: center + :class: with-shadow + :alt: Add a New IP Allocation + +Add Allocation Window + +-------------------------- + +Add a Subnet +------------ + +#. Navigate to Network → IPAM +#. Click the **Add** button +#. Select **Subnet** from the bottom select box +#. Fill in the rest of the fields based on the requirements listed below +#. Click the **Add** button + + +.. list-table:: Subnet fields + :widths: 25 50 + :header-rows: 0 + + * - **Prefix** + - Unique prefix for subnet, ust be included in one of allocations. + * - **Name** + - Unique name for current subnet. + * - **VPC** + - Select a VPC to which the subnet belongs. + * - **Tenant** + - Owner of the subnet. + * - **Purpose** + - This field describes for what kind of services the current subnet can be used. It can have the following values: + + - *common* - ordinary subnet, can be used in v-nets and ROH. + - *loopback* - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates). + - *management* - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates). + - *load-balancer* - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience. + - *nat* - hosts of this subnet or subnet itself can be used to define NAT services. + - *inactive* - can't be used in any services, useful for reserving/documenting prefixes for future use. + +.. image:: images/subnet_empty.png + :align: center + :alt: Add a New Subnet + :class: with-shadow + +Add Subnet Window + + +.. _bgp_def: + +######### +Basic BGP +######### + +BGP neighbors can be declared in the Network → E-BGP section. Netris software will automatically generate and program the network configuration to meet the requirements. + +Adding BGP Peers +---------------- +#. Navigate to Network → E-BGP in the web UI. +#. Click the Add button. +#. Fill in the fields as described in the table below. +#. Click the Add button. + + +.. .. csv-table:: BGP Peer Fields +.. :file: tables/bgp-basic.csv +.. :widths: 25, 75 +.. :header-rows: 0 + +Example: Declare a basic BGP neighbor. + +.. image:: images/bgp_empty.png + :align: center + +############ +Advanced BGP +############ + +BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies. + +Click Advanced to expand the BGP neighbor add/edit window. + + +.. .. csv-table:: BGP Peer Fields - Advanced +.. :file: tables/bgp-advanced.csv +.. :widths: 25, 75 +.. :header-rows: 0 + +-------------------------- + +BGP Objects +----------- +| Under Network → E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy. +| Supported objects include: + +* IPv4 Prefix +* IPv6 Prefix +* AS-PATH +* Community +* Extended Community +* Large Community + +IPv4 Prefix +^^^^^^^^^^^ +| The rules are defined one per line. +| Each line in IPv4 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv4 prefix (mandatory). +* Length - Possible values are: le , ge or ge le . + +Example: Creating an IPv4 Prefix list. + +.. image:: images/ipv4_prefix.png + :align: center + +IPv6 Prefix +^^^^^^^^^^^ +| Rules defined one per line. +| Each line in IPv6 prefix list field consists of three parts: + +* Action - Possible values are: permit or deny (mandatory). +* IP Prefix - Any valid IPv6 prefix (mandatory). +* Keyword - Possible values are: le , ge or ge le . + +Example: Creating an IPv6 Prefix list. + +.. image:: images/ipv6_prefix.png + :align: center + +Community +^^^^^^^^^ +| Community field has two parts: + +* Action - Possible values: permit or deny (mandatory). +* Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive). + +Example: Creating community. + +.. image:: images/community.png + :align: center + +-------------------------- + +BGP route-maps +-------------- +| Under the Network → E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound. + +| Description of route-map fields: + +* **Sequence Number** - Automatically assigned a sequence number. Drag and move sequences to organize the order. +* **Description** - Free description. +* **Policy** - Permit or deny the routes which match below all match clauses within the current sequence. +* **Match** - Rules for route matching. + + * **Type** - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag. + * **Object** - Select an object from the list. + +* **Action** - Action when all match clauses are met. + + * **Action type** - Define whether to manipulate a particular BGP attribute or go to another sequence. + * **Attribute** - The attribute to be manipulated. + * **Value** - New attribute value. + +Example: route-map + +.. image:: images/route-map.png + :align: center + :class: with-shadow + +-------------------------- + +############## +Static Routing +############## +Located under Network → Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. We recommend using the Routes only if BGP is not supported by the remote end. + + +| Typical use cases for static routing: + +* To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported. +* Temporary interconnection with the old network for a migration. +* Routing a subnet behind a VM hypervisor machine for an internal VM network. +* Specifically routing traffic destined to a particular prefix through an out-of-band management network. + +| Add new static route fields description: + +* **Prefix** - Route destination to match. +* **Next-Hop** - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network. +* **Description** - Free description. +* **VPC** - Select a VPC to which the static route belongs. +* **Site** - Site where Route belongs. +* **State** - Administrative (enable/disable) state of the Route. +* **Apply to** - Limit the scope to particular units. It's typically used for Null routes. + + +Example: Default route pointing to a Next-Hop that belongs to one of V-Nets. + +.. image:: images/static_route_empty.png + :align: center + +Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network. + +.. image:: images/static_route2_empty.png + :align: center + +Screenshot shows that the back route is actually applied on Softgate1 and Softgate2 . + +.. image:: images/static_route3.png + :align: center + +-------------------------- + +.. _nat_def: + +### +NAT +### + +Netris SoftGate nodes are required forNAT (Network Address Translation) functionality to work. + +**Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).** + +Enabling NAT +------------ +To enable NAT for a given site, you first need to create a subnet with NAT purpose in the IPAM section. The NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need. +Adding an IP Subnet under Network → IPAM. + + +1. Allocate a public IP subnet for NAT under Net→IPAM. + +Example: Adding an IP allocation under Net→Subnets. + +.. image:: images/nat_subnet_empty.png + :align: center + + +Defining NAT rules +------------------ +NAT rules are defined under Network → NAT. + +.. list-table:: NAT Rule Fields + :widths: 25 75 + :header-rows: 1 + + * - Name + - Unique name. + * - **State** + - State of rule (enabled or disabled). + * - **Site** + - Site to apply the rule. + * - **Action** + - *SNAT* - Replace the source IP address with specified NAT IP along with port overloading. + *DNAT* - Replace the destination IP address and/or destination port with specified NAT IP. + *ACCEPT* - Silently forward, typically used to add an exclusion to broader SNAT or DNAT rule. + *MASQUERADE* - Replace the source IP address with the IP address of the exit interface. + * - **Protocol** + - *All* - Match any IP protocol. + *TCP* - Match TCP traffic and ports. + *UDP* - Match UDP traffic and ports. + *ICMP* - Match ICMP traffic. + * - **Source** + - *Address* - Source IP address to match. + *Port* - Source ports range to match with this value (TCP/UDP). + * - **Destination** + - *Address* - Destination IP address to match. In the case of DNAT it should be one of the predefined NAT IP addresses. + *Port* - For DNAT only, to match a single destination port. + *Ports* - For SNAT/ACCEPT only. Destination ports range to match with this value (TCP/UDP). + * - **DNAT to IP** + - The global IP address for SNAT to be visible on the Public Internet. The internal IP address for DNAT to replace the original destination address with. + * - **DNAT to Port** + - The Port to which destination Port of the packet should be NAT'd. + * - **Status** + - Administrative state (enable/disable). + * - **Comment** + - Free optional comment. + + +Example: SNAT all hosts on 10.0.1.0/24subnet to the Internet using 192.0.2.128as a global IP. + +.. image:: images/create_snat_empty.png + :align: center + +Example: Port forwarding. DNAT the traffic destined to 192.0.2.130:8080 to be forwarded to the host 10.0.1.100 on port tcp/80. + +.. image:: images/create_dnat_empty.png + :align: center + +-------------------------- + +######## +SiteMesh +######## + +SiteMesh is a Netris service for site-to-site interconnection over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site. + +**Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).** + +Edit Network → Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below. + +* **Disabled** - Do not participate in SiteMesh. +* **Hub** - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites) +* **Spoke** - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites) +* **Dynamic Spoke** - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites) + +Screenshot: Site Mesh parameter editing a Site under Network → Sites. + +.. image:: images/sitemesh_edit.png + :align: center + +You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge. + +.. image:: images/SiteMesh_modes.png + :align: center + +Check the Network → Site Mesh section for the listing of tunnel statuses. + +Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh) + +.. image:: images/SiteMesh_listing.png + :align: center + +-------------------------- + +############# +Looking Glass +############# + +The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Network → Looking Glass then selecting the device from the top-left dropdown menu. + +Looking Glass controls described for IPv4/IPv6 protocol families. + +* **VPC** - select a VPC. +* **BGP Summary** - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes. +* **BGP Route** - Lookup the BGP table (RIB) for the given address. +* **Route** - Lookup switch routing table for the given address. +* **Traceroute** - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address. +* **Ping** - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address. + +Example: listing BGP neighbors of a switch and number of received prefixes for the Underlay VPC. + +.. image:: images/lg_summary.png + :align: center + +Example: BGP Route - looking up V-Net subnet from switch11 perspective. Switch11 is load balancing between four available paths. + +.. image:: images/lg_bgp_route.png + :align: center + +Example: Ping. + +.. image:: images/lg_ping.png + :align: center + +| Looking Glass controls described for the EVPN family. + +* **VPC** - select a VPC. +* **BGP Summary** - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received. +* **VNI** - List VNIs learned. +* **BGP EVPN** - List detailed EVPN routing information optionally for the given route distinguisher. +* **MAC table** - List MAC address table for the given VNI. + +Example: Listing MAC addresses on VNI 50. + +.. image:: images/lg_mac.png + :align: center + +Example: EVPN routing information listing for a specified route distinguisher. + +.. image:: images/lg_rd.png + :align: center diff --git a/en/latest/_sources/reference-designs.rst.txt b/en/latest/_sources/reference-designs.rst.txt new file mode 100644 index 0000000000..8c1b63a61d --- /dev/null +++ b/en/latest/_sources/reference-designs.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Network Reference Designs + +######################### +Network Reference Designs +######################### + +Netris can support any type of standard network design. + +The majority of designs fall into one of four patterns: + +#. Unmanaged Switch +#. Single L2 Domain +#. Collapsed Core +#. Data Center Leaf/Spine Designs (Modern High Throughput Redundant Design) +#. Data Center Core/Distribution/Access (Legacy High Throughput Redundant Design) \ No newline at end of file diff --git a/en/latest/_sources/release-notes.rst.txt b/en/latest/_sources/release-notes.rst.txt new file mode 100644 index 0000000000..ea6c869c70 --- /dev/null +++ b/en/latest/_sources/release-notes.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Netris Release Notes + +############# +Release notes +############# +* DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGate performance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on. +* L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, we now support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates with Kubernetes providing cloud-like load balancer service (type: load-balancer). +* Kubenet​ - a network service purpose-built for Kubernetes cluster nodes. Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking with modern physical networking. +* API logs​ - Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type. +* Site Mesh​ - a Netris service for automatically configuring site-to-site interconnect over the public Internet. Site Mesh supports configuration for WireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a few clicks, services in one site get connectivity to services in other sites over a mesh of WireGuard tunnels. +* Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loop cable. Removed the need for IP address reservation for V-NET, switching entirely to the anycast default gateway. +* Controller distributions​ - Netris controller, is now available in three deployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application. 3) Managed/Hosted in the cloud. +* Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes. +* Switch/SoftGate agents​ - New installer with easy initial config tool. Support for IP and FQDN as a controller address. Authentication key. +* GUI​ - Improved Net→Topology section, becoming the main and required place for defining the network topology. All sections got a column organizer, so every user can order and hide/show columns to their comfort. diff --git a/en/latest/_sources/roh.rst.txt b/en/latest/_sources/roh.rst.txt new file mode 100644 index 0000000000..f93046ad12 --- /dev/null +++ b/en/latest/_sources/roh.rst.txt @@ -0,0 +1,58 @@ +.. meta:: + :description: Routing on the Host + +.. _roh_def: + +######################### +ROH (Routing on the Host) +######################### +To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly on their servers. This is commonly known as ROH (Routing on the Host). + +In ROH architectures, servers use a routing daemon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon/suite is FRR. + +Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead an IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. This is a modern and optimal design, leveraging Layer-3 networking from the fabric to the servers. + +By using only Layer-3 interfaces, Layer-2 protocols such as Spanning Tree (STP) can be minized and the reliability of the network increases. + +The ROH architecture that is configured by Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). For each instance of ROH, you’ll need to create a ROH entry in Netris Controller. + +Adding ROH Hosts +---------------- + +#. Navigate in the Netris UI to **Services→Instances (ROH)** +#. Click the **Add** button +#. Fill out the form based on the fields in the table below. +#. Click the **Add** button + +Description of ROH instance fields: + +- **Name** - Unique name for the ROH instance +- **Site** - Site where the current ROH instance belongs +- **Type** - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor. +- **ROH Routing Profile** - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances + + - **Default route only (most common design)** - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch + - **Default + Aggregate** - Will add prefixes of defined assignments + "Default" profile + - **Full table** - Will advertise all prefixes available in the routing table of the connected switch + - **Inherit** - Will inherit policy from site objects defined under Net→Sites + +- **Legacy Mode** - Switch from default zero-config mode to using /30 IP addresses. Used for MSFT Windows Servers or other OS that doesn’t support FRR. +- **+Port** - Physical Switch Ports anywhere on the network. +- **+IPv4** - IPv4 addresses for the loopback interface. +- **+Inbound Prefix List** - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks. + +.. tip:: Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +.. image:: images/ROH-instance.png + :align: center + :class: with-shadow + :alt: ROH Instances + +Example: Adding an ROH instance. (Yes, you can use A.B.C.0/32 and A.B.C.255/32) + +.. image:: images/ROH-listing.png + :align: center + :class: with-shadow + :alt: ROH Listings + +Expanded view of ROH listing. BGP sessions are up, and the expected IP is in fact received from the actual ROH server. Traffic stats are available per port. diff --git a/en/latest/_sources/sandbox/Sandbox1/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox1/configurations.rst.txt new file mode 100644 index 0000000000..1db04fec71 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox1/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s1-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc1::1**) from the "**2607:f358:11:ffc1::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.0)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.201 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox1/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox1/creating-services.rst.txt new file mode 100644 index 0000000000..6edc9c0f92 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox1/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s1-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s1-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.201 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s1-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1012`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.22``. + 10. In the **Remote IP** field, type in ``45.38.161.21``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.0/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.1)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s1-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.201 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.4/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.4/32**" IP address. + + * This public IP address is part of **45.38.161.4/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s1-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.8/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.8**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.8**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.8**) into the browser's address bar or simply visit `http://45.38.161.8/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.8 (name_45.38.161.8)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.8** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s1-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.201 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox1.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox1/index.rst.txt b/en/latest/_sources/sandbox/Sandbox1/index.rst.txt new file mode 100644 index 0000000000..f01bc8615b --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox1/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox1 # Sandbox name Uppercase(case sensitive) + sandbox1 # Sandbox name Lowercase + 216.172.128.201 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1011 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1012 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.0/28 # PUBLIC IPv4 Allocation + 45.38.161.0/30 # PUBLIC LOOPBACK subnet + 45.38.161.1 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.4/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.4/32 # CUSTOMER V-NET SNAT IP + 45.38.161.8/30 # L3LB Subnet + 45.38.161.8/32 # L3LB IP + 45.38.161.12/30 # L4LB Subnet + 45.38.161.13 # Second usable ip address in load-balancer subnet + 45.38.161.14 # Third usable ip address in load-balancer subnet + 45.38.161.18/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.17/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.22/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.21/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc1::/64 # public IPv6 subnet + 2607:f358:11:ffc1::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::3/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::2/127 # isp1-ipv6-example BGP peer remote IPv6 + s1-pre-configured # LINK + s1-learn-by-doing # LINK + s1-e-bgp # LINK + s1-v-net # LINK + s1-nat # LINK + s1-acl # LINK + s1-l3lb # LINK + s1-k8s # LINK + s1-topology # LINK + +Sandbox1 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt new file mode 100644 index 0000000000..a1c64ffd14 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox1/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s1-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox1.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox1.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox1.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox1.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.13 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.13:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.13 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.13 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.13 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.13 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.13 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.14 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.14 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.14 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.14 + + SRV05-NYC + curl 45.38.161.14 + + SRV04-NYC + curl 45.38.161.14 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1012 + localIP: 45.38.161.22/30 + remoteIP: 45.38.161.21/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.0/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.22/30 45.38.161.21/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.22/30 45.38.161.21/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.22/30 45.38.161.21/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.22/30 45.38.161.21/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.13 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox1/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox1/sandbox-info.rst.txt new file mode 100644 index 0000000000..df2fb7b106 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox1/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s1-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox1.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.201 -p 30061 + srv02-nyc: ssh demo@216.172.128.201 -p 30062 + srv03-nyc: ssh demo@216.172.128.201 -p 30063 + srv04-nyc: ssh demo@216.172.128.201 -p 30064 + srv05-nyc: ssh demo@216.172.128.201 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1011 + Local Address: 45.38.161.18/30 + Remote Address: 45.38.161.17/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.0/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1011 + Local Address: 2607:f358:11:ffc0::3/127 + Remote Address: 2607:f358:11:ffc0::2/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc1::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1012 + Local Address: 45.38.161.22/30 + Remote Address: 45.38.161.21/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.0/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.0/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.0/30 + |___ NAT Subnet: 45.38.161.4/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.8/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.12/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc1::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc1::/64 diff --git a/en/latest/_sources/sandbox/Sandbox10/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox10/configurations.rst.txt new file mode 100644 index 0000000000..737adcd371 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox10/configurations.rst.txt @@ -0,0 +1,83 @@ +.. _s10-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +After logging into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigating to **Services → V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. + +If you examine the particular service settings ( select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service), you will find that the services is configured on the second port of **switch 21** named "**swp2(swp2)@sw21-nyc (Admin)**". + +The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, **192.168.45.1** (from the "**192.168.45.0/24 (EXAMPLE)**" subnet) and **2607:f358:11:ffca::1** (from the "**2607:f358:11:ffca::/64 (EXAMPLE IPv6)**" subnet) respectively. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net → Looking Glass**. +2. Select switch "**sw21-nyc(10.254.46.21)**" (the switch the "**vnet-example**" service is configured on) from the **Select device** drop-down menu. +3. Select "**Ping**" from the **Command** drop-down menu. +4. Type ``192.168.45.64`` (the IP address of **srv04-nyc** connected to **swp2@sw21-nyc**) in the field labeled **IPv4 address**. +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between switch **sw21-nyc** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4092ms + rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net → E-BGP**. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as example with **IRIS ISP1**. This ensures communication between the internal network with the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.210 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Net → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. \ No newline at end of file diff --git a/en/latest/_sources/sandbox/Sandbox10/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox10/creating-services.rst.txt new file mode 100644 index 0000000000..015a7dfd8b --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox10/creating-services.rst.txt @@ -0,0 +1,200 @@ +.. _s10-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s10-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.210 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **Owner** drop-down menu, select "**Demo**". + 6. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 7. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 8. From the **Add Network Interface** drop-down menu put a check mark next to switch port "**swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)**", which we can see is the the port where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains this single switch port as it is the only port that has been assigned to the **Demo** tenant. + + 9. Check the **Untag** check-box and click the **Add** button. + 10. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-NET"` page. + +.. _s10-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Net → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, uncheck the **Untag** check-box and type in ``1102``. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.126``. + 10. In the **Remote IP** field, type in ``50.117.59.125``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Inbound** field, type in ``permit 0.0.0.0/0`` + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.208/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Select "**SoftGate2(50.117.59.209)**" (the border router where our newly created BGP session is terminated on) from the **Select device** drop-down menu. + 2. Leaving the **Family** drop-down menu on IPv4 and the **Command** drop-down menu on "**BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s10-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.210 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Net → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. From the **Protocol** drop-down menu, select "**ALL**". + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, type in ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.212/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.212/32**" IP address. + + * This public IP is part of **50.117.59.212/30 (NAT)** subnet which is configured in the **NET → IPAM** section with the purpose of **NAT** and indicated in the SoftGate configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`.. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s10-l3lb: + +L3LB (Anycast L3 load balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running Web Server configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.216/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.216**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.216**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.216**) into the browser's address bar or simply visit `http://50.117.59.216/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** to the left of "**50.117.59.216 (name_50.117.59.216)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.216** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s10-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.210 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox10.netris.io `_ and navigate to **Net → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable.Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the newly created "**V-Net Customer to WAN**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox10/index.rst.txt b/en/latest/_sources/sandbox/Sandbox10/index.rst.txt new file mode 100644 index 0000000000..a6a5f4852a --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox10/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox10 # Sandbox name Uppercase(case sensitive) + sandbox10 # Sandbox name Lowercase + 216.172.128.210 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1101 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1102 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.208/28 # PUBLIC IPv4 Allocation + 50.117.59.208/30 # PUBLIC LOOPBACK subnet + 50.117.59.209 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.212/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.212/32 # CUSTOMER V-NET SNAT IP + 50.117.59.216/30 # L3LB Subnet + 50.117.59.216/32 # L3LB IP + 50.117.59.220/30 # L4LB Subnet + 50.117.59.221 # Second usable ip address in load-balancer subnet + 50.117.59.222 # Third usable ip address in load-balancer subnet + 50.117.59.122/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.121/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.126/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.125/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffca::/64 # public IPv6 subnet + 2607:f358:11:ffca::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::15/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::14/127 # isp1-ipv6-example BGP peer remote IPv6 + s10-pre-configured # LINK + s10-learn-by-doing # LINK + s10-e-bgp # LINK + s10-v-net # LINK + s10-nat # LINK + s10-acl # LINK + s10-l3lb # LINK + s10-k8s # LINK + s10-topology # LINK + +Sandbox10 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt new file mode 100644 index 0000000000..875c5ea3f7 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox10/onprem-k8s.rst.txt @@ -0,0 +1,621 @@ +.. _s10-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox10.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox10.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.221 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.221:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.221 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.221 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.221 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.221 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.221 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.222 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.222 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.222 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.222 + + SRV05-NYC + curl 50.117.59.222 + + SRV05-NYC + curl 50.117.59.222 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1102 + localIP: 50.117.59.126/30 + remoteIP: 50.117.59.125/30 + description: Example BGP to ISP2 + prefixListInbound: + - permit 0.0.0.0/0 + prefixListOutbound: + - permit 50.117.59.208/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.126/30 50.117.59.125/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:01:27 Link Up 65007 50.117.59.126/30 50.117.59.125/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:06:18 Link Up 65007 50.117.59.126/30 50.117.59.125/30 7m59s + sandbox10-srv06-nyc-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 26s + sandbox10-srv07-nyc-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 26s + sandbox10-srv08-nyc-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 160; time: 00:07:48 Link Up 65007 50.117.59.126/30 50.117.59.125/30 8m41s + sandbox10-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 68s + sandbox10-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 68s + sandbox10-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.221 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/latest/_sources/sandbox/Sandbox10/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox10/sandbox-info.rst.txt new file mode 100644 index 0000000000..9a6a8a42db --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox10/sandbox-info.rst.txt @@ -0,0 +1,134 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s10-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology.png + + + +Netris Controller +================= +https://sandbox10.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.** + * **srv01-nyc**, **srv02-nyc** - are behind :ref:`"Anycast L3 load balancer"`, see **Services → Load Balancer**. + * **srv04-nyc**, **srv05-nyc** - are consuming :ref:`"V-NET (routed VXLAN)"` Netris service, see **Services → V-NET**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.210 -p 30061 + srv02-nyc: ssh demo@216.172.128.210 -p 30062 + srv03-nyc: ssh demo@216.172.128.210 -p 30063 + srv04-nyc: ssh demo@216.172.128.210 -p 30064 + srv05-nyc: ssh demo@216.172.128.210 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **NET → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1101 + Local Address: 50.117.59.122/30 + Remote Address: 50.117.59.121/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.208/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1101 + Local Address: 2607:f358:11:ffc0::15/127 + Remote Address: 2607:f358:11:ffc0::14/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffca::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1102 + Local Address: 50.117.59.126/30 + Remote Address: 50.117.59.125/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.208/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Net → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.208/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.208/30 + |___ NAT Subnet: 50.117.59.212/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.216/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.220/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffca::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffca::/64 + diff --git a/en/latest/_sources/sandbox/Sandbox11/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox11/configurations.rst.txt new file mode 100644 index 0000000000..df738cbaef --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox11/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s11-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffcb::1**) from the "**2607:f358:11:ffcb::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.96)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.211 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox11/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox11/creating-services.rst.txt new file mode 100644 index 0000000000..3cdea0f688 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox11/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s11-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s11-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.211 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s11-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1112`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.118``. + 10. In the **Remote IP** field, type in ``45.38.161.117``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.96/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.97)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s11-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.211 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.100/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.100/32**" IP address. + + * This public IP address is part of **45.38.161.100/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s11-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.104/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.104**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.104**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.104**) into the browser's address bar or simply visit `http://45.38.161.104/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.104 (name_45.38.161.104)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.104** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s11-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.211 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox11.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox11/index.rst.txt b/en/latest/_sources/sandbox/Sandbox11/index.rst.txt new file mode 100644 index 0000000000..8e43bbe497 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox11/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox11 # Sandbox name Uppercase(case sensitive) + sandbox11 # Sandbox name Lowercase + 216.172.128.211 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1111 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1112 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.96/28 # PUBLIC IPv4 Allocation + 45.38.161.96/30 # PUBLIC LOOPBACK subnet + 45.38.161.97 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.100/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.100/32 # CUSTOMER V-NET SNAT IP + 45.38.161.104/30 # L3LB Subnet + 45.38.161.104/32 # L3LB IP + 45.38.161.108/30 # L4LB Subnet + 45.38.161.109 # Second usable ip address in load-balancer subnet + 45.38.161.110 # Third usable ip address in load-balancer subnet + 45.38.161.114/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.113/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.118/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.117/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcb::/64 # public IPv6 subnet + 2607:f358:11:ffcb::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::17/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::16/127 # isp1-ipv6-example BGP peer remote IPv6 + s11-pre-configured # LINK + s11-learn-by-doing # LINK + s11-e-bgp # LINK + s11-v-net # LINK + s11-nat # LINK + s11-acl # LINK + s11-l3lb # LINK + s11-k8s # LINK + s11-topology # LINK + +Sandbox11 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt new file mode 100644 index 0000000000..7fedb28775 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox11/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s11-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox11.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox11.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox11.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox11.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.109 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.109:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.109 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.109 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.109 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.109 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.109 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.110 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.110 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.110 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.110 + + SRV05-NYC + curl 45.38.161.110 + + SRV04-NYC + curl 45.38.161.110 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1112 + localIP: 45.38.161.118/30 + remoteIP: 45.38.161.117/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.96/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.118/30 45.38.161.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.118/30 45.38.161.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.118/30 45.38.161.117/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.118/30 45.38.161.117/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.109 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox11/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox11/sandbox-info.rst.txt new file mode 100644 index 0000000000..a623002cd2 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox11/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s11-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox11.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.211 -p 30061 + srv02-nyc: ssh demo@216.172.128.211 -p 30062 + srv03-nyc: ssh demo@216.172.128.211 -p 30063 + srv04-nyc: ssh demo@216.172.128.211 -p 30064 + srv05-nyc: ssh demo@216.172.128.211 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1111 + Local Address: 45.38.161.114/30 + Remote Address: 45.38.161.113/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.96/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1111 + Local Address: 2607:f358:11:ffc0::17/127 + Remote Address: 2607:f358:11:ffc0::16/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcb::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1112 + Local Address: 45.38.161.118/30 + Remote Address: 45.38.161.117/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.96/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.96/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.96/30 + |___ NAT Subnet: 45.38.161.100/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.104/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.108/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcb::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcb::/64 diff --git a/en/latest/_sources/sandbox/Sandbox12/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox12/configurations.rst.txt new file mode 100644 index 0000000000..1f87facf01 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox12/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _2607:f358:11:ffc0::18: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://Sandbox12.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**45.38.161.1251**) from the "**45.38.161.125/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(1122)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@sandbox12 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox12/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox12/creating-services.rst.txt new file mode 100644 index 0000000000..fd9834bcd8 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox12/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s12-pre-configured: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s12-e-bgp: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@sandbox12 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://Sandbox12.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s12-learn-by-doing: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://Sandbox12.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1121`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.121``. + 10. In the **Remote IP** field, type in ``45.38.161.126``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 1122/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.128)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s12-v-net: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@sandbox12 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://Sandbox12.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.129/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.129/32**" IP address. + + * This public IP address is part of **45.38.161.129/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s12-acl: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://Sandbox12.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.132/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.132**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.132**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.132**) into the browser's address bar or simply visit `http://45.38.161.132/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.132 (name_45.38.161.132)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.132** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s12-nat: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@sandbox12 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://Sandbox12.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox12/index.rst.txt b/en/latest/_sources/sandbox/Sandbox12/index.rst.txt new file mode 100644 index 0000000000..86f50ddef5 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox12/index.rst.txt @@ -0,0 +1,61 @@ + +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox12 # Sandbox name Uppercase(case sensitive) + sandbox12 # Sandbox name Lowercase + 216.172.128.212 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1121 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1122 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.128/28 # PUBLIC IPv4 Allocation + 45.38.161.128/30 # PUBLIC LOOPBACK subnet + 45.38.161.129 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.132/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.132/32 # CUSTOMER V-NET SNAT IP + 45.38.161.136/30 # L3LB Subnet + 45.38.161.136/32 # L3lB IP + 45.38.161.140/30 # L4LB Subnet + 45.38.161.141 # Second usable ip address in load-balancer subnet + 45.38.161.142 # Third usable ip address in load-balancer subnet + 45.38.161.122/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.121/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.126/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.125/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcc::/64 # public IPv6 subnet + 2607:f358:11:ffcc::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::19/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::18/127 # isp1-ipv6-example BGP peer remote IPv6 + s12-pre-configured # LINK + s12-learn-by-doing # LINK + s12-e-bgp # LINK + s12-v-net # LINK + s12-nat # LINK + s12-acl # LINK + s12-l3lb # LINK + s12-k8s # LINK + s12-topology # LINK + +Sandbox12 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt new file mode 100644 index 0000000000..915bcfd094 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox12/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s12-l3lb: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-Sandbox12.netris.io:6443 + CoreDNS is running at https://api.k8s-Sandbox12.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-Sandbox12.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://Sandbox12.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.140 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.140:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.140 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.140 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.140 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.140 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.140 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.141 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.141 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.141 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.141 + + SRV05-NYC + curl 45.38.161.141 + + SRV04-NYC + curl 45.38.161.141 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1121 + localIP: 45.38.161.121/30 + remoteIP: 45.38.161.126/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 1122/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.121/30 45.38.161.126/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.121/30 45.38.161.126/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.121/30 45.38.161.126/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.121/30 45.38.161.126/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.140 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox12/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox12/sandbox-info.rst.txt new file mode 100644 index 0000000000..f7a486bfd8 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox12/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"<2607:f358:11:ffc0::18>` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s12-k8s: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://Sandbox12.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@sandbox12 -p 30061 + srv02-nyc: ssh demo@sandbox12 -p 30062 + srv03-nyc: ssh demo@sandbox12 -p 30063 + srv04-nyc: ssh demo@sandbox12 -p 30064 + srv05-nyc: ssh demo@sandbox12 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 216.172.128.212 + Local Address: 45.38.161.142/30 + Remote Address: 45.38.161.122/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 1122/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 216.172.128.212 + Local Address: 2607:f358:11:ffcc::1/127 + Remote Address: 2607:f358:11:ffc0::19/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 45.38.161.125/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1121 + Local Address: 45.38.161.121/30 + Remote Address: 45.38.161.126/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 1122/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 1122/28 + |___ PUBLIC LOOPBACK Subnet: 1122/30 + |___ NAT Subnet: 45.38.161.129/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.132/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.136/30 + + | EXAMPLE IPv6 Allocation: 45.38.161.125/64 + |___ EXAMPLE IPv6 Subnet: 45.38.161.125/64 diff --git a/en/latest/_sources/sandbox/Sandbox13/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox13/configurations.rst.txt new file mode 100644 index 0000000000..dd2b66a5c9 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox13/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s13-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffcd::1**) from the "**2607:f358:11:ffcd::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.144)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.213 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox13/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox13/creating-services.rst.txt new file mode 100644 index 0000000000..f92c8fdf2a --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox13/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s13-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s13-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.213 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s13-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1132`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.166``. + 10. In the **Remote IP** field, type in ``45.38.161.165``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.144/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.145)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s13-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.213 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.148/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.148/32**" IP address. + + * This public IP address is part of **45.38.161.148/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s13-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.152/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.152**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.152**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.152**) into the browser's address bar or simply visit `http://45.38.161.152/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.152 (name_45.38.161.152)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.152** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s13-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.213 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox13.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox13/index.rst.txt b/en/latest/_sources/sandbox/Sandbox13/index.rst.txt new file mode 100644 index 0000000000..ae3a0259be --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox13/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox13 # Sandbox name Uppercase(case sensitive) + sandbox13 # Sandbox name Lowercase + 216.172.128.213 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1131 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1132 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.144/28 # PUBLIC IPv4 Allocation + 45.38.161.144/30 # PUBLIC LOOPBACK subnet + 45.38.161.145 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.148/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.148/32 # CUSTOMER V-NET SNAT IP + 45.38.161.152/30 # L3LB Subnet + 45.38.161.152/32 # L3LB IP + 45.38.161.156/30 # L4LB Subnet + 45.38.161.157 # Second usable ip address in load-balancer subnet + 45.38.161.158 # Third usable ip address in load-balancer subnet + 45.38.161.162/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.161/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.166/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.165/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcd::/64 # public IPv6 subnet + 2607:f358:11:ffcd::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1b/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1a/127 # isp1-ipv6-example BGP peer remote IPv6 + s13-pre-configured # LINK + s13-learn-by-doing # LINK + s13-e-bgp # LINK + s13-v-net # LINK + s13-nat # LINK + s13-acl # LINK + s13-l3lb # LINK + s13-k8s # LINK + s13-topology # LINK + +Sandbox13 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt new file mode 100644 index 0000000000..0ff1a97110 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox13/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s13-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox13.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox13.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox13.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox13.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.157 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.157:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.157 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.157 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.157 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.157 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.157 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.158 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.158 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.158 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.158 + + SRV05-NYC + curl 45.38.161.158 + + SRV04-NYC + curl 45.38.161.158 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1132 + localIP: 45.38.161.166/30 + remoteIP: 45.38.161.165/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.144/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.166/30 45.38.161.165/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.166/30 45.38.161.165/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.166/30 45.38.161.165/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.166/30 45.38.161.165/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.157 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox13/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox13/sandbox-info.rst.txt new file mode 100644 index 0000000000..10e9782ce5 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox13/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s13-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox13.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.213 -p 30061 + srv02-nyc: ssh demo@216.172.128.213 -p 30062 + srv03-nyc: ssh demo@216.172.128.213 -p 30063 + srv04-nyc: ssh demo@216.172.128.213 -p 30064 + srv05-nyc: ssh demo@216.172.128.213 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1131 + Local Address: 45.38.161.162/30 + Remote Address: 45.38.161.161/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.144/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1131 + Local Address: 2607:f358:11:ffc0::1b/127 + Remote Address: 2607:f358:11:ffc0::1a/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcd::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1132 + Local Address: 45.38.161.166/30 + Remote Address: 45.38.161.165/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.144/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.144/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.144/30 + |___ NAT Subnet: 45.38.161.148/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.152/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.156/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcd::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcd::/64 diff --git a/en/latest/_sources/sandbox/Sandbox14/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox14/configurations.rst.txt new file mode 100644 index 0000000000..388346727e --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox14/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s14-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffce::1**) from the "**2607:f358:11:ffce::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.176)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.214 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox14/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox14/creating-services.rst.txt new file mode 100644 index 0000000000..f9b2a1e757 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox14/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s14-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s14-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.214 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s14-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1142`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.174``. + 10. In the **Remote IP** field, type in ``45.38.161.173``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.176/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.177)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s14-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.214 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.180/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.180/32**" IP address. + + * This public IP address is part of **45.38.161.180/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s14-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.184/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.184**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.184**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.184**) into the browser's address bar or simply visit `http://45.38.161.184/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.184 (name_45.38.161.184)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.184** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s14-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.214 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox14/index.rst.txt b/en/latest/_sources/sandbox/Sandbox14/index.rst.txt new file mode 100644 index 0000000000..f41555557d --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox14/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox15 # Sandbox name Uppercase(case sensitive) + sandbox15 # Sandbox name Lowercase + 216.172.128.214 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1141 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1142 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.176/28 # PUBLIC IPv4 Allocation + 45.38.161.176/30 # PUBLIC LOOPBACK subnet + 45.38.161.177 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.180/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.180/32 # CUSTOMER V-NET SNAT IP + 45.38.161.184/30 # L3LB Subnet + 45.38.161.184/32 # L3LB IP + 45.38.161.188/30 # L4LB Subnet + 45.38.161.189 # Second usable ip address in load-balancer subnet + 45.38.161.190 # Third usable ip address in load-balancer subnet + 45.38.161.170/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.169/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.174/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.173/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffce::/64 # public IPv6 subnet + 2607:f358:11:ffce::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1d/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1c/127 # isp1-ipv6-example BGP peer remote IPv6 + s14-pre-configured # LINK + s14-learn-by-doing # LINK + s14-e-bgp # LINK + s14-v-net # LINK + s14-nat # LINK + s14-acl # LINK + s14-l3lb # LINK + s14-k8s # LINK + s14-topology # LINK + +Sandbox14 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt new file mode 100644 index 0000000000..aa780bb9af --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox14/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s14-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox15.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox15.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.189 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.189:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.189 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.189 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.189 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.189 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.189 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.190 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.190 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.190 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.190 + + SRV05-NYC + curl 45.38.161.190 + + SRV04-NYC + curl 45.38.161.190 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1142 + localIP: 45.38.161.174/30 + remoteIP: 45.38.161.173/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.176/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.174/30 45.38.161.173/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.174/30 45.38.161.173/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.174/30 45.38.161.173/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.174/30 45.38.161.173/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.189 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox14/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox14/sandbox-info.rst.txt new file mode 100644 index 0000000000..38fe71bcbb --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox14/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s14-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox15.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.214 -p 30061 + srv02-nyc: ssh demo@216.172.128.214 -p 30062 + srv03-nyc: ssh demo@216.172.128.214 -p 30063 + srv04-nyc: ssh demo@216.172.128.214 -p 30064 + srv05-nyc: ssh demo@216.172.128.214 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1141 + Local Address: 45.38.161.170/30 + Remote Address: 45.38.161.169/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.176/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1141 + Local Address: 2607:f358:11:ffc0::1d/127 + Remote Address: 2607:f358:11:ffc0::1c/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffce::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1142 + Local Address: 45.38.161.174/30 + Remote Address: 45.38.161.173/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.176/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.176/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.176/30 + |___ NAT Subnet: 45.38.161.180/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.184/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.188/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffce::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffce::/64 diff --git a/en/latest/_sources/sandbox/Sandbox15/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox15/configurations.rst.txt new file mode 100644 index 0000000000..117a88e76e --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox15/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s15-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffcf::1**) from the "**2607:f358:11:ffcf::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.192)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.215 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox15/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox15/creating-services.rst.txt new file mode 100644 index 0000000000..269e740532 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox15/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s15-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s15-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.215 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s15-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1152`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.214``. + 10. In the **Remote IP** field, type in ``45.38.161.213``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.192/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.193)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s15-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.215 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.196/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.196/32**" IP address. + + * This public IP address is part of **45.38.161.196/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s15-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.200/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.200**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.200**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.200**) into the browser's address bar or simply visit `http://45.38.161.200/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.200 (name_45.38.161.200)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.200** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s15-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.215 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox15.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox15/index.rst.txt b/en/latest/_sources/sandbox/Sandbox15/index.rst.txt new file mode 100644 index 0000000000..27b4ee6744 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox15/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox15 # Sandbox name Uppercase(case sensitive) + sandbox15 # Sandbox name Lowercase + 216.172.128.215 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1151 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1152 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.192/28 # PUBLIC IPv4 Allocation + 45.38.161.192/30 # PUBLIC LOOPBACK subnet + 45.38.161.193 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.196/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.196/32 # CUSTOMER V-Net SNAT IP + 45.38.161.200/30 # L3LB Subnet + 45.38.161.200/32 # L3LB IP + 45.38.161.204/30 # L4LB Subnet + 45.38.161.205 # Second usable ip address in load-balancer subnet + 45.38.161.206 # Third usable ip address in load-balancer subnet + 45.38.161.210/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.209/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.214/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.213/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffcf::/64 # public IPv6 subnet + 2607:f358:11:ffcf::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::1f/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::1e/127 # isp1-ipv6-example BGP peer remote IPv6 + s15-pre-configured # LINK + s15-learn-by-doing # LINK + s15-e-bgp # LINK + s15-v-net # LINK + s15-nat # LINK + s15-acl # LINK + s15-l3lb # LINK + s15-k8s # LINK + s15-topology # LINK + +Sandbox15 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt new file mode 100644 index 0000000000..81880c20cc --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox15/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s15-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox15.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox15.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.205 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.205:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.205 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.205 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.205 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.205 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.205 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.206 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.206 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.206 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.206 + + SRV05-NYC + curl 45.38.161.206 + + SRV04-NYC + curl 45.38.161.206 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1152 + localIP: 45.38.161.214/30 + remoteIP: 45.38.161.213/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.192/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.214/30 45.38.161.213/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.214/30 45.38.161.213/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.214/30 45.38.161.213/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.214/30 45.38.161.213/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.205 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox15/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox15/sandbox-info.rst.txt new file mode 100644 index 0000000000..f05ada9e6a --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox15/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s15-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox15.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.215 -p 30061 + srv02-nyc: ssh demo@216.172.128.215 -p 30062 + srv03-nyc: ssh demo@216.172.128.215 -p 30063 + srv04-nyc: ssh demo@216.172.128.215 -p 30064 + srv05-nyc: ssh demo@216.172.128.215 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1151 + Local Address: 45.38.161.210/30 + Remote Address: 45.38.161.209/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.192/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1151 + Local Address: 2607:f358:11:ffc0::1f/127 + Remote Address: 2607:f358:11:ffc0::1e/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffcf::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1152 + Local Address: 45.38.161.214/30 + Remote Address: 45.38.161.213/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.192/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.192/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.192/30 + |___ NAT Subnet: 45.38.161.196/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.200/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.204/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffcf::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffcf::/64 diff --git a/en/latest/_sources/sandbox/Sandbox2/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox2/configurations.rst.txt new file mode 100644 index 0000000000..92b835ebda --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox2/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s2-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc2::1**) from the "**2607:f358:11:ffc2::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.32)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.202 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox2/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox2/creating-services.rst.txt new file mode 100644 index 0000000000..1159750b8d --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox2/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s2-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s2-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.202 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s2-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1022`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.30``. + 10. In the **Remote IP** field, type in ``45.38.161.29``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.32/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.33)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s2-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.202 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.36/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.36/32**" IP address. + + * This public IP address is part of **45.38.161.36/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s2-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.40/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.40**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.40**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.40**) into the browser's address bar or simply visit `http://45.38.161.40/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.40 (name_45.38.161.40)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.40** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s2-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.202 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox2.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox2/index.rst.txt b/en/latest/_sources/sandbox/Sandbox2/index.rst.txt new file mode 100644 index 0000000000..332fc92405 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox2/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox2 # Sandbox name Uppercase(case sensitive) + sandbox2 # Sandbox name Lowercase + 216.172.128.202 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1021 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1022 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.32/28 # PUBLIC IPv4 Allocation + 45.38.161.32/30 # PUBLIC LOOPBACK subnet + 45.38.161.33 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.36/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.36/32 # CUSTOMER V-NET SNAT IP + 45.38.161.40/30 # L3LB Subnet + 45.38.161.40/32 # L3LB IP + 45.38.161.44/30 # L4LB Subnet + 45.38.161.45 # Second usable ip address in load-balancer subnet + 45.38.161.46 # Third usable ip address in load-balancer subnet + 45.38.161.26/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.25/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.30/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.29/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc2::/64 # public IPv6 subnet + 2607:f358:11:ffc2::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::5/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::4/127 # isp1-ipv6-example BGP peer remote IPv6 + s2-pre-configured # LINK + s2-learn-by-doing # LINK + s2-e-bgp # LINK + s2-v-net # LINK + s2-nat # LINK + s2-acl # LINK + s2-l3lb # LINK + s2-k8s # LINK + s2-topology # LINK + +Sandbox2 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt new file mode 100644 index 0000000000..376ddb6ae1 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox2/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s2-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox2.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox2.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox2.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox2.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.45 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.45:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.45 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.45 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.45 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.45 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.45 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.46 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.46 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.46 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.46 + + SRV05-NYC + curl 45.38.161.46 + + SRV04-NYC + curl 45.38.161.46 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1022 + localIP: 45.38.161.30/30 + remoteIP: 45.38.161.29/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.32/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.30/30 45.38.161.29/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.30/30 45.38.161.29/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.30/30 45.38.161.29/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.30/30 45.38.161.29/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.45 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox2/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox2/sandbox-info.rst.txt new file mode 100644 index 0000000000..6959a12470 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox2/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s2-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox2.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.202 -p 30061 + srv02-nyc: ssh demo@216.172.128.202 -p 30062 + srv03-nyc: ssh demo@216.172.128.202 -p 30063 + srv04-nyc: ssh demo@216.172.128.202 -p 30064 + srv05-nyc: ssh demo@216.172.128.202 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1021 + Local Address: 45.38.161.26/30 + Remote Address: 45.38.161.25/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.32/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1021 + Local Address: 2607:f358:11:ffc0::5/127 + Remote Address: 2607:f358:11:ffc0::4/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc2::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1022 + Local Address: 45.38.161.30/30 + Remote Address: 45.38.161.29/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.32/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.32/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.32/30 + |___ NAT Subnet: 45.38.161.36/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.40/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.44/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc2::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc2::/64 diff --git a/en/latest/_sources/sandbox/Sandbox3/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox3/configurations.rst.txt new file mode 100644 index 0000000000..ac48ab83fb --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox3/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s3-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc3::1**) from the "**2607:f358:11:ffc3::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.48)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.203 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox3/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox3/creating-services.rst.txt new file mode 100644 index 0000000000..43acfec241 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox3/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s3-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s3-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.203 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s3-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1032`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.70``. + 10. In the **Remote IP** field, type in ``45.38.161.69``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.48/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.49)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s3-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.203 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.52/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.52/32**" IP address. + + * This public IP address is part of **45.38.161.52/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s3-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.56/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.56**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.56**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.56**) into the browser's address bar or simply visit `http://45.38.161.56/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.56 (name_45.38.161.56)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.56** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s3-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.203 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox3.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox3/index.rst.txt b/en/latest/_sources/sandbox/Sandbox3/index.rst.txt new file mode 100644 index 0000000000..f1ae80a3f5 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox3/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox3 # Sandbox name Uppercase(case sensitive) + sandbox3 # Sandbox name Lowercase + 216.172.128.203 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1031 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1032 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.48/28 # PUBLIC IPv4 Allocation + 45.38.161.48/30 # PUBLIC LOOPBACK subnet + 45.38.161.49 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.52/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.52/32 # CUSTOMER V-NET SNAT IP + 45.38.161.56/30 # L3LB Subnet + 45.38.161.56/32 # L3LB IP + 45.38.161.60/30 # L4LB Subnet + 45.38.161.61 # Second usable ip address in load-balancer subnet + 45.38.161.62 # Third usable ip address in load-balancer subnet + 45.38.161.66/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.65/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.70/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.69/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc3::/64 # public IPv6 subnet + 2607:f358:11:ffc3::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::7/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::6/127 # isp1-ipv6-example BGP peer remote IPv6 + s3-pre-configured # LINK + s3-learn-by-doing # LINK + s3-e-bgp # LINK + s3-v-net # LINK + s3-nat # LINK + s3-acl # LINK + s3-l3lb # LINK + s3-k8s # LINK + s3-topology # LINK + +Sandbox3 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt new file mode 100644 index 0000000000..dfda1fa9d7 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox3/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s3-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox3.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox3.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox3.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox3.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.61 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.61:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.61 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.61 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.61 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.61 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.61 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.62 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.62 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.62 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.62 + + SRV05-NYC + curl 45.38.161.62 + + SRV04-NYC + curl 45.38.161.62 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1032 + localIP: 45.38.161.70/30 + remoteIP: 45.38.161.69/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.48/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.70/30 45.38.161.69/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.70/30 45.38.161.69/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.70/30 45.38.161.69/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.70/30 45.38.161.69/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.61 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox3/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox3/sandbox-info.rst.txt new file mode 100644 index 0000000000..51ea85a869 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox3/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s3-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox3.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.203 -p 30061 + srv02-nyc: ssh demo@216.172.128.203 -p 30062 + srv03-nyc: ssh demo@216.172.128.203 -p 30063 + srv04-nyc: ssh demo@216.172.128.203 -p 30064 + srv05-nyc: ssh demo@216.172.128.203 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1031 + Local Address: 45.38.161.66/30 + Remote Address: 45.38.161.65/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.48/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1031 + Local Address: 2607:f358:11:ffc0::7/127 + Remote Address: 2607:f358:11:ffc0::6/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc3::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1032 + Local Address: 45.38.161.70/30 + Remote Address: 45.38.161.69/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.48/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.48/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.48/30 + |___ NAT Subnet: 45.38.161.52/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.56/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.60/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc3::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc3::/64 diff --git a/en/latest/_sources/sandbox/Sandbox4/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox4/configurations.rst.txt new file mode 100644 index 0000000000..77d913b854 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox4/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s4-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc4::1**) from the "**2607:f358:11:ffc4::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(45.38.161.80)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.204 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox4/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox4/creating-services.rst.txt new file mode 100644 index 0000000000..d9235f065f --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox4/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s4-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s4-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.204 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s4-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1042`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``45.38.161.78``. + 10. In the **Remote IP** field, type in ``45.38.161.77``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 45.38.161.80/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(45.38.161.81)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s4-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.204 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**45.38.161.84/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**45.38.161.84/32**" IP address. + + * This public IP address is part of **45.38.161.84/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s4-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**45.38.161.88/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**45.38.161.88**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**45.38.161.88**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**45.38.161.88**) into the browser's address bar or simply visit `http://45.38.161.88/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**45.38.161.88 (name_45.38.161.88)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **45.38.161.88** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s4-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.204 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox4.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox4/index.rst.txt b/en/latest/_sources/sandbox/Sandbox4/index.rst.txt new file mode 100644 index 0000000000..988205b91f --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox4/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox4 # Sandbox name Uppercase(case sensitive) + sandbox4 # Sandbox name Lowercase + 216.172.128.204 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1041 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1042 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 45.38.161.80/28 # PUBLIC IPv4 Allocation + 45.38.161.80/30 # PUBLIC LOOPBACK subnet + 45.38.161.81 # PUBLIC Loopback IPv4 of SoftGate2 + 45.38.161.84/30 # PUBLIC IPv4 NAT Subnet + 45.38.161.84/32 # CUSTOMER V-NET SNAT IP + 45.38.161.88/30 # L3LB Subnet + 45.38.161.88/32 # L3LB IP + 45.38.161.92/30 # L4LB Subnet + 45.38.161.93 # Second usable ip address in load-balancer subnet + 45.38.161.94 # Third usable ip address in load-balancer subnet + 45.38.161.74/30 # isp1-ipv4-example BGP peer local IPv4 + 45.38.161.73/30 # isp1-ipv4-example BGP peer remote IPv4 + 45.38.161.78/30 # isp2-ipv4-customer BGP peer local IPv4 + 45.38.161.77/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc4::/64 # public IPv6 subnet + 2607:f358:11:ffc4::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::9/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::8/127 # isp1-ipv6-example BGP peer remote IPv6 + s4-pre-configured # LINK + s4-learn-by-doing # LINK + s4-e-bgp # LINK + s4-v-net # LINK + s4-nat # LINK + s4-acl # LINK + s4-l3lb # LINK + s4-k8s # LINK + s4-topology # LINK + +Sandbox4 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt new file mode 100644 index 0000000000..0c35df0801 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox4/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s4-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox4.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox4.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox4.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox4.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 45.38.161.93 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 45.38.161.93:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 45.38.161.93 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.93 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.93 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 45.38.161.93 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 45.38.161.93 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 45.38.161.94 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 45.38.161.94 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 45.38.161.94 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 45.38.161.94 + + SRV05-NYC + curl 45.38.161.94 + + SRV04-NYC + curl 45.38.161.94 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1042 + localIP: 45.38.161.78/30 + remoteIP: 45.38.161.77/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 45.38.161.80/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 45.38.161.78/30 45.38.161.77/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.78/30 45.38.161.77/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 45.38.161.78/30 45.38.161.77/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 45.38.161.78/30 45.38.161.77/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 45.38.161.93 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox4/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox4/sandbox-info.rst.txt new file mode 100644 index 0000000000..cb37610bc6 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox4/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s4-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox4.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.204 -p 30061 + srv02-nyc: ssh demo@216.172.128.204 -p 30062 + srv03-nyc: ssh demo@216.172.128.204 -p 30063 + srv04-nyc: ssh demo@216.172.128.204 -p 30064 + srv05-nyc: ssh demo@216.172.128.204 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1041 + Local Address: 45.38.161.74/30 + Remote Address: 45.38.161.73/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.80/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1041 + Local Address: 2607:f358:11:ffc0::9/127 + Remote Address: 2607:f358:11:ffc0::8/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc4::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1042 + Local Address: 45.38.161.78/30 + Remote Address: 45.38.161.77/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 45.38.161.80/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 45.38.161.80/28 + |___ PUBLIC LOOPBACK Subnet: 45.38.161.80/30 + |___ NAT Subnet: 45.38.161.84/30 + |___ L3 LOAD BALANCER Subnet: 45.38.161.88/30 + |___ L4 LOAD BALANCER Subnet: 45.38.161.92/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc4::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc4::/64 diff --git a/en/latest/_sources/sandbox/Sandbox5/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox5/configurations.rst.txt new file mode 100644 index 0000000000..8815bd6522 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox5/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s5-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc5::1**) from the "**2607:f358:11:ffc5::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(50.117.59.128)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.205 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox5/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox5/creating-services.rst.txt new file mode 100644 index 0000000000..86858b8c6f --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox5/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s5-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s5-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.205 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s5-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1052`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.86``. + 10. In the **Remote IP** field, type in ``50.117.59.85``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 50.117.59.128/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(50.117.59.129)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s5-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.205 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.132/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.132/32**" IP address. + + * This public IP address is part of **50.117.59.132/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s5-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.136/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.136**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.136**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.136**) into the browser's address bar or simply visit `http://50.117.59.136/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**50.117.59.136 (name_50.117.59.136)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.136** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s5-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.205 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox5.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox5/index.rst.txt b/en/latest/_sources/sandbox/Sandbox5/index.rst.txt new file mode 100644 index 0000000000..e0e24ffbdf --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox5/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox5 # Sandbox name Uppercase(case sensitive) + sandbox5 # Sandbox name Lowercase + 216.172.128.205 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1051 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1052 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.128/28 # PUBLIC IPv4 Allocation + 50.117.59.128/30 # PUBLIC LOOPBACK subnet + 50.117.59.129 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.132/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.132/32 # CUSTOMER V-NET SNAT IP + 50.117.59.136/30 # L3LB Subnet + 50.117.59.136/32 # L3LB IP + 50.117.59.140/30 # L4LB Subnet + 50.117.59.141 # Second usable ip address in load-balancer subnet + 50.117.59.142 # Third usable ip address in load-balancer subnet + 50.117.59.82/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.81/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.86/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.85/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc5::/64 # public IPv6 subnet + 2607:f358:11:ffc5::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::b/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::a/127 # isp1-ipv6-example BGP peer remote IPv6 + s5-pre-configured # LINK + s5-learn-by-doing # LINK + s5-e-bgp # LINK + s5-v-net # LINK + s5-nat # LINK + s5-acl # LINK + s5-l3lb # LINK + s5-k8s # LINK + s5-topology # LINK + +Sandbox5 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt new file mode 100644 index 0000000000..937801520f --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox5/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s5-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox5.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox5.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox5.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox5.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 50.117.59.141 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.141:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.141 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.141 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.141 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.141 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.141 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 50.117.59.142 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.142 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.142 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.142 + + SRV05-NYC + curl 50.117.59.142 + + SRV04-NYC + curl 50.117.59.142 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1052 + localIP: 50.117.59.86/30 + remoteIP: 50.117.59.85/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.128/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.86/30 50.117.59.85/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.86/30 50.117.59.85/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 50.117.59.86/30 50.117.59.85/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 50.117.59.86/30 50.117.59.85/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.141 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox5/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox5/sandbox-info.rst.txt new file mode 100644 index 0000000000..032acd1e04 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox5/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s5-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox5.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.205 -p 30061 + srv02-nyc: ssh demo@216.172.128.205 -p 30062 + srv03-nyc: ssh demo@216.172.128.205 -p 30063 + srv04-nyc: ssh demo@216.172.128.205 -p 30064 + srv05-nyc: ssh demo@216.172.128.205 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1051 + Local Address: 50.117.59.82/30 + Remote Address: 50.117.59.81/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.128/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1051 + Local Address: 2607:f358:11:ffc0::b/127 + Remote Address: 2607:f358:11:ffc0::a/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc5::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1052 + Local Address: 50.117.59.86/30 + Remote Address: 50.117.59.85/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.128/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.128/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.128/30 + |___ NAT Subnet: 50.117.59.132/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.136/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.140/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc5::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc5::/64 diff --git a/en/latest/_sources/sandbox/Sandbox6/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox6/configurations.rst.txt new file mode 100644 index 0000000000..b01341afd5 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox6/configurations.rst.txt @@ -0,0 +1,71 @@ +.. _s6-pre-configured: + +******************************** +Provided Example Configurations +******************************** +Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +Once you log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigating to **Services > V-Net**, you will find a V-Net service named "**vnet-example**" already configured for you as an example. You can examine the particular service settings by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**vnet-example**" service. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Net > Looking Glass**. +2. Select the **sw01-nyc(10.254.46.1)** switch from the **Select device** drop-down menu. +3. Select the **Ping** radio button from the row of available choices. +4. Type in ``192.168.45.64`` in the field labeled "**IPv4 address**". +5. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between the **sw01-nyc** switch and **srv04-nyc** server is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + sw01-nyc# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms + + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4074ms + rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section in the :ref:`"Learn by Creating Services"` document. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Net > E-BGP**. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1 & SoftGate2** ) and their respective adjacent spine switches ( **sw01-nyc & sw02-nyc** ), you will also find an E-BGP session named "**iris-isp1-example**" configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet. + +You can examine the particular session settings of the E-BGP connection by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**iris-isp1-example**" connection. While viewing the settings, you may also expand the **Advanced** section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` in the :ref:`"Learn by Creating Services"` document. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Net > NAT** and you will find a NAT service named "**NAT Example**" configured as an example . The configured services ensures that there can be communication between the the private **192.168.45.0/24** network and the Internet. + +You can examine the particular settings of the NAT service by clicking **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**NAT Example**" service. + +You may observe the functioning service by pinging the pubblic **1.1.1.1** IP address from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to srv04-nyc by typing ``ssh demo@216.172.128.206 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the public **1.1.1.1** IP address. + +If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the :ref:`"NAT (Network Address Translation)"` section in the in the :ref:`"Learn by Creating Services"` document. + +ACL (Access Control List) Example +================================= +Navigate to **Services > ACL** and you will find an ACL services named "**V-Net to WAN Example**" set up as an example. This particular ACL ensures that the connectivity between the the private **192.168.45.0/24** network and the Internet is permitted through all protocols and ports, even in a scenario where the the **Default Site Policy** for the "**US/NYC**" site configured in the our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the "**V-Net to WAN Example**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the :ref:`"ACL (Access Control List)"` section in the in the :ref:`"Learn by Creating Services"` document. diff --git a/en/latest/_sources/sandbox/Sandbox6/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox6/creating-services.rst.txt new file mode 100644 index 0000000000..8aa15deb7b --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox6/creating-services.rst.txt @@ -0,0 +1,136 @@ +.. _s6-learn-by-doing: + +************************** +Learn by Creating Services +************************** +.. _s6-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to the **srv05-nyc** server by typing ``ssh demo@216.172.128.206 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see ``192.168.46.1`` is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway by typing ``ping 192.168.46.1`` and keep it running as an indicator for when the service becomes fully provisioned. + 5. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**" + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Services > V-Net**. + 2. Click **+Add** to create a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select **US/NYC**. + 5. Click **+IPv4 Gateway**, select subnet ``192.168.46.0/24(CUSTOMER)`` and IP ``192.168.46.1`` to match the results of the ``ip route ls`` output on **srv05-nyc**. + 6. Click **+Port** to define the port(s) to be included in the current V-Net service. + + * For the purposes of this exercise, you can easily find the required port by typing "``srv05``" in the Search field. + + 7. Select the port named **swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin)**, check the **Untag** check-box and click **Add**. + 8. Click **Save** and the service will start provisioning. + +Once fully provisioned, you will start seeing replies similar in form to "**64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms**" to the ping previously started in the terminal window, indicating that now the gateway address is reachable. + +.. _s6-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Net > E-BGP**. + 2. Click **+Add** to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-customer``). + 4. From the **Site** drop-down menu, select **US/NYC**. + 5. From the **NFV Node** drop-down menu, select **SoftGate2**. + 6. In the **Neighbor AS** field, type in ``65007``. + 7. In the **Switch port** field, define the port on the switch that is connected to the ISP2. + + * For the purposes of this exercise, you can easily find the required port by typing "``ISP2``" in the Search field. + + 8. Select the port named **swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc** + 9. For the **VLAN ID** field, unselect ``Untag`` and type in ``1062``. + 10. In the **Local IP** field, type in ``50.117.59.94`` + 11. In the **Remote IP** field, type in ``50.117.59.93``. + 12. Expand the **Advanced** section + 13. In the **Prefix List Outbound** field, type in ``permit 50.117.59.144/28 le 32`` + 14. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Net > E-BGP** page as well as on **Telescope > Dashboard** pages will turn green, indication a successfully established BGP session. . + +.. _s6-nat: + +NAT (Network Address Translation) +================================= +Now when we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@216.172.128.206 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running as an indicator for when the service starts to work. + +Let's configure a source NAT so our V-Net subnet **192.168.46.0/24** can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Net > NAT**. + 2. Click **+Add** to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Action** drop-down menu, select **SNAT**. + 5. From the **Protocol** drop-down menu, select **ALL**. + 6. In the **Source** field, type in ``192.168.46.0/24``. + 7. The **Destination** field can remain as ``0.0.0.0/0``. + 8. From the **Nat IP** drop-down menu, select **50.117.59.150/32(US/NYC)**. + + * This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT. + + 9. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +.. _s6-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to srv05-nyc by typing ``ssh demo@216.172.128.206 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session by typing ``ping 1.1.1.1`` and keep it running for the duration of this exercise. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris GUI by visiting `https://sandbox6.netris.io `_ and navigate to **Net > Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from **Permit** to **Deny**. + 4. Click **Save**. + +* Back in the terminal window: + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command, indicating that the **1.1.1.1** IP address is no longer reachable. + +Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: + + 1. Navigate to **Services > ACL**. + 2. Click **+Add** to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net to WAN Customer``). + 4. From the **Protocol** drop-down menu, select **ALL**. + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + 8. Select **Approve** from the **Actions** menu indicated by three vertical dots (⋮) on the right side of the newly created "**V-Net to WAN Example**" ACL. + 9. Click **Approve** one more time in the pop-up window. + +* Back in the terminal window again: + +Once the Netris Controller has finished syncing the new ACL policy with all member devices, you can see that replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again. diff --git a/en/latest/_sources/sandbox/Sandbox6/index.rst.txt b/en/latest/_sources/sandbox/Sandbox6/index.rst.txt new file mode 100644 index 0000000000..a6239b9e0d --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox6/index.rst.txt @@ -0,0 +1,46 @@ +.. + ################## + values for replace + ################## + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + sandbox6 # sandbox name + 216.172.128.206 # hypervisor public ip + 300 # *STATIC NO NEED TO REPLACE* ssh NAT port *SHORT QUERY BE CAREFUL WHILE REPLACING* + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* management subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* loopback subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv4 ip address + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv5 ip address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer gateway + 192.168.110. # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1061 # Iris 1nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1062 # Iris 2nd peer vlanid, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.144/28 # customer public subnet + 50.117.59.154 # second usable ip address in load-balancer subnet + 50.117.59.155 # third usable ip address in load-balancer subnet + 50.117.59.90/30 # isp1-customer bgp peer local ip + 50.117.59.89/30 # isp1-customer bgp peer remote ip + 50.117.59.94/30 # isp2-customer bgp peer local ip + 50.117.59.93/30 # isp2-customer bgp peer remote ip + 50.117.59.150/32 # customer v-net nat ip + s6-pre-configured # LINKS + s6-learn-by-doing # LINKS + s6-e-bgp # LINKS + s6-v-net # LINKS + s6-nat # LINKS + s6-acl # LINKS + s6-k8s # LINKS + +Sandbox6 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt new file mode 100644 index 0000000000..2a8c5a5e77 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox6/onprem-k8s.rst.txt @@ -0,0 +1,618 @@ +.. _s6-k8s: + +*************************************** +Learn Netris operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +This Sandbox environment provides an existing Kubernetes cluster that has been deployed via `Kubespray `_. For this scenario, we will be using the `external LB `_ option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + +To access the built-in Kubernetes cluster, put "Kubeconfig" file which you received by the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable ``export KUBECONFIG=~/Downloads/config`` on your local machine. After that try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +The output below means you've successfully connected to the sandbox cluster: + +.. code-block:: shell-session + + Kubernetes master is running at https://api.k8s-sandbox6.netris.io:6443 + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox6.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-576d5bf6bd-7z9jl 1/1 Running 0 49s + pod/podinfo-576d5bf6bd-nhlmh 1/1 Running 0 33s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/podinfo ClusterIP 172.21.65.106 9898/TCP,9999/TCP 50s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 9898:32584/TCP,9999:30365/TCP 8m57s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + podinfo LoadBalancer 172.21.65.106 50.117.59.154 9898:32584/TCP,9999:30365/TCP 9m17s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.154:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.154 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-nhlmh", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 33m + podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 32m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80 active 50.117.59.154 80/TCP US/NYC Admin OK 2m17s + podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999 active 50.117.59.154 9999/TCP US/NYC Admin OK 3m47s + srv04-5-nyc-http active 50.117.59.155 80/TCP US/NYC Admin Provisioning 6s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.155 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Admin Provisioning 7s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.155 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV05-NYC + curl 50.117.59.155 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Admin + guestTenants: [] + sites: + - name: US/NYC + gateways: + - 192.168.46.1/24 + switchPorts: + - name: swp2@sw22-nyc + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 7s + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > isp2-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: isp2-customer + spec: + site: US/NYC + softgate: SoftGate2 + neighborAs: 65007 + transport: + name: swp14@sw02-nyc + vlanId: 1062 + localIP: 50.117.59.94/30 + remoteIP: 50.117.59.93/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.144/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f isp2-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled 65007 50.117.59.94/30 50.117.59.93/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 30; time: 00:00:51 UP 65007 50.117.59.94/30 50.117.59.93/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Net → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + +You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled, and ``asNumber`` is 64512 (it's Calico default AS number): + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + asNumber: 64512 + logSeverityScreen: Info + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:06:18 UP 65007 50.117.59.94/30 50.117.59.93/30 7m59s + sandbox6-srv06-nyc-192.168.110.66 enabled 4200070000 192.168.110.1/24 192.168.110.66/24 26s + sandbox6-srv07-nyc-192.168.110.67 enabled 4200070001 192.168.110.1/24 192.168.110.67/24 26s + sandbox6-srv08-nyc-192.168.110.68 enabled 4200070002 192.168.110.1/24 192.168.110.68/24 26s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + isp2-customer enabled bgp: Established; prefix: 28; time: 00:07:48 UP 65007 50.117.59.94/30 50.117.59.93/30 8m41s + sandbox6-srv06-nyc-192.168.110.66 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070000 192.168.110.1/24 192.168.110.66/24 68s + sandbox6-srv07-nyc-192.168.110.67 enabled bgp: Established; prefix: 5; time: 00:00:19 N/A 4200070001 192.168.110.1/24 192.168.110.67/24 68s + sandbox6-srv08-nyc-192.168.110.68 enabled bgp: Established; prefix: 5; time: 00:00:44 N/A 4200070002 192.168.110.1/24 192.168.110.68/24 68s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: crd.projectcalico.org/v1 + kind: BGPConfiguration + metadata: + annotations: + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + asNumber: 64512 + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.154 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-576d5bf6bd-mfpdt", + "version": "6.0.0", + "revision": "", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.0.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.16.5", + "num_goroutine": "8", + "num_cpu": "4" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is! diff --git a/en/latest/_sources/sandbox/Sandbox6/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox6/sandbox-info.rst.txt new file mode 100644 index 0000000000..4bb02030b7 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox6/sandbox-info.rst.txt @@ -0,0 +1,88 @@ +************************* +Welcome to Netris Sandbox +************************* + +Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you by email in response to your Sandbox request. + +The Sandbox environment includes: + +* **Netris Controller**: A cloud-hosted Netris Controller, loaded with examples. +* **Switching fabric**: Two spine switches and four leaf switches, all operated by Netris. +* **SoftGates**: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses. + + +Topology diagram +================ + +.. image:: /images/sandbox_topology.png + :align: center + +Netris GUI +========== +https://sandbox6.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services-->ROH. + * srv01, srv02 - are behind Anycast L3 load balancer. + * srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services-->V-NET. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01: ssh demo@216.172.128.206 -p 30061 + srv02: ssh demo@216.172.128.206 -p 30062 + srv03: ssh demo@216.172.128.206 -p 30063 + srv04: ssh demo@216.172.128.206 -p 30064 + srv05: ssh demo@216.172.128.206 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET-->E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured example) + Name: iris-isp1-example + Neighbor AS: 65007 + Vlan: 1061 + IP customer: 50.117.59.90/30 + IP Iris: 50.117.59.89/30 + + Neighbor AS: 65007 + Vlan: 1062 + IP customer: 50.117.59.94/30 + IP Iris: 50.117.59.93/30 + + + +Networks used +============= +.. code-block:: shell-session + + Management subnet: 10.254.45.0/24 + Loopback subnet: 10.254.46.0/24 + Example subnet: 192.168.45.0/24 + Customer subnet: 192.168.46.0/24 + K8s subnet: 192.168.110.0/24 + Public subnet: 50.117.59.144/28 + diff --git a/en/latest/_sources/sandbox/Sandbox7/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox7/configurations.rst.txt new file mode 100644 index 0000000000..7377a8baa1 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox7/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s7-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc7::1**) from the "**2607:f358:11:ffc7::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(50.117.59.160)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.207 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox7/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox7/creating-services.rst.txt new file mode 100644 index 0000000000..7b762af628 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox7/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s7-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s7-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.207 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s7-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1072`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.102``. + 10. In the **Remote IP** field, type in ``50.117.59.101``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 50.117.59.160/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(50.117.59.161)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s7-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.207 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.164/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.164/32**" IP address. + + * This public IP address is part of **50.117.59.164/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s7-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.168/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.168**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.168**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.168**) into the browser's address bar or simply visit `http://50.117.59.168/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**50.117.59.168 (name_50.117.59.168)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.168** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s7-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.207 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox7.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox7/index.rst.txt b/en/latest/_sources/sandbox/Sandbox7/index.rst.txt new file mode 100644 index 0000000000..170516e2ce --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox7/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox7 # Sandbox name Uppercase(case sensitive) + sandbox7 # Sandbox name Lowercase + 216.172.128.207 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1071 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1072 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.160/28 # PUBLIC IPv4 Allocation + 50.117.59.160/30 # PUBLIC LOOPBACK subnet + 50.117.59.161 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.164/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.164/32 # CUSTOMER V-NET SNAT IP + 50.117.59.168/30 # L3LB Subnet + 50.117.59.168/32 # L3LB IP + 50.117.59.172/30 # L4LB Subnet + 50.117.59.173 # Second usable ip address in load-balancer subnet + 50.117.59.174 # Third usable ip address in load-balancer subnet + 50.117.59.98/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.97/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.102/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.101/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc7::/64 # public IPv6 subnet + 2607:f358:11:ffc7::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::f/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::e/127 # isp1-ipv6-example BGP peer remote IPv6 + s7-pre-configured # LINK + s7-learn-by-doing # LINK + s7-e-bgp # LINK + s7-v-net # LINK + s7-nat # LINK + s7-acl # LINK + s7-l3lb # LINK + s7-k8s # LINK + s7-topology # LINK + +Sandbox7 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt new file mode 100644 index 0000000000..9bd20f5c5a --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox7/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s7-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox7.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox7.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox7.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox7.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 50.117.59.173 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.173:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.173 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.173 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.173 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.173 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.173 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 50.117.59.174 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.174 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.174 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.174 + + SRV05-NYC + curl 50.117.59.174 + + SRV04-NYC + curl 50.117.59.174 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1072 + localIP: 50.117.59.102/30 + remoteIP: 50.117.59.101/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.160/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.102/30 50.117.59.101/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.102/30 50.117.59.101/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 50.117.59.102/30 50.117.59.101/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 50.117.59.102/30 50.117.59.101/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.173 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox7/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox7/sandbox-info.rst.txt new file mode 100644 index 0000000000..c1df99a205 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox7/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s7-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox7.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.207 -p 30061 + srv02-nyc: ssh demo@216.172.128.207 -p 30062 + srv03-nyc: ssh demo@216.172.128.207 -p 30063 + srv04-nyc: ssh demo@216.172.128.207 -p 30064 + srv05-nyc: ssh demo@216.172.128.207 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1071 + Local Address: 50.117.59.98/30 + Remote Address: 50.117.59.97/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.160/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1071 + Local Address: 2607:f358:11:ffc0::f/127 + Remote Address: 2607:f358:11:ffc0::e/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc7::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1072 + Local Address: 50.117.59.102/30 + Remote Address: 50.117.59.101/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.160/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.160/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.160/30 + |___ NAT Subnet: 50.117.59.164/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.168/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.172/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc7::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc7::/64 diff --git a/en/latest/_sources/sandbox/Sandbox8/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox8/configurations.rst.txt new file mode 100644 index 0000000000..147c4ecad5 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox8/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s8-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc8::1**) from the "**2607:f358:11:ffc8::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(50.117.59.176)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.208 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox8/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox8/creating-services.rst.txt new file mode 100644 index 0000000000..55cd97e62c --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox8/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s8-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s8-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.208 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s8-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1082`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.110``. + 10. In the **Remote IP** field, type in ``50.117.59.109``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 50.117.59.176/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(50.117.59.177)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s8-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.208 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.180/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.180/32**" IP address. + + * This public IP address is part of **50.117.59.180/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s8-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.184/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.184**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.184**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.184**) into the browser's address bar or simply visit `http://50.117.59.184/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**50.117.59.184 (name_50.117.59.184)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.184** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s8-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.208 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox8.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox8/index.rst.txt b/en/latest/_sources/sandbox/Sandbox8/index.rst.txt new file mode 100644 index 0000000000..770d250a4b --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox8/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox8 # Sandbox name Uppercase(case sensitive) + sandbox8 # Sandbox name Lowercase + 216.172.128.208 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1081 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1082 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.176/28 # PUBLIC IPv4 Allocation + 50.117.59.176/30 # PUBLIC LOOPBACK subnet + 50.117.59.177 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.180/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.180/32 # CUSTOMER V-NET SNAT IP + 50.117.59.184/30 # L3LB Subnet + 50.117.59.184/32 # L3LB IP + 50.117.59.188/30 # L4LB Subnet + 50.117.59.189 # Second usable ip address in load-balancer subnet + 50.117.59.190 # Third usable ip address in load-balancer subnet + 50.117.59.106/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.105/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.110/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.109/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc8::/64 # public IPv6 subnet + 2607:f358:11:ffc8::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::11/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::10/127 # isp1-ipv6-example BGP peer remote IPv6 + s8-pre-configured # LINK + s8-learn-by-doing # LINK + s8-e-bgp # LINK + s8-v-net # LINK + s8-nat # LINK + s8-acl # LINK + s8-l3lb # LINK + s8-k8s # LINK + s8-topology # LINK + +Sandbox8 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt new file mode 100644 index 0000000000..45796ca801 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox8/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s8-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox8.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox8.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox8.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox8.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 50.117.59.189 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.189:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.189 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.189 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.189 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.189 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.189 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 50.117.59.190 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.190 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.190 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.190 + + SRV05-NYC + curl 50.117.59.190 + + SRV04-NYC + curl 50.117.59.190 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1082 + localIP: 50.117.59.110/30 + remoteIP: 50.117.59.109/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.176/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.110/30 50.117.59.109/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.110/30 50.117.59.109/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 50.117.59.110/30 50.117.59.109/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 50.117.59.110/30 50.117.59.109/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.189 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox8/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox8/sandbox-info.rst.txt new file mode 100644 index 0000000000..2751df5387 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox8/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s8-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox8.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.208 -p 30061 + srv02-nyc: ssh demo@216.172.128.208 -p 30062 + srv03-nyc: ssh demo@216.172.128.208 -p 30063 + srv04-nyc: ssh demo@216.172.128.208 -p 30064 + srv05-nyc: ssh demo@216.172.128.208 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1081 + Local Address: 50.117.59.106/30 + Remote Address: 50.117.59.105/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.176/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1081 + Local Address: 2607:f358:11:ffc0::11/127 + Remote Address: 2607:f358:11:ffc0::10/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc8::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1082 + Local Address: 50.117.59.110/30 + Remote Address: 50.117.59.109/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.176/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.176/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.176/30 + |___ NAT Subnet: 50.117.59.180/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.184/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.188/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc8::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc8::/64 diff --git a/en/latest/_sources/sandbox/Sandbox9/configurations.rst.txt b/en/latest/_sources/sandbox/Sandbox9/configurations.rst.txt new file mode 100644 index 0000000000..88da23697b --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox9/configurations.rst.txt @@ -0,0 +1,91 @@ +.. _s9-pre-configured: + +******************************** +Provided Example Configurations +******************************** + +.. contents:: + :local: + +Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the :ref:`"Learn by Creating Services"` document. + +V-Net (Ethernet/Vlan/VXlan) Example +=================================== +To access the V-Net service example, first log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigating to **Services → V-Net**, where you will find a pre-configured V-Net service named "**vnet-example**" available as an example. + +To examine the service settings, select **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**vnet-example**" service, where you'll see that the V-Net service is configured with VLAN ID **45** in order to enable **EVPN Multihoming** on the underlying switches. + +You'll also see that the V-Net service is configured with both an IPv4 gateway (**192.168.45.1**) from the "**192.168.45.0/24 (EXAMPLE)**" subnet and an IPv6 gateway (**2607:f358:11:ffc9::1**) from the "**2607:f358:11:ffc9::/64 (EXAMPLE IPv6)**" subnet. + +Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to **swp4(swp4)@sw21-nyc (Admin)** on switch 21 and **swp4(swp4)@sw22-nyc (Admin)** on switch 22. + +You may also verify that the service is working properly from within the GUI: (*\*Fields not specified should remain unchanged and retain default values*) + +1. Navigate to **Network → Looking Glass**. +2. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. +3. Select "**SoftGate1(50.117.59.192)**" from the **Hardware** drop-down menu. +4. Leave the "**Family: IPV4**" as the selected choice from the **Adrress Family** drop-down menu. +5. Select "**Ping**" from the **Command** drop-down menu. +6. Leave the "**Selecet IP address**" as the selected choice from the **Source** drop-down menu. +7. Type ``192.168.45.64`` (the IP address configured on **bond0.45** on **srv04-nyc**) in the field labeled **IPv4 address**. +8. Click **Submit**. + +The result should look similar to the output below, indicating that the communication between SoftGate **SoftGate1** and server **srv04-nyc** is working properly thanks to the configured V-Net service. + +.. code-block:: shell-session + + SoftGate1# ping -c 5 192.168.45.64 + PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data. + 64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms + 64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms + 64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms + 64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms + --- 192.168.45.64 ping statistics --- + 5 packets transmitted, 5 received, 0% packet loss, time 4002ms + rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms + +If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the :ref:`"V-Net (Ethernet/Vlan/VXlan)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`V-Net"` page. + +E-BGP (Exterior Border Gateway Protocol) Example +================================================ + +Navigate to **Network → E-BGP**. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( **SoftGate1** & **SoftGate2** ) and the rest of the switching fabric (which can be toggled on/off using the **Show System Generated** toggle at the top of the page), you will also find two E-BGP sessions named "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" configured as examples with **IRIS ISP1**. This ensures communication between the internal network and the Internet. + +You may examine the particular session configurations of the E-BGP connections by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of either the "**iris-isp1-ipv4-example**" and "**iris-isp1-ipv6-example**" connections. You may also expand the **Advanced** section located toward the bottom of the **Edit** window to be able to access the more advanced settings available while configuring an E-BGP session. + +If you are interested in learning how to create an additional E-BGP session with **IRIS ISP2** in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the :ref:`"E-BGP (Exterior Border Gateway Protocol)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +NAT (Network Address Translation) Example +========================================= +Navigate to **Network → NAT** and you will find a NAT rule named "**NAT Example**" configured as an example for you. The configured "**SNAT**" rule ensures that there can be communication between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet. + +You can examine the particular settings of the NAT rule by clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**NAT Example**" service. + +You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. **1.1.1.1**) from the **srv04-nyc** server. + +* In a terminal window: + + 1. SSH to server **srv04-nyc**: ``ssh demo@216.172.128.209 -p 30064``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping4 1.1.1.1`` + +You will see replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms**" indicating proper communication with the **1.1.1.1** public IP address. + +If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the :ref:`"NAT (Network Address Translation)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +ACL (Access Control List) Example +================================= +Navigate to **Services → ACL** and you will find an ACL services named "**V-Net Example to WAN**" set up as an example for you. This particular ACL ensures that the connectivity between the the private "**192.168.45.0/24 (EXAMPLE)**" subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the "**ACL Default Policy**" for the "**US/NYC**" site configured under **Network → Sites** in our Sandbox is changed from **Permit** to **Deny**. + +You can examine the particular settings of this ACL policy by selecting **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**V-Net Example to WAN**" ACL policy. + +By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the :ref:`"ACL (Access Control List)"` section of the :ref:`"Learn by Creating Services"` document. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox9/creating-services.rst.txt b/en/latest/_sources/sandbox/Sandbox9/creating-services.rst.txt new file mode 100644 index 0000000000..6ee3611467 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox9/creating-services.rst.txt @@ -0,0 +1,206 @@ +.. _s9-learn-by-doing: + +************************** +Learn by Creating Services +************************** + +.. contents:: + :local: + +Following these short exercises we will be able to demonstrate how the :ref:`Netris Controller`, in conjunction with the :ref:`Netris Agents` deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds. + +.. _s9-v-net: + +V-Net (Ethernet/Vlan/VXlan) +=========================== +Let's create a V-Net service to give server **srv05-nyc** the ability to reach its gateway address. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.209 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Type ``ip route ls`` and we can see **192.168.46.1** is configured as the default gateway, indicated by the "**default via 192.168.46.1 dev eth1 proto kernel onlink**" line in the output. + 4. Start a ping session towards the default gateway: ``ping 192.168.46.1`` + 5. Keep the ping running as an indicator for when the service becomes fully provisioned. + 6. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of "**From 192.168.46.64 icmp_seq=1 Destination Host Unreachable**". + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Services → V-Net**. + 2. Click the **+ Add** button in the top right corner of the page to get started with creating a new V-Net service. + 3. Define a name in the **Name** field (e.g. ``vnet-customer``). + 4. From the **Sites** drop-down menu, select "**US/NYC**". + 5. From the **VLAN ID** drop-down menu, select "**Enter manually**" and type in "**46**" in the field to the right. + 6. From the **Owner** drop-down menu, select "**Demo**". + 7. From the **IPv4 Gateway** drop-down menu, select the "**192.168.46.0/24(CUSTOMER)**" subnet. + 8. The first available IP address "**192.168.46.1**" is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ``ip route ls`` command output on **srv05-nyc** we observed earlier. + 9. From the **Add Network Interface** drop-down menu put a check mark next to both network interfaces "**swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)**" and "**swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)**", which we can see are the interfaces where **srv05-nyc** is wired into when we reference the :ref:`"Sandbox Topology diagram"`. + + * The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the **Demo** tenant. + + 10. Click the **Add** button. + 11. Click the **Add** button at the bottom right of the "**Add new V-Net**" window and the service will start provisioning. + +After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to "**64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms**", to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host **srv05-nyc**. + +More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the :ref:`"V-Network"` page. + +.. _s9-e-bgp: + +E-BGP (Exterior Border Gateway Protocol) +======================================== +Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named "**iris-isp1-example**". + +Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Network → E-BGP**. + 2. Click the **+ Add** button in the top right corner of the page to configure a new E-BGP session. + 3. Define a name in the **Name** field (e.g. ``iris-isp2-ipv4-customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **BGP Router** drop-down menu, select "**SoftGate2**". + 6. From the **Switch Port** drop-down menu, select port "**swp16(swp16 | ISP2)@sw02-nyc (Admin)**" on the switch that is connected to the ISP2. + + * For the purposes of this exercise, the required switch port can easily be found by typing ``ISP2`` in the Search field. + + 7. For the **VLAN ID** field, type in ``1092`` while leaving the **Untagged** check-box unchecked. + 8. In the **Neighbor AS** field, type in ``65007``. + 9. In the **Local IP** field, type in ``50.117.59.118``. + 10. In the **Remote IP** field, type in ``50.117.59.117``. + 11. Expand the **Advanced** section + 12. In the **Prefix List Outbound** field, type in ``permit 50.117.59.192/28 le 32`` + 13. And finally click **Add** + +Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on **Network → E-BGP** page as well as on **Telescope → Dashboard** pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to **Net → Looking Glass**. + + 1. Make sure "**vpc-1:Default**" is selected from the **VPC** drop-down menu. + 2. Select "**SoftGate2(50.117.59.193)**" (the border router where our newly created BGP session is terminated on) from the **Hardware** drop-down menu. + 3. Leaving the **Address Family** drop-down menu on "**Family: IPV4**" and the **Command** drop-down menu on "**Command: BGP Summary**", click on the **Submit** button. + +We are presented with the summary of the BGP sessions terminated on **SoftGate2**. You can also click on each BGP neighbor name to further see the "**Advertised routes**" and "**Routes**" received to/from that BGP neighbor. + +More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the :ref:`"BGP"` page. + +.. _s9-nat: + +NAT (Network Address Translation) +================================= +Now that we have both internal and external facing services, we can aim for our **srv05-nyc** server to be able to communicate with the Internet. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.209 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session towards any public IP address (e.g. ``ping 1.1.1.1``). + 4. Keep the ping running as an indicator for when the service starts to work. + +Let's configure a Source NAT so our Customer subnet **192.168.46.0/24**, which is used in the V-Net services called **vnet-customer**, can communicate with the Internet. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Network → NAT**. + 2. Click the **+ Add** button in the top right corner of the page to define a new NAT rule. + 3. Define a name in the **Name** field (e.g. ``NAT Customer``). + 4. From the **Site** drop-down menu, select "**US/NYC**". + 5. From the **Action** drop-down menu, select "**SNAT**". + 6. Leave **ALL** selected in the **Protocol** drop-down menu. + 7. In the **Source Address** field, type in ``192.168.46.0/24``. + 8. In the **Destination Address** field, leave the default value of ``0.0.0.0/0``. + 9. Toggle the switch from **SNAT to Pool** to **SNAT to IP**. + 10. From the **Select subnet** drop-down menu, select the "**50.117.59.196/30 (NAT)**" subnet. + 11. From the **Select IP** drop-down menu, select the "**50.117.59.196/32**" IP address. + + * This public IP address is part of **50.117.59.196/30 (NAT)** subnet which is configured in the **Network → IPAM** section with the purpose of **NAT** and indicated in the **SoftGate** configurations to be used as a global IP for NAT by the :ref:`"Netris SoftGate Agent"`. + + 12. Click **Add** + +Soon you will start seeing replies similar in form to "**64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms**" to the ping previously started in the terminal window, indicating that now the Internet is reachable from **srv05-nyc**. + +More details about NAT (Network Address Translation) can be found on the :ref:`"NAT"` page. + +.. _s9-l3lb: + +L3LB (Anycast L3 Load Balancer) +=============================== +In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our :ref:`"ROH (Routing on the Host)"` servers (**srv01-nyc** & **srv02-nyc**) which both have a running **Web Server** configured to display a simple HTML webpage and observe **ECMP** load balancing it in action. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Services → ROH**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv01-nyc**" server. + 3. From the **IPv4** drop-down menu, select the "**50.117.59.200/30 (L3 LOAD BALANCER)**" subnet. + 4. From the second drop-down menu that appears to the right, select the first available IP "**50.117.59.200**". + 5. Check the **Anycast** check-box next to the previously selected IP and click the **Save** button. + 6. Repeat steps **3** through **4** for "**srv02-nyc**" by first clicking **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the "**srv02-nyc**" server. + + * While editing "**srv02-nyc**", after selecting the "**50.117.59.200**" IP address , the **Anycast** check-box will already be automatically checked as we had designated the IP address as such in step **5**. + +* In a new web browser window/tab: + + 1. Type in the Anycast IP address we just configured (**50.117.59.200**) into the browser's address bar or simply visit `http://50.117.59.200/ `_. + 2. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the **L3LB** will use **ECMP** to load balance the traffic from your browser to either **srv01-nyc** or **srv02-nyc**, with the text on the website indicating where the traffic ended up. + + * It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on **srv01-nyc**. + +.. image:: /images/l3lb_srv01.png + :align: center + :alt: SRV01 L3LB + :target: ../../_images/l3lb_srv01.png + +In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from **srv01-nyc** to **srv02-nyc**, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in **Maintenance** mode. + +* Back in the Netris Controller, navigate to **Services → L3 Load Balancer**. + + 1. Expand the **LB Vip** that was created when we defined the **Anycast** IP address earlier by clicking on the **>** button to the left of "**50.117.59.200 (name_50.117.59.200)**". + 2. Click **Action v** to the right of the server you originally ended up on (in this case **srv01-nyc**). + 3. Click **Maintenance on**. + 4. Click **Maintenance** one more time in the pop-up window. + +* Back in the browser window/tab directed at the **50.117.59.200** Anycast IP address. + + 1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to **srv02-nyc** (once more, your case could be opposite for you based on the original hash). + +.. image:: /images/l3lb_srv02.png + :align: center + :alt: SRV02 L3LB + :target: ../../_images/l3lb_srv02.png + +More details about AL3LB (Anycast L3 load balancer) can be found on the :ref:`"L3 Load Balancer (Anycast LB)"` page. + +.. _s9-acl: + +ACL (Access Control List) +========================= +Now that **srv05-nyc** can communicate with both internal and external hosts, let's check Access Policy and Control options. + +* In a terminal window: + + 1. SSH to server **srv05-nyc**: ``ssh demo@216.172.128.209 -p 30065``. + 2. Enter the password provided in the introductory e-mail. + 3. Start a ping session: ``ping 1.1.1.1``. + 4. If the previous steps were followed, you should see successful ping replies in the form of "**64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms**". + 5. Keep the ping running as an indicator for when the service starts to work. + +* In a web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Log into the Netris Controller by visiting `https://sandbox9.netris.io `_ and navigate to **Network → Sites**. + 2. Click **Edit** from the **Actions** menu indicated by three vertical dots (**⋮**) on the right side of the **UC/NYC** site. + 3. From the **ACL Default Policy** drop-down menu, change the value from "**Permit**" to "**Deny**". + 4. Click **Save**. + +Soon you will notice that there are no new replies to our previously started ``ping 1.1.1.1`` command in the terminal window, indicating that the **1.1.1.1** IP address is no longer reachable. Now that the **Default ACL Policy** is set to **Deny**, we need to configure an **ACL** entry that will allow the **srv05-nyc** server to communicate with the Internet. + +* Back in the web browser: (*\*Fields not specified should remain unchanged and retain default values*) + + 1. Navigate to **Services → ACL**. + 2. Click the **+ Add** button in the top right corner of the page to define a new ACL. + 3. Define a name in the **Name** field (e.g. ``V-Net Customer to WAN``). + 4. From the **Protocol** drop-down menu, select "**ALL**". + 5. In the Source field, type in ``192.168.46.0/24``. + 6. In the Destination field, type in ``0.0.0.0/0``. + 7. Click **Add**. + +You can observer the status of the syncing process by clicking on the **Syncing** yellow label at the top right of the **ACL** windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as **Synced**. Back in the terminal window we can observer that the replies to our ``ping 1.1.1.1`` command have resumed, indicating that the **srv05-nyc** server can communicate with the Internet once again.. + +More details about ACL (Access Control List) can be found on the :ref:`"ACL"` page. diff --git a/en/latest/_sources/sandbox/Sandbox9/index.rst.txt b/en/latest/_sources/sandbox/Sandbox9/index.rst.txt new file mode 100644 index 0000000000..a5c19c4944 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox9/index.rst.txt @@ -0,0 +1,60 @@ +.. + ################# + Sandbox Variables + ################# + ------------------------------------------------------------------------------------------------ + values | description + ------------------------------------------------------------------------------------------------ + Sandbox9 # Sandbox name Uppercase(case sensitive) + sandbox9 # Sandbox name Lowercase + 216.172.128.209 # Hypervisor PUBLIC IP + 10.254.45.0/24 # *STATIC NO NEED TO REPLACE* MANAGEMENT Allocation/Subnet + 10.254.46.0/24 # *STATIC NO NEED TO REPLACE* LOOPBACK Allocation/Subnet + 192.168.44.0/24 # *STATIC NO NEED TO REPLACE* ROH Allocation/Subnet + 192.168.45.64 # *STATIC NO NEED TO REPLACE* srv04 IP Address + 192.168.45.1 # *STATIC NO NEED TO REPLACE* vnet-example IP4v GW + 192.168.46.65 # *STATIC NO NEED TO REPLACE* srv05 IP Address + 192.168.46.1 # *STATIC NO NEED TO REPLACE* vnet-customer IPv4 GW + 192.168.110.0/24 # *STATIC NO NEED TO REPLACE* k8s subnet + 65007 # *STATIC NO NEED TO REPLACE* Iris AS number bgp peer + 1091 # Iris 1st peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 1092 # Iris 2nd peer VLAN ID, *SHORT QUERY BE CAREFUL WHILE REPLACING* + 50.117.59.192/28 # PUBLIC IPv4 Allocation + 50.117.59.192/30 # PUBLIC LOOPBACK subnet + 50.117.59.193 # PUBLIC Loopback IPv4 of SoftGate2 + 50.117.59.196/30 # PUBLIC IPv4 NAT Subnet + 50.117.59.196/32 # CUSTOMER V-NET SNAT IP + 50.117.59.200/30 # L3LB Subnet + 50.117.59.200/32 # L3LB IP + 50.117.59.204/30 # L4LB Subnet + 50.117.59.205 # Second usable ip address in load-balancer subnet + 50.117.59.206 # Third usable ip address in load-balancer subnet + 50.117.59.114/30 # isp1-ipv4-example BGP peer local IPv4 + 50.117.59.113/30 # isp1-ipv4-example BGP peer remote IPv4 + 50.117.59.118/30 # isp2-ipv4-customer BGP peer local IPv4 + 50.117.59.117/30 # isp2-ipv4-customer BGP peer remote IPv4 + 2607:f358:11:ffc9::/64 # public IPv6 subnet + 2607:f358:11:ffc9::1 # vnet-example IP6v gateway + 2607:f358:11:ffc0::13/127 # isp1-ipv6-example BGP peer local IPv6 + 2607:f358:11:ffc0::12/127 # isp1-ipv6-example BGP peer remote IPv6 + s9-pre-configured # LINK + s9-learn-by-doing # LINK + s9-e-bgp # LINK + s9-v-net # LINK + s9-nat # LINK + s9-acl # LINK + s9-l3lb # LINK + s9-k8s # LINK + s9-topology # LINK + +Sandbox9 +========= +**Contents**: + +.. toctree:: + :maxdepth: 2 + + sandbox-info + configurations + creating-services + onprem-k8s diff --git a/en/latest/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt b/en/latest/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt new file mode 100644 index 0000000000..25e59faf54 --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox9/onprem-k8s.rst.txt @@ -0,0 +1,638 @@ +.. _s9-k8s: + +*************************************** +Learn Netris Operations with Kubernetes +*************************************** + +.. contents:: + :local: + +Intro +===== +The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in `HA Mode `_. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as ``K3S_URL`` environment variable for all nodes within the cluster. + +.. image:: /images/sandbox-l4lb-kubeapi.png + :align: center + :alt: Sandbox L4LB KubeAPI + :target: ../../_images/sandbox-l4lb-kubeapi.png + +To access the built-in Kubernetes cluster, put the "Kubeconfig" file which you received via the introductory email into your ``~/.kube/config`` or set "KUBECONFIG" environment variable using ``export KUBECONFIG=~/Downloads/config`` on your local machine. Afterwards, try to connect to the k8s cluster: + +.. code-block:: shell-session + + kubectl cluster-info + +If your output matches the one below, that means you've successfully connected to the Sandbox cluster: + +.. code-block:: shell-session + + Kubernetes control plane is running at https://api.k8s-sandbox9.netris.io:6443 + CoreDNS is running at https://api.k8s-sandbox9.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy + Metrics-server is running at https://api.k8s-sandbox9.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy + + To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. + +Install Netris Operator +======================= + +The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a `helm chart `_. For this example we will use the Kubernetes regular manifests: + +1. Install the latest Netris Operator: + +.. code-block:: shell-session + + kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml + +2. Create credentials secret for Netris Operator: + +.. code-block:: shell-session + + kubectl -nnetris-operator create secret generic netris-creds \ + --from-literal=host='https://sandbox9.netris.io' \ + --from-literal=login='demo' --from-literal=password='Your Demo user pass' + +3. Inspect the pod logs and make sure the operator is connected to Netris Controller: + +.. code-block:: shell-session + + kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f + +Example output demonstrating the successful operation of Netris Operator: + +.. code-block:: shell-session + + {"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1} + +.. note:: + + After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected. + +Deploy an Application with an On-Demand Netris Load Balancer +============================================================ + +In this scenario we will be installing a simple application that requires a network load balancer: + +Install the application `"Podinfo" `_: + +.. code-block:: shell-session + + kubectl apply -k github.com/stefanprodan/podinfo/kustomize + +Get the list of pods and services in the default namespace: + +.. code-block:: shell-session + + kubectl get po,svc + +As you can see, the service type is "ClusterIP": + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + pod/podinfo-7cf557d9d7-6gfwx 1/1 Running 0 34s + pod/podinfo-7cf557d9d7-nb2t7 1/1 Running 0 18s + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + service/kubernetes ClusterIP 10.43.0.1 443/TCP 33m + service/podinfo ClusterIP 10.43.68.103 9898/TCP,9999/TCP 35s + +In order to request access from outside, change the type to "LoadBalancer": + +.. code-block:: shell-session + + kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}' + +Check the services again: + +.. code-block:: shell-session + + kubectl get svc + +Now we can see that the service type has changed to LoadBalancer, and "EXTERNAL-IP" switched to pending state: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 37m + podinfo LoadBalancer 10.43.68.103 9898:32486/TCP,9999:30455/TCP 3m45s + +Going into the Netris Controller web interface, navigate to **Services → L4 Load Balancer**, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name **"podinfo-xxxxxxxx"** + +.. image:: /images/sandbox-podinfo-prov.png + :align: center + :alt: Sandbox PodInfo Provisioning + :target: ../../_images/sandbox-podinfo-prov.png + +After provisioning has finished, let's one more time look at service in k8s: + +.. code-block:: shell-session + + kubectl get svc + +You can see that "EXTERNAL-IP" has been injected into Kubernetes: + +.. code-block:: shell-session + + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.43.0.1 443/TCP 29m + podinfo LoadBalancer 10.43.42.190 50.117.59.205 9898:30771/TCP,9999:30510/TCP 5m14s + +Let's try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command): + +.. code-block:: shell-session + + curl 50.117.59.205:9898 + +The application is now accessible directly on the internet: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +As seen, "PodInfo" developers decided to expose 9898 port for HTTP, let's switch it to 80: + +.. code-block:: shell-session + + kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]' + +Wait a few seconds, you can see the provisioning process on the controller: + +.. image:: /images/sandbox-podinfo-ready.png + :align: center + :alt: Sandbox PodInfo Ready + :target: ../../_images/sandbox-podinfo-ready.png + +Curl again, without specifying a port: + +.. code-block:: shell-session + + curl 50.117.59.205 + +The output is similar to this: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-6gfwx", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +You can also verify the application is reachable by putting this IP address directly into your browser. + +.. topic:: Milestone 1 + + Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee. + + +Using Netris Custom Resources +============================= + +Introduction to Netris Custom Resources +--------------------------------------- + +In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let's take a look at a few common examples: + +L4LB Custom Resource +-------------------- + +In the previous section, when we changed the service type from "ClusterIP" to "LoadBalancer", Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let's see them: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, there are two L4LB resources, one for each podinfo's service port: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.205 80/TCP US/NYC Admin OK 7m21s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.205 9999/TCP US/NYC Admin OK 15m + +You can't edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications. + +Instead, let's create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB's backends will be "srv04-nyc" & "srv05-nyc" on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > srv04-5-nyc-http.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: L4LB + metadata: + name: srv04-5-nyc-http + spec: + ownerTenant: Admin + site: US/NYC + state: active + protocol: tcp + frontend: + port: 80 + backend: + - 192.168.45.64:80 + - 192.168.46.65:80 + check: + type: tcp + timeout: 3000 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f srv04-5-nyc-http.yaml + +Inspect the new L4LB resources via kubectl: + +.. code-block:: shell-session + + kubectl get l4lb + +As you can see, provisioning started: + +.. code-block:: shell-session + + NAME STATE FRONTEND PORT SITE TENANT STATUS AGE + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80 active 50.117.59.205 80/TCP US/NYC Admin OK 9m56s + podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999 active 50.117.59.205 9999/TCP US/NYC Admin OK 17m + srv04-5-nyc-http active 50.117.59.206 80/TCP US/NYC Admin Provisioning 5s + +When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output: + +.. code-block:: shell-session + + curl 50.117.59.206 + +You will see the servers' hostname in curl output: + +.. code-block:: shell-session + + SRV04-NYC + +You can also inspect the L4LB in the Netris Controller web interface: + +.. image:: /images/sandbox-l4lbs.png + :align: center + :alt: Sandbox L4LBs + :target: ../../_images/sandbox-l4lbs.png + +V-Net Custom Resource +--------------------- + +If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn't create "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"` manual. If that's the case, let's create it from Kubernetes using the V-Net custom resource. + +Let's create our V-Net manifest: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +Let's check our V-Net resources in Kubernetes: + +.. code-block:: shell-session + + kubectl get vnet + +As you can see, provisioning for our new V-Net has started: + +.. code-block:: shell-session + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 10s + +After provisioning has completed, the L4LB's checks should work for both backend servers, and incoming requests should be balanced between them. + +Let's curl several times to see that: + +.. code-block:: shell-session + + curl 50.117.59.206 + +As we can see, the curl request shows the behavior of "round robin" between the backends: + +.. code-block:: shell-session + + SRV05-NYC + curl 50.117.59.206 + + SRV05-NYC + curl 50.117.59.206 + + SRV04-NYC + curl 50.117.59.206 + + SRV04-NYC + +.. note:: + + *If intermittently the result of the curl command is "Connection timed out", it is likely that the request went to the srv05-nyc backend, and the "Default ACL Policy" is set to "Deny". To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the* :ref:`ACL documentation`. + +BTW, if you already created "vnet-customer" V-Net as described in the :ref:`"Learn by Creating Services"`, you may import that to k8s, by adding ``resource.k8s.netris.ai/import: "true"`` annotation in V-Net manifest, the manifest should look like this: + +.. code-block:: shell-session + + cat << EOF > vnet-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: VNet + metadata: + name: vnet-customer + annotations: + resource.k8s.netris.ai/import: "true" + spec: + ownerTenant: Demo + guestTenants: [] + vlanId: "46" + sites: + - name: US/NYC + gateways: + - prefix: 192.168.46.1/24 + switchPorts: + - name: swp5@sw12-nyc + untagged: "no" + - name: swp5@sw21-nyc + untagged: "no" + EOF + +Apply it: + +.. code-block:: shell-session + + kubectl apply -f vnet-customer.yaml + +After applying the manifest containing "import" annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes. + +.. code-block:: shell-session + + kubectl get vnet + + NAME STATE GATEWAYS SITES OWNER STATUS AGE + vnet-customer active 192.168.46.1/24 US/NYC Demo Active 2m + +BGP Custom Resource +------------------- + +Let's create a new BGP peer, that is listed in the :ref:`"Learn by Creating Services"`. + +Create a yaml file: + +.. code-block:: shell-session + + cat << EOF > iris-isp2-ipv4-customer.yaml + apiVersion: k8s.netris.ai/v1alpha1 + kind: BGP + metadata: + name: iris-isp2-ipv4-customer + spec: + site: US/NYC + hardware: SoftGate2 + neighborAs: 65007 + transport: + name: swp16@sw02-nyc + vlanId: 1092 + localIP: 50.117.59.118/30 + remoteIP: 50.117.59.117/30 + description: Example BGP to ISP2 + prefixListOutbound: + - permit 50.117.59.192/28 le 32 + EOF + +And apply it: + +.. code-block:: shell-session + + kubectl apply -f iris-isp2-ipv4-customer.yaml + +Check created BGP: + +.. code-block:: shell-session + + kubectl get bgp + +Allow up to 1 minute for both sides of the BGP sessions to come up: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled Link Up 65007 50.117.59.118/30 50.117.59.117/30 15s + +Then check the state again: + +.. code-block:: shell-session + + kubectl get bgp + +The output is similar to this: + +.. code-block:: shell-session + +NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.118/30 50.117.59.117/30 2m3s + +Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously. + +Return to the Netris Controller and navigate to **Network → Topology** to see the new BGP neighbor you created. + +Importing Existing Resources from Netris Controller to Kubernetes +----------------------------------------------------------------- + + You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/import: "true" + +Otherwise, if try to apply them without the "import" annotation, the Netris Operator will complain that the resource with such name or specs already exists. + +After importing resources to k8s, they will belong to the Netris Operator, and you won't be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources. + +Reclaim Policy +-------------- + +There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use "reclaimPolicy" annotation: + +.. code-block:: yaml + + resource.k8s.netris.ai/reclaimPolicy: "retain" + +Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the ``"delete"`` value to ``"retain"`` for key ``resource.k8s.netris.ai/reclaimPolicy`` in the resource annotation. After that, you'll be able to delete any Netris Custom Resource from Kubernetes, and it won't be deleted from the Netris Controller. + +.. seealso:: + + See all options and examples for Netris Custom Resources `here `_. + + +Netris Calico CNI Integration +============================= + +Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the `Calico docs `_. + +Integration is very simple, you just need to add the annotation in calico's ``bgpconfigurations`` custom resource. Before doing that, let's see the current state of ``bgpconfigurations``: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +As we can see, ``nodeToNodeMeshEnabled`` is enabled: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: true + +Let's enable the "netris-calico" integration: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true' + +Let's check our BGP resources in k8s: + +.. code-block:: shell-session + + kubectl get bgp + +Here are our freshly created BGPs, one for each k8s node: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957241; time: 00:15:03 65007 50.117.59.118/30 50.117.59.117/30 16m + sandbox-srv06-192.168.110.66 enabled 4230000000 192.168.110.1/24 192.168.110.66/24 37s + sandbox-srv07-192.168.110.67 enabled 4230000001 192.168.110.1/24 192.168.110.67/24 37s + sandbox-srv08-192.168.110.68 enabled 4230000002 192.168.110.1/24 192.168.110.68/24 37s + +You might notice that peering neighbor AS is different from Calico's default 64512. The is because the Netris Operator is setting a particular AS number for each node. + +Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again: + +.. code-block:: shell-session + + kubectl get bgp + +As we can see, our BGP peers have become established: + +.. code-block:: shell-session + + NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE + iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957194; time: 00:18:24 65007 50.117.59.118/30 50.117.59.117/30 19m + sandbox-srv06-192.168.110.66 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000000 192.168.110.1/24 192.168.110.66/24 2m7s + sandbox-srv07-192.168.110.67 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000001 192.168.110.1/24 192.168.110.67/24 2m7s + sandbox-srv08-192.168.110.68 enabled bgp: Established; prefix: 1; time: 00:01:26 N/A 4230000002 192.168.110.1/24 192.168.110.68/24 2m7s + +Now let's check if ``nodeToNodeMeshEnabled`` is still enabled: + +.. code-block:: shell-session + + kubectl get bgpconfigurations default -o yaml + +It is disabled, which means the "netris-calico" integration process is finished: + +.. code-block:: yaml + + apiVersion: projectcalico.org/v3 + kind: BGPConfiguration + metadata: + annotations: + ... + manage.k8s.netris.ai/calico: "true" + ... + name: default + ... + spec: + nodeToNodeMeshEnabled: false + +.. note:: + + Netris Operator won't disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established. + +Finally, let's check if our earlier deployed "Podinfo" application is still working when Calico Node-to-Node mesh is disabled: + +.. code-block:: shell-session + + curl 50.117.59.205 + +Yes, it works: + +.. code-block:: json + + { + "hostname": "podinfo-7cf557d9d7-nb2t7", + "version": "6.6.0", + "revision": "357009a86331a987811fefc11be1350058da33fc", + "color": "#34577c", + "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif", + "message": "greetings from podinfo v6.6.0", + "goos": "linux", + "goarch": "amd64", + "runtime": "go1.21.7", + "num_goroutine": "8", + "num_cpu": "2" + } + +Disabling Netris-Calico Integration +----------------------------------- + +To disable "Netris-Calico" integration, delete the annotation from Calico's ``bgpconfigurations`` resource: + +.. code-block:: shell-session + + kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico- + +or change its value to ``"false"``. + +.. topic:: Milestone 2 + + Congratulations on completing Milestone 2! diff --git a/en/latest/_sources/sandbox/Sandbox9/sandbox-info.rst.txt b/en/latest/_sources/sandbox/Sandbox9/sandbox-info.rst.txt new file mode 100644 index 0000000000..1750c6c36e --- /dev/null +++ b/en/latest/_sources/sandbox/Sandbox9/sandbox-info.rst.txt @@ -0,0 +1,136 @@ +************************* +Welcome to Netris Sandbox +************************* + +.. contents:: + :local: + +Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the :ref:`"Provided Example Configurations"` document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on `Slack `__. + +The credentials for the sandbox have been provided to you via email in response to your Sandbox request. + +The Sandbox environment includes: + + +* :ref:`Netris Controller`: A cloud-hosted Netris Controller, loaded with examples. +* :ref:`Switching fabric`: Two spine switches and four leaf switches, all operated by Netris. +* :ref:`SoftGates`: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris. +* **Linux servers**: Five Linux servers, with root access where you can run any applications for your tests. +* **Kubernetes cluster**: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests. +* **ISP**: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses. + +.. _s9-topology: + +Topology diagram +================ + +.. image:: /images/sandbox_topology_new.png + :align: center + :alt: Sandbox Topology + :target: ../../_images/sandbox_topology_new.png + + +Netris Controller +================= +https://sandbox9.netris.io + +Linux servers +============= + +Example pre-configured Netris services: + * **srv01-nyc**, **srv02-nyc**, **srv03-nyc** & **Netris Controller** - are consuming :ref:`"ROH (Routing on the Host)"` Netris example service, see **Services → ROH.**. + * **srv01-nyc**, **srv02-nyc** - can be configured with :ref:`"L3 Load Balancer (Anycast LB)"`, see **Services → L3 Load Balancer**. + * **srv04-nyc**, **srv05-nyc**, **srv06-nyc**, **srv07-nyc** & **srv08-nyc** - are consuming :ref:`"V-Net (routed VXLAN)"` Netris service, see **Services → V-Net**. + * **srv06-nyc**, **srv07-nyc**, **srv08-nyc** - are members of a 3 node Kubernetes cluser, and the K8s API servers are behind :ref:`"L4 Load Balancer (L4LB)"`, see **Services → L4 Load Balancer**. + + +**Accessing the Linux servers:** + +.. code-block:: shell-session + + srv01-nyc: ssh demo@216.172.128.209 -p 30061 + srv02-nyc: ssh demo@216.172.128.209 -p 30062 + srv03-nyc: ssh demo@216.172.128.209 -p 30063 + srv04-nyc: ssh demo@216.172.128.209 -p 30064 + srv05-nyc: ssh demo@216.172.128.209 -p 30065 + + +Kubernetes cluster +================== +This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the **netris-operator**. Step-by-step instructions are included in the :ref:`"Learn Netris operations with Kubernetes"` document. + + +Upstream ISP +============ +This Sandbox also provides an upstream ISP service with real-world Internet routing configured through :ref:`"BGP"`. +There are two pre-configured examples under **Network → E-BGP** , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS. + +ISP settings: + +.. code-block:: shell-session + + (pre-configured examples) + Name: iris-isp1-ipv4-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1091 + Local Address: 50.117.59.114/30 + Remote Address: 50.117.59.113/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.192/28 le 32 + + Name: iris-isp1-ipv6-example + BGP Router: Softage1 + Switch Port: swp16@sw01-nyc + Neighbor AS: 65007 + VLAN ID: 1091 + Local Address: 2607:f358:11:ffc0::13/127 + Remote Address: 2607:f358:11:ffc0::12/127 + Prefix List Inbound: permit ::/0 + Prefix List Outbound: permit 2607:f358:11:ffc9::/64 + + (configurable by you) + BGP Router: Softage2 + Switch Port: swp16@sw02-nyc + Neighbor AS: 65007 + VLAN ID: 1092 + Local Address: 50.117.59.118/30 + Remote Address: 50.117.59.117/30 + Prefix List Inbound: permit 0.0.0.0/0 + Prefix List Outbound: permit 50.117.59.192/28 le 32 + + +Networks Used +============= +Allocations and subnets defined under :ref:`"IPAM"`, see **Network → IPAM**. + +.. code-block:: shell-session + + | MANAGEMENT Allocation: 10.254.45.0/24 + |___ MANAGEMENT Subnet: 10.254.45.0/24 + + | LOOPBACK Allocation: 10.254.46.0/24 + |___ LOOPBACK Subnet: 10.254.46.0/24 + + | ROH Allocation: 192.168.44.0/24 + |___ ROH Subnet: 192.168.44.0/24 + + | EXAMPLE Allocation: 192.168.45.0/24 + |___ EXAMPLE Subnet: 192.168.45.0/24 + + | CUSTOMER Allocation: 192.168.46.0/24 + |___ CUSTOMER Subnet: 192.168.46.0/24 + + | K8s Allocation: 192.168.110.0/24 + |___ K8s Subnet: 192.168.110.0/24 + + | PUBLIC IPv4 Allocation: 50.117.59.192/28 + |___ PUBLIC LOOPBACK Subnet: 50.117.59.192/30 + |___ NAT Subnet: 50.117.59.196/30 + |___ L3 LOAD BALANCER Subnet: 50.117.59.200/30 + |___ L4 LOAD BALANCER Subnet: 50.117.59.204/30 + + | EXAMPLE IPv6 Allocation: 2607:f358:11:ffc9::/64 + |___ EXAMPLE IPv6 Subnet: 2607:f358:11:ffc9::/64 diff --git a/en/latest/_sources/softgate-performance.rst.txt b/en/latest/_sources/softgate-performance.rst.txt new file mode 100644 index 0000000000..1b1de3a508 --- /dev/null +++ b/en/latest/_sources/softgate-performance.rst.txt @@ -0,0 +1,15 @@ +.. meta:: + :description: Netris Network Policies & Protocol Configuration + +#################### +SoftGate Performance +#################### + +The following tested results are offered to help properly size the hardware needed for a SoftGate with various types of services: + + +.. csv-table:: SoftGate Performance + :file: tables/softgate-performance.csv + :widths: 30, 10, 15, 15, 10, 10, 10 + :header-rows: 0 + diff --git a/en/latest/_sources/supported-networks.rst.txt b/en/latest/_sources/supported-networks.rst.txt new file mode 100644 index 0000000000..e68c713389 --- /dev/null +++ b/en/latest/_sources/supported-networks.rst.txt @@ -0,0 +1,43 @@ +.. meta:: + :description: Reference Network Architectures + +############################### +Reference Network Architectures +############################### + + +Unmanaged Switch & Netris SoftGate +---------------------------------- + +.. image:: images/slide-1.png + :align: center + :alt: Unmanaged Switch & SoftGate + +-------------------------- + +Unmanaged Switch & SoftGate (HA) +--------------------------------------- + +.. image:: images/slide-2.png + :align: center + :alt: Unmanaged Switch & SoftGate (HA) + +-------------------------- + +Netris Managed Switch & SoftGate (HA) +-------------------------------------------- + +.. image:: images/slide-3.png + :align: center + :alt: Netris Managed Switch & SoftGate small data center (HA) + +-------------------------- + +Netris Managed Switch & SoftGate scalable data center (HA) +---------------------------------------------------------- + +.. image:: images/slide-4.png + :align: center + :alt: Netris Managed Switch & SoftGate scalable data center (HA) + + diff --git a/en/latest/_sources/supported-platform-matrix.rst.txt b/en/latest/_sources/supported-platform-matrix.rst.txt new file mode 100644 index 0000000000..258a15a369 --- /dev/null +++ b/en/latest/_sources/supported-platform-matrix.rst.txt @@ -0,0 +1,621 @@ +================================================= +Netris Supported Functionality & Platforms Matrix +================================================= + +Switch Fabric Management Functions +================================== +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Nvidia Spectrum + - Dell SONiC + - Arista EOS + - EdgeCore SONiC + - Equinix Metal + - PhoenixNAP + * - Fabric Manager + - Day0, Day1, and Day2 switch fabric operations. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - Parallel Fabrics + - Manage multiple isolated switch fabrics. (example: East-West and North-South) + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - Topology Manager + - Design and operate the switch fabric. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - Maintenance Mode + - Offload a network node for a maintenance. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - IPAM + - Manage IP subnets. Assign RBAC, multi-tenancy, and service-based rules and roles to IP address resources. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - Looking Glass + - Lookup underlay and overlay routing info of any managed network node without SSH-ing. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - Monitoring: Switch Ports + - Automatic monitoring of Link statuses, link utilization, laser signal levels, errors, packets. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - Monitoring: Resources + - Automatic monitoring of CPU, RAM, Disk, and ASIC resources. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - Monitoring: Sensors + - Automatic monitoring of temperature, fans, power supply statuses. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - Monitoring: System Processes + - Automatic monitoring of critical system processes. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - Topology Validation + - Detect wiring errors. + - ✔ + - Dec/2024 + - Nov/2024 + - Dec/2024 + - N/A + - N/A + * - BGP Unnumbered + - Any network topology with BGP unnumbered underlay + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - BGP Numbered + - Any network topology with BGP numbered underlay + - ✔ + - Dec/2024 + - Nov/2024 + - Dec/2024 + - N/A + - N/A + +External Routing Functions +========================== + +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Nvidia Spectrum + - Dell-SONiC + - Arista EOS + - EdgeCore-SONiC + - Equinix Metal + - PhoenixNAP + * - External BGP (SoftGate) + - Terminate full routing table on SoftGate Gateway-server. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - External BGP (Switch) + - Peer with external routers. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - BGP Route-Maps + - Create chain of BGP rules. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - Static Routes + - Define static routing rules. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + + + +Cloud Networking Functions & Constructs +======================================= + +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Nvidia Spectrum + - Dell-SONiC + - Arista EOS + - EdgeCore-SONiC + - Equinix Metal + - PhoenixNAP + * - VPC (Virtual Private Cloud) + - Isolated VPCs, VRFs. Overlapping IPs supported. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - V-Net (Subnet) + - L3VPN VXLAN or L2VPN VXLAN with an anycast default Gateway, and built-in DHCP. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - ✔ + - ✔ + * - Server Cluster (Profiling) + - Create network constructs template, then apply it on groups of servers. + - ✔ + - Dec/2024 + - Nov/2024 + - Dec/2024 + - TBD + - TBD + * - Internet Gateway + - Provide shared Internet access to V-Nets and VPC + - ✔ + - ✔ + - Nov/2024 + - Dec/2024 + - ✔ (single VPC) + - ✔ (single VPC) + * - NAT Gateway + - Provide shared DNAT, PAT, 1:1 NAT to multiple V-Nets and multiple VPCs + - ✔ + - ✔ + - Nov/2024 + - Dec/2024 + - ✔ (single VPC) + - ✔ (single VPC) + * - L4 Load Balancer + - Provide on-demand elastic load balancer service to hosts in multiple V-Nets and multiple VPCs + - ✔ + - ✔ + - Nov/2024 + - Dec/2024 + - ✔ (single VPC) + - ✔ (single VPC) + * - SiteMesh + - Wireguard-based Site-to-Site VPN between multiple regions/sites. (single VPC) + - ✔ + - TBD + - Nov/2024 + - ✔ + - ✔ + - ✔ + + +Overlay Network Features +========================== +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Nvidia Spectrum + - Dell-SONiC + - Arista EOS + - EdgeCore-SONiC + - Equinix Metal + - PhoenixNAP + * - L2VPN VXLAN VLAN Aware + - L2VPN VXLAN with VLAN tagged or untagged termination on switch port. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + * - L2VPN VXLAN VLAN Unaware + - L2VPN VXLAN with VLAN tagged or untagged termination on switch port supporting different VLAN IDs on different end points. + - N/A + - N/A + - Nov/2024 + - N/A + - N/A + - N/A + * - L3VPN VXLAN + - L3VPN VXLAN, Commonly used in high performance computing, such as AI clusters. + - ✔ + - TBD + - TBD + - Dec/2024 + - N/A + - N/A + * - EVPN-MH / VXLAN-ESI + - EVPN MultiHoming based on VXLAN and ESI for automatic Active-Active server network multihoming + - ✔ + - Dec/2024 + - Nov/2024 + - N/A + - N/A + - N/A + * - LACP + - Link Aggregation or Active-Standby server multihoming. + - ✔ + - ✔ + - Nov/2024 + - Dec/2024 + - N/A + - N/A + * - MC-LAG + - Traditional MC-LAG-based server multihoming + - ✔ + - TBD + - TBD + - Dec/2024 + - N/A + - N/A + + +AI Specific Functions +===================== +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Nvidia Spectrum + - Dell-SONiC + - Arista EOS + - EdgeCore-SONiC + - Equinix Metal + - PhoenixNAP + * - Spectrum-X + - AI GPU cluster switch fabric operation for Nvidia Spectrum-X + - ✔ + - N/A + - N/A + - N/A + - N/A + - N/A + * - Rail-optimized designs + - Topology and best practices initialization module for rail-optimized fabrics + - ✔ + - N/A + - N/A + - Dec/2024 + - N/A + - N/A + * - QoS for RoCE + - Enable QoS for RoCE workloads + - ✔ + - N/A + - N/A + - N/A + - N/A + - N/A + * - RoCE Adaptive Routing + - Enable RoCE adaptive routing + - ✔ + - N/A + - N/A + - N/A + - N/A + - N/A + * - RoCE Congestion Control + - Enable automatic congestion control for RoCE workloads + - ✔ + - N/A + - N/A + - N/A + - N/A + - N/A + * - DPU/Host zero-touch configuration + - Automatically configure IP addresses, routing, RoCE and other DPU/SuperNIC specific configuration on GPU servers + - ✔ + - N/A + - N/A + - N/A + - N/A + - N/A + + +Security +======== +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Nvidia Spectrum + - Dell-SONiC + - Arista EOS + - EdgeCore-SONiC + - Equinix Metal + - PhoenixNAP + * - Network ACLs + - Centralized Network Access Control Lists. + - ✔ + - ✔ + - Nov/2024 + - Dec/2024 + - N/A + - N/A + * - Managed Device Profiling + - Managed switch & SoftGate protection from unwanted access, push administrative and system settings (NTP, DNS, timezone, etc.) + - ✔ + - ✔ + - Nov/2024 + - Dec/2024 + - N/A + - N/A + * - Audit Logs + - Log all controller access and changes. + - ✔ + - ✔ + - Nov/2024 + - ✔ + - N/A + - N/A + + +Administration +============== + +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Globally + * - Role Based Access Control + - Who can view and edit which aspects of the system. + - ✔ + * - Multi-Tenancy + - Network resource delegation to tenants. + - ✔ + +Management Interfaces +===================== + +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Globally + * - Web Console + - Manage through intuitive web interface. + - ✔ + * - RestAPI + - Integrate your other systems or your customer-facing portal with Netris consuming RestAPIs. + - ✔ + * - IaC: Terraform + - Manage your infrastructure as a code using Terraform. + - ✔ + + +Hypervisor/Worker node specific functionality +============================================= + +.. list-table:: + :header-rows: 0 + + * - Function + - Description + - Kubernetes + - Vmware + - Apache Cloud Stack + - OpenStack + - Harvester + - Proxmox + * - L4 Load Balancer + - Layer-4 container or vm/server load balancer with health checks. + - ✔ (native & automatic) + - ✔ (need to specify backend IPs) + - Dec/2024 + - ✔ (need to specify backend IPs) + - ✔ (need to specify backend IPs) + - ✔ (need to specify backend IPs) + * - VPC to internal routing peering + - Automatically route internal networks into VPC routing table (allow containers communicate with VMs). + - ✔ + - N/A + - Dec/2024 + - Dec/2024 + - TBD + - TBD + * - Automatic VXLAN/VLAN + - Automatically provision VXLAN/VLAN on switch fabric and include appropriate switch ports when virtual network is created in the hypervisor. + - TBD + - ✔ + - ✔ + - Dec/2024 + - TBD + - TBD + * - HBN Host-based networking. + - Terminate VTEPs on the hypervisor host. Scale beyond VLAN limits + - Dec/2024 + - TBD + - Dec/2024 + - Dec/2024 + - TBD + - TBD + * - HBN on DPU + - Host-based networking. Terminate VTEPs on the hypervisor host DPU. Scale beyond VLAN limits with accelerated performance + - 2025 + - TBD + - 2025 + - 2025 + - TBD + - TBD + +============================== +SoftGate Data Plane Variations +============================== + +SoftGate is Netris data plane for Internet Gateway, NAT Gateway, Network Access Control, Elastic Load Balancer, and Site-to-Site VPN functions. + +.. list-table:: + :header-rows: 0 + + * - Flavor + - Common Use Case + - Availability + - Tenancy/VPC + - Handoff + - Packet Forwarding + - HA & Scalability + - Ethernet Environment + - NIC + - CPU + - RAM + - Disk + - Performance (w/ 100 NAT rules) + * - SoftGate + - Bare metal cloud site, Edge site, Remote office. + - ✔ + - Single + - VLAN + - Linux w/ Netris optimizations + - Active/Standby - 2 nodes + - Dot1q: Equinix Metal, PhoenixNAP, pre-configured VLAN-range on any Ethernet switches. + - Any + - Intel or AMD + - 16-64GB + - 300GB + - Dual Gold 6336Y (48c x 2.3GHz) - 11Gbps / 1.8Mpps + * - SoftGate PRO + - Private Cloud, Public Cloud Border Gateway, Enterprise Cloud, Vmware NSX alternative. + - ✔ + - Single + - VLAN + - Netris DPDK + - Active/Standby - 2 nodes + - Netris Switch-Fabric + - Nvidia Connect-X 5, 6 100Gbe + - Intel XEON (required for DPDK) + - 128GB + - 300GB + - Intel XEON Platinum 20+ cores - 100Gbps / 25Mpps + * - SoftGate HS (HyperScale) + - Scalable GPU & CPU Cloud Services Provider. + - ✔ + - Multi + - VXLAN + - Linux w/ Netris optimizations + - Active/Active - Horizontally scalable + - Netris Switch-Fabric + - Any OK. Nvidia Connect-X is recommended + - Intel or AMD + - 128-256GB + - 300GB + - Dual Platinum 8352Y (64c x 2.2GHz) - 22Gbps / 3.5 Mpps + * - SoftGate HS PRO + - Scalable GPU & CPU Cloud Services Provider. + - 2025/Q2 + - Multi + - VXLAN + - Netris + - Active/Active - Horizontally scalable + - Netris Switch-Fabric + - Nvidia Connect-X 5, 6, 7 + - Intel, AMD (TBD) + - 256GB+ + - 300GB + - TBD + +============================================ +Netris and NOS versions compatibility matrix +============================================ + +.. list-table:: + :header-rows: 0 + + * - **Netris Version** + - **Switch & OS** + - **Bare Metal Cloud** + - **SoftGate OS** + - **Availability** + * - 4.4.0 + - Nvidia Cumulus 5.11, Dell SONiC 4.4, EdgeCore SONiC 202211-331 + - Equinix Metal, PhoenixNAP BMC + - SoftGate HS: Ubuntu 24.04, SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 + - Dec/2024 + * - 4.3.0 + - Nvidia Cumulus 5.9, Dell SONiC 4.1, EdgeCore SONiC 12.3 + - Equinix Metal, PhoenixNAP BMC + - SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 (non-pro) + - ✔ + * - 4.2.0 + - Nvidia Cumulus 5.7, Dell SONiC 4.1, EdgeCore SONiC 12.3 + - Equinix Metal, PhoenixNAP BMC + - SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 + - ✔ + * - 4.1.1 + - Nvidia Cumulus 5.7, EdgeCore SONiC 12.3 + - Equinix Metal, PhoenixNAP BMC + - SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 + - ✔ + * - 4.0.0 + - Nvidia Cumulus 5.7, EdgeCore SONiC 12.3 + - Equinix Metal, PhoenixNAP BMC + - SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 + - ✔ + * - 3.5.0 + - Nvidia Cumulus 5.7, EdgeCore SONiC 12.3 + - Equinix Metal, PhoenixNAP BMC + - SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 + - ✔ + * - 3.4.1 + - Nvidia Cumulus 5.7, EdgeCore SONiC 12.3 + - Equinix Metal, PhoenixNAP BMC + - SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 + - ✔ diff --git a/en/latest/_sources/supported-switch-hardware.rst.txt b/en/latest/_sources/supported-switch-hardware.rst.txt new file mode 100644 index 0000000000..d10251977b --- /dev/null +++ b/en/latest/_sources/supported-switch-hardware.rst.txt @@ -0,0 +1,431 @@ +===================== +Hardware Requirements +===================== + +Netris Controller +================= + +We recommend two Ubuntu 24.04 servers with the below specs. Netris controller can also run on other Linux OSes or on a shared Kubernetes cluster. + +.. list-table:: + :header-rows: 0 + + * - **Use Case** + - **CPU Cores** + - **RAM** + - **SSD/NVME** + - **Network** + * - Leaf/Spine 1-30 switches + - 4 + - 32 GB + - 300 GB + - 2x 1GbE+ NIC + * - Leaf/Spine 30-100 switches + - 8 + - 32 GB + - 600 GB + - 2x 1GbE+ NIC + * - Leaf/Spine 100-300 switches + - 16 + - 64 GB + - 1 TB + - 2x 10GbE+ NIC + * - Leaf/Spine 300+ switches + - 32 + - 128 GB + - 2 TB + - 2x 10GbE+ NIC + * - Spectrum-X 1-8 SUs + - 16 + - 64 GB + - 1 TB + - 2x 10GbE+ NIC + * - Spectrum-X 16-32 SUs + - 32 + - 128 GB + - 2 TB + - 2x 10GbE+ NIC + * - Spectrum-X 32+ SUs + - 64 + - 256 GB + - 10 TB + - 2x 10GbE+ NIC + +Netris SoftGate HS (The Horizontally Scalable version) +========================================== + +A minimum of 4 dedicated servers are required for an HA (highly available) active-active SoftGate HS cluster. Two SoftGates will forward stateful traffic (SNAT), and two others will forward the stateless traffic (DNAT, 1:1 NAT, Layer-4 Load Balancing, etc.) Each group (stateful and stateless) can be scaled horizontally by deploying more servers as CPU & RAM utilization necessitates. + +Server specs: + +.. list-table:: + :header-rows: 0 + + * - + - **Minimum** + - **Recommended** + * - CPU (Modern Intel/AMD X86) + - 16 Cores + - 32 Cores + * - RAM + - 128 GB + - 256 GB + * - NIC prod + - 2x 10GbE + - 2x 25GbE + * - NIC OOB + - 1x 1GbE + - 1x 1GbE + * - Disk + - 300 GB + - 300 GB + * - OS + - Ubuntu 24.04 + - Ubuntu 24.04 + + + +========================= +Supported Switch Hardware +========================= + +Nvidia +====== +.. list-table:: + :header-rows: 0 + + * - **Manufacturer** + - **Model** + - **ASIC** + - **Ports** + - **NOS** + - **Caveats** + - **Supported** + * - Nvidia + - SN2010 + - Spectrum + - 18 x SFP28 25GbE + 4 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN2100 + - Spectrum + - 16 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN2201 + - Spectrum + - 48 x RJ45 + 4 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN2410 + - Spectrum + - 48 x SFP28 25GbE + 8 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN2700 + - Spectrum + - 32 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN3420 + - Spectrum 2 + - 48 x SFP28 25GbE + 12 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN3700C + - Spectrum 2 + - 32 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN3700 + - Spectrum 2 + - 32 x QSFP56 200GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN4410 + - Spectrum 3 + - 24 x QSFP28-DD 100G + 8 x QSFP-DD 400GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN4600C + - Spectrum 3 + - 64 x QSFP28 100GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN4600 + - Spectrum 3 + - 64 QSFP56 200GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN4700 + - Spectrum 3 + - 32 x QSFP-DD 400GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN5400 + - Spectrum 4 + - 64 x QSFP-DD 400GbE + 2 x SFP28 25GbE + - Cumulus Linux + - + - ✔ + * - Nvidia + - SN5600 + - Spectrum 4 + - 64 x OSFP 800GbE + 1 x SFP28 25GbE + - Cumulus Linux + - + - ✔ + + +Dell +====== +.. list-table:: + :header-rows: 0 + + * - **Manufacturer** + - **Model** + - **ASIC** + - **Ports** + - **NOS** + - **Caveats** + - **Supported** + * - Dell + - PowerSwitch S Series S5212F-ON + - Broadcom Trident III + - 12 x 25G SFP28 + 3 x 100GbE QSFP28 + - Dell-SONiC + - + - ✔ + * - Dell + - PowerSwitch S Series S5224F-ON + - Broadcom Trident III + - 24 x 25G SFP28 + 4 x 100GbE QSFP28 + - Dell-SONiC + - + - ✔ + * - Dell + - PowerSwitch S Series S5232F-ON + - Broadcom Trident III + - 32 x 100GbE QSFP28 + - Dell-SONiC + - + - ✔ + * - Dell + - PowerSwitch S Series S5248F-ON + - Broadcom Trident III + - 48 x 25G SFP28 + 4 x 100GbE QSFP28 + 2 x 200GbE QSFP28 + - Dell-SONiC + - + - ✔ + * - Dell + - PowerSwitch S Series S5296F-ON + - Broadcom Trident III + - 96 x 25G SFP28 + 8 x 100GbE QSFP28 + - Dell-SONiC + - + - ✔ + * - Dell + - PowerSwitch S Series S5448F-ON + - Broadcom Trident IV + - 48 x QSFP-DD 100GbE + 8 x QSFP-DD 400GbE + - Dell-SONiC + - + - ✔ + * - Dell + - PowerSwitch Z Series Z9664F-ON + - Broadcom Tomahawk IV + - 64 x QSFP-DD 400GbE + - Dell-SONiC + - + - ✔ + * - Dell + - PowerSwitch Z Series Z9432F-ON + - Broadcom Trident IV + - 32 x QSFP-DD 400GbE + - Dell-SONiC + - + - ✔ + + +EdgeCore +======== +.. list-table:: + :header-rows: 0 + + * - **Manufacturer** + - **Model** + - **ASIC** + - **Ports** + - **NOS** + - **Caveats** + - **Supported** + * - EdgeCore + - DCS201 (AS5835-54X) + - Broadcom Trident III + - 48 x 10G SFP+ + 6 x 100G QSFP28 + - EC-SONiC + - + - ✔ + * - EdgeCore + - DCS202 (AS5835-54T) + - Broadcom Trident III + - 48 x 10G RJ-45 + 6 x 100G QSFP28 + - EC-SONiC + - + - ✔ + * - EdgeCore + - DCS203 (AS7326-56X) + - Broadcom Trident III + - 48 x 25G SFP28 + 8 x 100G QSFP28+ 2 x 10G + - EC-SONiC + - + - ✔ + * - EdgeCore + - AS7726-32X + - Broadcom Trident III + - 32 x 100G QSFP28 + 2 x 10G SFP+ + - EC-SONiC + - + - ✔ + * - EdgeCore + - DCS510 (AS9716-32D) + - Broadcom Tomahawk 3 + - 32 x 400G QSFP-DD + - EC-SONiC + - + - ✔ + * - EdgeCore + - DCS511 (AS9737-32DB) + - Broadcom Tomahawk 4 + - 32 x 400G QSFP56-DD + - EC-SONiC + - + - ✔ + * - EdgeCore + - AIS800-64O + - Broadcom Tomahawk 5 + - 64 x OSFP800 + - EC-SONiC + - + - ✔ + +Arista +======== +.. list-table:: + :header-rows: 0 + + * - **Manufacturer** + - **Model** + - **ASIC** + - **Ports** + - **NOS** + - **Caveats** + - **Supported** + * - Arista + - 7020R + - Qumran + - 24 x 10G + 2 QSFP100; 32 x 10G + 2 QSFP100; 48 x 100/1000Mb + 6 SFP+; 48 x 100/1000Mb + 6 SFP+ + - EOS + - + - Dec/2024 + * - Arista + - 7050X3 + - Broadcom Trident III + - 32 x QSFP100; 48 x SFP25 + 12 x QSFP100; 48 x SFP25 + 8 x QSFP100; 48 x 10G-T + 8 x QSFP100 + - EOS + - + - Dec/2024 + * - Arista + - 7050X4 + - Trident-4 + - 32 QSFP-DD 400G + 2SFP+; 32 OSFP 400G + 2SFP+; 48 SFP-DD 100G + 8 QSFP-DD 400G; 48 DSFP 100G + 8 QSFPDD 400G; 24 QSFP56 200G + 8 QSFPDD 400G + 2SFP+; 48 QSFP28 + 8 QSFP-DD 400G + 2SFP+ + - EOS + - + - Dec/2024 + * - Arista + - 7060X4 + - Trident-4 + - 32 x QSFP-DD 800G + 2 x SFP+; 32 x QSFP-DD 800G + 2 x SFP+; 32 x OSFP 800G + 2 x SFP+; 64 x QSFP-DD 400G, 2 x SFP+; 32 x QSFP-DD + 1x SFP+; 56x QSFP100, 8 x QSFP-DD 400G + 1x SFP+ + - EOS + - + - Dec/2024 + * - Arista + - 7060X5 + - Tomahawk 4 + - 32 x QSFP-DD 800G + 2x SFP+; 32 x QSFP-DD 800G + 2x SFP+; 32 x OSFP 800G + 2x SFP+; 64 x QSFP-DD 400G + 2x SFP+; : 32 x QSFP-DD + 1 x SFP+; 56x QSFP100, 8 x QSFP-DD 400G, 1x SFP+ + - EOS + - + - Dec/2024 + * - Arista + - 7280R3A + - Jericho2 + - 144 x 100G or 36 x 400G + - EOS + - + - Dec/2024 + * - Arista + - 7280R3 + - Jericho2 + - 24 x 400G; 96 x 100G; 25G + 8 x 100G + - EOS + - + - Dec/2024 + * - Arista + - 7358X4 + - Trident-4 + - 128 x QSFP or 32 x OSFP / QSFP-DD + - EOS + - + - Dec/2024 + * - Arista + - 7358X4 + - Trident-4 + - 128 x QSFP or 32 x OSFP / QSFP-DD + - EOS + - + - Dec/2024 + * - Arista + - 7368X4 + - Tomahawk 3 + - 128 x 100G or 32 x 400G + - EOS + - + - Dec/2024 + * - Arista + - 7300R3 + - Trident-4 + - 256 wire-speed 40GbE ports + - EOS + - + - Dec/2024 + * - Arista + - 7500R3 + - Jericho, Jericho2 + - Up to 288 wire-speed 400G ports + - EOS + - + - Dec/2024 diff --git a/en/latest/_sources/switch-agent-installation.rst.txt b/en/latest/_sources/switch-agent-installation.rst.txt new file mode 100644 index 0000000000..7315fc5137 --- /dev/null +++ b/en/latest/_sources/switch-agent-installation.rst.txt @@ -0,0 +1,17 @@ +.. _Network-Switch-initial-setup: +.. meta:: + :description: Network Switch initial setup + +############################ +Network Switch Initial Setup +############################ + +.. toctree:: + :maxdepth: 2 + + Nvidia-Cumulus-v5.9+-Switch-initial-setup + Nvidia-Cumulus-v5-Switch-initial-setup + Ubuntu-SwitchDev-Switch-initial-setup + EdgeCore-SONiC-Switch-initial-setup + Dell-SONiC-Switch-initial-setup + Nvidia-Cumulus-v3.7-Switch-initial-setup diff --git a/en/latest/_sources/switch-ports.rst.txt b/en/latest/_sources/switch-ports.rst.txt new file mode 100644 index 0000000000..1a5946e3f1 --- /dev/null +++ b/en/latest/_sources/switch-ports.rst.txt @@ -0,0 +1,46 @@ +.. meta:: + :description: Switch Ports + +============ +Switch Ports +============ + +Switch ports can be directly managed in the **Switch Port** UI section. Both physical and virtual ports (extended, aggregate, etc…) will appear in this section once they have been added to inventory. The Netris Controller will automatically sync the list of available ports that appear on each device. + +The following options are available for editing on each port: + +* Description - Description of the port. +* Tenant - Tenant to whom the port is assigned, by default it is the owner tenant of the device to whom the port belongs to. +* Breakout - Available only for physical switch ports, used to split physical ports into multiple physical ports. When there is a need to use other supported option supported by switch not shown in the dropdown list user must set breakout to "Manual" and configure breakout manually on the switch. For certain platforms some ports need to be disabled to support breakout into other ports, for that option use "Disable" mode of breakout. For Cumulus, after configuration, user must manually restart switchd daemon on the switch via command "systemctl restart switchd". +* MTU - Maximum transmission unit of the port. +* Autoneg - Toggle autonegotiation. Available only for physical ports. +* Speed - Toggle speed. Available only for physical ports. +* Duplex - Toggle duplex. Available only for physical ports. +* Extension - Create extension ports. Available for physical and aggregate ports. +* Extension Name - Name for new extension. +* VLAN Range - VLAN id range for new extension port. + +.. image:: images/edit-port.png + :align: center + :class: with-shadow + +Example: Edit physical port + + +Quick action menu provides following actions for ports (note that Bulk Action also available for multiple ports: + +Edit - Edit the port. +Admin UP/Down - Toggle admin status of the port. +Add to V-net - Add selected port(s) to a V-net. + +.. image:: images/quick-action-ports.png + :align: center + :class: with-shadow + +Add to LAG - Add selected ports into a LAG. + +.. image:: images/add-to-lag-port.png + :align: center + :class: with-shadow + +Free Up Port - Detach port from all resources. diff --git a/en/latest/_sources/terraform-integration.rst.txt b/en/latest/_sources/terraform-integration.rst.txt new file mode 100644 index 0000000000..7fd6bedad2 --- /dev/null +++ b/en/latest/_sources/terraform-integration.rst.txt @@ -0,0 +1,317 @@ +.. meta:: + :description: Terraform: Netris provider + +########################## +Terraform: Netris provider +########################## + +Use Netris provider to interact with the many resources supported by Netris. You must configure the provider with the proper credentials before you can use it. +To learn the basics of Terraform using this provider, follow the hands-on `get started tutorials `_ on HashiCorp’s Learn platform. + +.. image:: images/diagrams_terraform.png + :align: center + +When you make changes in the Terraform files and apply them, Terraform automatically decides which part of your configuration is already deployed into Netris controller and what should be added or removed. + + +.. contents:: To create your first Terraform configuration: + :local: + + +Install Terraform +================= + +Download and install the `Terraform `_ + + + +Create a directory for Terraform files +====================================== + +#. Create a directory with any name, for example, ``netris-terraform``. It stores the configuration files and saved states for Terraform and your infrastructure. +#. Create a configuration file with the ``.tf`` extension in this directory, such as ``main.tf``. + + +Configure a provider +==================== + +1. At the beginning of the configuration file, specify the provider settings. + +.. code-block:: + + terraform { + required_providers { + netris = { + source = "netrisai/netris" + version = ">= 2.0.0" + } + } + } + + provider "netris" { + address = "" + login = "" + password = "" + } + +Specify the provider required arguments: + +* ``address`` - This is your Netris-Controller address (http://example.com). This can also be specified with the ``NETRIS_ADDRESS`` environment variable. +* ``login`` - This is your Netris-Controller login. This can also be specified with the ``NETRIS_LOGIN`` environment variable. +* ``password`` - This is your Netris-Controller password. This can also be specified with the ``NETRIS_PASSWORD`` environment variable. + +2. Execute the command ``terraform init`` in the folder with the configuration file. This command initializes the providers specified in the configuration files and lets you work with the provider resources and data sources. + + +Prepare an infrastructure plan +============================== + +By using Netris Terraform Provider, you can create all kinds of resources, such as Sites, IPAMs, Topology, Inventory, etc. +To create a resource, specify a set of required and optional parameters that define the resource properties. Such resource descriptions make up an infrastructure plan. + +Infrastructure provisioning in Netris starts with Site resources. The Netris-Controller comes with the initial site ``Default``. You can use it in your Terraform configuration files by getting its ID with the Terraform `Data source element `_. + +Let's create a separate file for site resource, and get its ID via Terraform Data source element. + +.. code-block:: + + cat << EOF > site.tf + data "netris_site" "default" { + name = "Default" + } + EOF + +Or, you can create a new ``Site`` resource, `here `_ is the detailed documentation with examples. + +Now, when we're clear on ``Site`` resource usage, let's define our IPAM. There are two types of IPAM resources in the Netris-Controller it’s ``Allocation`` and ``Subnet``. +IPAM resources only require ``tenantid`` field, let's get our default Admin tenant ID with the Data source element. + +.. code-block:: + + cat << EOF > tenant.tf + data "netris_tenant" "admin"{ + name = "Admin" + } + EOF + +Then, when we have the ``tenantid``, we can create IPAM resources. + +.. code-block:: + + cat << EOF > ipam.tf + resource "netris_allocation" "my-allocation-mgmt" { + name = "my-allocation-mgmt" + prefix = "192.0.2.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_allocation" "my-allocation-loopback" { + name = "my-allocation-loopback" + prefix = "198.51.100.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_allocation" "my-allocation-common" { + name = "my-allocation-common" + prefix = "203.0.113.0/24" + tenantid = data.netris_tenant.admin.id + } + + resource "netris_subnet" "my-subnet-mgmt" { + name = "my-subnet-mgmt" + prefix = "192.0.2.0/24" + tenantid = data.netris_tenant.admin.id + purpose = "management" + defaultgateway = "192.0.2.254" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-mgmt, + ] + } + + resource "netris_subnet" "my-subnet-loopback" { + name = "my-subnet-loopback" + prefix = "198.51.100.0/24" + tenantid = data.netris_tenant.admin.id + purpose = "loopback" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-loopback, + ] + } + + resource "netris_subnet" "my-subnet-common" { + name = "my-subnet-common" + prefix = "203.0.113.0/25" + tenantid = data.netris_tenant.admin.id + purpose = "common" + siteids = [data.netris_site.default.id] + depends_on = [ + netris_allocation.my-allocation-common, + ] + } + EOF + +With the command above, we've defined 6 resources, 3 of the type of Allocation, 3 of the type of Subnet, each Subnet resource has a different purpose. +For more details, get familiar with the IPAM `docs `_. + +Now, when we have all the required resources let's define our Inventory. +We're going to create 1 SoftGate, 1 switch and connect them with a link. + +.. code-block:: + + cat << EOF > inventory.tf + resource "netris_softgate" "my-softgate" { + name = "my-softgate" + tenantid = data.netris_tenant.admin.id + siteid = data.netris_site.default.id + description = "Softgate 1" + mainip = "auto" + mgmtip = "auto" + depends_on = [ + netris_subnet.my-subnet-mgmt, + netris_subnet.my-subnet-loopback, + ] + } + + resource "netris_switch" "my-switch" { + name = "my-switch" + tenantid = data.netris_tenant.admin.id + siteid = data.netris_site.default.id + description = "Switch 01" + nos = "cumulus_linux" + asnumber = "auto" + mainip = "auto" + mgmtip = "auto" + portcount = 16 + depends_on = [ + netris_subnet.my-subnet-mgmt, + netris_subnet.my-subnet-loopback, + ] + } + + resource "netris_link" "sg-to-sw" { + ports = [ + "swp1@my-softgate", + "swp16@my-switch" + ] + depends_on = [ + netris_softgate.my-softgate, + netris_switch.my-switch, + ] + } + EOF + +Next, let's define a local L3 network for our servers, suppose we want to connect 3 servers to our switch first 3 ports + +.. code-block:: + + cat << EOF > vnet.tf + resource "netris_vnet" "my-vnet" { + name = "my-vnet" + tenantid = data.netris_tenant.admin.id + state = "active" + sites{ + id = data.netris_site.default.id + gateways { + prefix = "203.0.113.1/25" + } + ports { + name = "swp1@my-switch" + vlanid = 1050 + } + ports { + name = "swp2@my-switch" + vlanid = 1050 + } + ports { + name = "swp3@my-switch" + } + } + depends_on = [ + netris_switch.my-switch, + netris_subnet.my-subnet-common, + ] + } + EOF + +And finally, we have to provide internet connectivity to our fabric, for that we'll define BGP resource. Suppose we're going to connect our ISP cable to the 10th port of our switch, and want to establish the BGP session on our Softgate. + +.. code-block:: + + cat << EOF > bgp.tf + data "netris_port" "swp10_my_switch"{ + name = "swp10@my-switch" + depends_on = [netris_switch.my-switch] + } + + resource "netris_bgp" "my-bgp" { + name = "my-bgp" + siteid = data.netris_site.default.id + hardware = "my-softgate" + neighboras = 23456 + portid = data.netris_port.swp10_my_switch.id + vlanid = 3000 + localip = "172.16.0.2/30" + remoteip = "172.16.0.1/30" + description = "My First BGP" + prefixlistinbound = ["deny 127.0.0.0/8 le 32", "permit 0.0.0.0/0 le 24"] + prefixlistoutbound = ["permit 192.0.2.0/24", "permit 198.51.100.0/24 le 25", "permit 203.0.113.0/24 le 26"] + depends_on = [netris_link.sg-to-sw] + } + EOF + +.. note:: + + For more information about all resources, how to create and manage them in Terraform, see the `provider's documentation `_. + +Now, when we've done with the configuration files, let's check whether they are valid + +.. code-block:: shell-session + + terraform validate + +If the configuration is valid, the following message is returned: + +.. code-block:: shell-session + + Success! The configuration is valid. + + + +Create resources +================ + +1. After preparing and checking the configuration, run the command: + +.. code-block:: shell-session + + terraform plan + +The terminal will display a list of resources with parameters. This is a test step. No resources are created. If there are errors in the configuration, Terraform points them out. + +2. To create resources, run the command: + +.. code-block:: shell-session + + terraform apply + +3. Confirm the resource creation: type ``yes`` in the terminal and press **Enter**. + +Terraform will create all the required resources and the terminal will display the progress. After creation, you can check resource availability and their settings in the Netris-Controller UI. + + +Delete resources +================ + +1. To delete resources created using Terraform: + +Run the command: + +.. code-block:: shell-session + + terraform destroy + +After the command is executed, the terminal will display a list of resources to be deleted. + +2. Type ``yes`` to confirm their deletion and press **Enter**. diff --git a/en/latest/_sources/topology-management.rst.txt b/en/latest/_sources/topology-management.rst.txt new file mode 100644 index 0000000000..134f281a53 --- /dev/null +++ b/en/latest/_sources/topology-management.rst.txt @@ -0,0 +1,118 @@ +.. _topology-management: +.. meta:: + :description: Topology Management + +========= +Inventory +========= + +The Inventory section allows you to add/edit/delete network switches and SoftGates (VPC gateways). Initial setup of a Netris managed network is a three step process: + +#. Create Inventory Profiles. +#. Adding Switches. +#. Adding Softgates. + +Inventory Profiles +================== + +Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/SoftGate is allowed. As soon as the inventory profile is attached to a device it denies all traffic destined to the device except Netris-defined and user-defined custom flows. Automatically allowed flows include: + +* SSH from user defined subnets +* NTP from user defined ntp services +* DNS from user defined DNS servers +* Custom user defined rules + +.. csv-table:: Inventory Profile Fields + :file: tables/inventory-profile-fields.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** This example Inventory profile is used to provide NTP and DNS services to the switches (common setup). A custom rule is created to allow UDP connections to the port 161. + +.. image:: images/inventory_profile_custom.png + :align: center + +.. _topology-management-adding-switches: + +Adding Switches +=============== + +Every switch needs to be added to the Netris Controller inventory. You can add new devices with the following process: + +#. Navigate to **Network → Inventory** +#. Click the **Add** button +#. Fill in the fields as described below +#. Click the **Add** button + + .. csv-table:: Add Inventory Fields - Switch + :file: tables/inventory-add-switch.csv + :widths: 25, 75 + :header-rows: 0 + +**Example:** Add a new Switch. + + .. image:: images/add-new-hardware.png + :align: center + +.. note:: Repeat this process to define all your switches. + +.. _topology-management-adding-softgates: + +Adding SoftGates +================ +Every SoftGate node needs to be added to the Netris Controller inventory. To add a SoftGate node: + +#. Navigate to **Network → Topology** +#. Click **Add** +#. Fill in the fields as described below +#. Click the **Add** button + +.. csv-table:: Add Inventory Fields - SoftGate + :file: tables/inventory-add-softgate.csv + :widths: 25, 75 + :header-rows: 0 + +Example: Adding a SoftGate Node to Topology. + +.. image:: images/add-softgate.png + :align: center + +Viewing Inventory +================= + +Inventory Listing provides information about network nodes known to Netris. + +Heartbeat - Indicates whether the node can communicate with Netris Controller. +Health - Indicates the health status of the node. + + .. image:: images/inventory-listing.png + :align: center + +.. note:: You can also add new devices in the Topology view. + +================ +Topology Manager +================ + +The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents software will automatically configure the underlying network devices according to this topology and will watch against potential failures. + +Adding Links +============ + +To define the links in the network: + +#. Right-click on the spine switch +#. Click **Create Link** +#. Select the **From Port** and the **To Port** + +See the example below: + +.. image:: images/create_link.png + :align: center + +.. image:: images/topology_2.png + :align: center + +Once the links have been defined, the network is automatically configured as long as physical connectivity is in place and Netris Agents can communicate with Netris Controller. + +.. tip:: You can drag/move the units to your desired positions and click “Save positions”. diff --git a/en/latest/_sources/try-learn/index.rst.txt b/en/latest/_sources/try-learn/index.rst.txt new file mode 100644 index 0000000000..e99ea0512b --- /dev/null +++ b/en/latest/_sources/try-learn/index.rst.txt @@ -0,0 +1,10 @@ +================== +Netris Try & Learn +================== + +.. toctree:: + :maxdepth: 2 + + netris-cloudsim + using-netris-cloudsim + nvidia-spectrum-x-scenario diff --git a/en/latest/_sources/try-learn/netris-cloudsim.rst.txt b/en/latest/_sources/try-learn/netris-cloudsim.rst.txt new file mode 100644 index 0000000000..848fd7577e --- /dev/null +++ b/en/latest/_sources/try-learn/netris-cloudsim.rst.txt @@ -0,0 +1,8 @@ +============================================= +Netris Infrastructure Simulation Platform +============================================= + +Netris infrastructure simulation platform was initially created by and for the Netris R&D team for internal use. Netris Infrastructure Simulation Platform is also available to customers and partners for testing and educational purposes per request. Each request is subject to approval by the Netris team. + +Please use the below link to submit a Netris trial request. +https://www.netris.io/try/ diff --git a/en/latest/_sources/try-learn/nvidia-spectrum-x-scenario.rst.txt b/en/latest/_sources/try-learn/nvidia-spectrum-x-scenario.rst.txt new file mode 100644 index 0000000000..16c7fb13d2 --- /dev/null +++ b/en/latest/_sources/try-learn/nvidia-spectrum-x-scenario.rst.txt @@ -0,0 +1,309 @@ +====================================================================== +Lab Scenario: GPU-as-a-Service network with NVIDIA Spectrum-X architecture +====================================================================== + +Initialize the Netris controller +================================ + +Start with a blank Netris Controller. SSH to the Netris controller server and ``cd /home/ubuntu/netris-init/netris-spectrum-x-init``. + +Optionally Edit ``terraform.tfvars`` file to set cluster scale parameters. + +Below, we describe the role of a few parameters that directly define the scale. The description of the rest of the parameters is available in the ``terraform.tfvars`` file itself. For the purpose of this try & learn scenario, there is no need to change the other parameters. + +The **east-west** switch fabric is responsible for high performance data transmission between GPU servers. It rail-optimized design allows to non-blocking max-rate data transmission between any GPUs on the network. You only need to define the number of GPU servers in the ``terraform.tfvars`` file. When you execute the initialization module, it will automatically calculate the proper number of links and will generate the rail-optimized blueprint in the Netris controller according to the NVIDIA Spectrum-X guidelines. + +* Define ``gpu-server-count`` using increments of 32 (1 SU = 32 servers, 2 SUs = 64 servers, etc.) + +The **north-south** switch fabric is responsible for everything else - for connectivity from the outside, to manage the GPU nodes and operate workloads. OOB management switches are responsible for out-of-band management of the network switches and GPU servers. In production, OOB management is also used for PXE booting the GPU servers. In this simulation scenario, GPU servers will be booted by means of the Netris infrastructure simulation platform for your conveninece of teasting and learning. + +* Define ``leaf-count`` - the rule of thumb is that at least 1/4th of the number of SUs - so 4 leaf switches can handle up to 4 SUs +* Define ``oob-leaf-count`` - Should be equal to the number of SUs. +* Define ``spine-count`` - Typically 2, although other values are welcome. + +Save the changes and exit. + +Execute ``tofu-apply`` or ``tofu-destroy`` to insert/clean up relevant declarations into the Netris controller. + +Navigate to the Netris controller in your web browser to see the results. + +Start a simulation +================== + +Check ``Inventory``, ``IPAM``, and ``Topology`` sections under ``Network`` menu in the Netris controller web console. (In the topology section, you may need to select the right site to see a diagram) + +Go back to the SSH session and cd to ``/home/ubuntu/netris-cloudsim`` + +Execute pulumi up or pulumi destroy to start/stop a simulation of what’s described in the Netris Controller. + +Monitoring Dashboard +==================== + +Once simulation creation is done, go back to the Netris web console and wait up to 5 minutes for the infrastructure to come up. You can monitor the status of the network either from the dashboard (click on the Netris icon in the top left corner) or from the Topology section. + +Click on ``Agent Heartbeats`` donut to see its detailed view on the right. Agent Heartbeats section shows whether Netris agent heartbeat is being received from each Netris-managed host. + +Once heartbeats are received, automatic configuration will start, as well as automatic monitoring. Netris automatic monitoring provides information about the health of your network, such as Interface up/down status, BGP status, Topology/Wiring errors, RAM/CPU/PSU/Fan status, and else. + +Click on the ``Managed HW Health`` donut to see monitoring check statuses for each Netris-managed node on the right side. + +Topology +======== + +The group of spine and leaf switches at the top part is the east-west network (backend network). The group of spine and leaf switches in the bottom is the north-south network (Tennant Access Network). GPU servers are located in the middle between two switch fabrics. If you zoom in, you can see that eth ports 1-8 of each GPU server are connected to the east-west fabric through rail-optimized design - that's where high-performance computing traffic runs. Interfaces 9-10 are connected to the leaf switches of north-south fabric, later you will see that interfaces 9 & 10 will be bonded from the GPU server side - that's where production traffic, storage traffic, dataset management, and workload management traffic runs. Finally, interface 11 is connected to the OOB (out of band) management switch. OOB interfaces are used for PXE booting the GPU nodes. (in the current simulation there's no PXE booting - the VMs that simulate GPU servers just come alive from an image) + +``Network->Topology`` The main purpose of the topology section is to define the topology. In this scenario, the topology has been defined by means of the initialization module. However, manual changes can still be done through the web console. + +When deploying with physical hardware, the procedure would be to wire the switches and servers according to the topology diagram in Netris. During that process, the MAC address of each physical switch should be entered into the Netris controller through Topology, by editing every switch node (only for switches) and entering the actual MAC address. These MAC addresses will be required for binding the physical switches to logical switches in the Netris controller. + +When running a simulation, like in this scenario there's no need to enter any MAC addresses. The simulation platform will take care of everything. + +The Topology section also reflects some monitoring information. Links change their color based on link status and utilization. You can zoom in/out and then right-click on any link to check its details to see traffic statistics and any relevant healthcheck info. Switch and SoftGate nodes show numbers on a red/yellow/green background reflecting the quantity of critical/warning/ok checks per each node. + +Once your newly created simulation is converged, you will see only 1 check in critical state on every switch - that's the time synchronization, which takes up to 10 minutes to go green. + +SSH to Switches +=============== + +During regular operation, there's no need to SSH to any switches because configuration and management is fully automated by Netris software. + +There are rare cases when the administrator needs to SSH to a switch. To do so, Find the switch management IP from Topology or Inventory sections in the Netris web console. SSH from the Netris Controller to the management IP of the switch using 'cumulus' username. No password is needed when working with the simulation. + +Example: + + +.. code-block:: shell-session + + ubuntu@test-ctl:~/netris-cloudsim$ ssh cumulus@10.7.0.4 + Debian GNU/Linux 12 + Linux leaf-pod00-su0-r3 6.1.0-cl-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4+cl5.9.1u6 (2024-05-13) x86_64 + Last login: Thu Sep 12 16:21:35 2024 from 10.8.0.2 + cumulus@leaf-pod00-su0-r3:mgmt:~$ + + + +SSH to GPU servers +================== + +GPU servers are connected to the east-west and north-south fabrics. At this point of the lab scenario, we haven't created any VPC/V-Net/VLAN services to instruct the fabric to provide connectivity to any interfaces of any GPU nodes. However, for the purpose of learning and experimenting, the simulation platform has an additional management network that allows you to connect to GPU servers anytime. + +SSH from the Netris controller server to a few GPU servers using 'root' username and IP addresses starting 192.168.16.2. ( 192.168.16.2 is host 0 in SU0, 192.168.16.3 is host 1 in SU0, etc.) + +Example: + +.. code-block:: shell-session + + ubuntu@test-ctl:~$ ssh root@192.168.16.2 + Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64) + + * Documentation: https://help.ubuntu.com + * Management: https://landscape.canonical.com + * Support: https://ubuntu.com/pro + + System information as of Fri Sep 13 01:34:28 UTC 2024 + + System load: 0.13 Processes: 95 + Usage of /: 29.2% of 5.64GB Users logged in: 0 + Memory usage: 22% IPv4 address for ens4: 192.168.16.2 + Swap usage: 0% + + + Expanded Security Maintenance for Applications is not enabled. + + 39 updates can be applied immediately. + 12 of these updates are standard security updates. + To see these additional updates run: apt list --upgradable + + Enable ESM Apps to receive additional future security updates. + See https://ubuntu.com/esm or run: sudo pro status + + New release '24.04.1 LTS' available. + Run 'do-release-upgrade' to upgrade to it. + + + Last login: Thu Sep 12 23:20:36 2024 from 10.8.0.2 + root@hgx-pod00-su0-h00:~# + + +On the GPU host, you'll find `./cluster-ping.sh`, which is a bash script that helps you execute parallel pings across every east-west and north-south interface towards any GPU node. The script knows the IP addressing scheme used in this scenario, and it only needs the SU number and host number. + +In the below example, the host pings itself. So, local interface IPs are responding, while the default gateways are not. If you ping another host, you'll get timeouts on all interfaces. + +.. code-block:: shell-session + + root@hgx-pod00-su0-h00:~# ./cluster-ping.sh 0 0 + Usage: ./cluster-ping.sh + + Ping from hgx-pod00-su0-h00 to SU:0 host:0 + + ------ East-West Fabric ------ + ping rail0 (172.0.0.0) : OK + ping rail1 (172.32.0.0) : OK + ping rail2 (172.64.0.0) : OK + ping rail3 (172.96.0.0) : OK + ping rail4 (172.128.0.0) : OK + ping rail5 (172.160.0.0) : OK + ping rail6 (172.192.0.0) : OK + ping rail7 (172.224.0.0) : OK + + ------ North-South Fabric ------ + ping bond0 (192.168.0.1) : OK + ping default GW (192.168.7.254) : Timeout + + ------ IPMI/BMC ------ + ping eth11 (192.168.8.1) : OK + ping default GW (192.168.15.254) : Timeout + + + root@hgx-pod00-su0-h00:~# + +NHN (Netris host networking plugin) +=================================== + +Netris host networking plugin is an optional plugin that runs on a GPU host for automatic configuration of IP addresses, static routes, and DPU parameters. The plugin does not use any management network and does not carry any sensitive information. It's important for multi-tenant situations because the cloud provider should not have access to the tenant servers -- therefore, any host configuration method shall not use any kind of shared management network. Also, the tenant should not be able to access any sensitive information of the cloud provider or other tenants. Netris host networking plugin addresses both issues. The plugin reads the necessary IP and static route information by leveraging LLDP, topology discovery, and custom TLVs. + +The below example shows how to check if the pluggin is running: + +.. code-block:: shell-session + + root@hgx-pod00-su0-h00:~# systemctl status netris-hnp.service + ● netris-hnp.service - Netris Host Networking Plugin + Loaded: loaded (/etc/systemd/system/netris-hnp.service; enabled; vendor preset: enabled) + Active: active (running) since Thu 2024-09-12 23:01:22 UTC; 21h ago + Main PID: 2906 (netris-hnp) + Tasks: 4 (limit: 1102) + Memory: 7.7M + CPU: 3min 35.913s + CGroup: /system.slice/netris-hnp.service + └─2906 /opt/netris/bin/netris-hnp + +You can also check the running IP addresses and static routes on the GPU server, and if you right-click on the server to switch links in the Network->Topology and check the details, you will see that the actual IP addresses on the GPU servers are aligned with those in the Topology blueprint. + +Server Cluster Template +======================= + +Now, when the switch fabric is in an operational state, the underlay is established, it is the time to start defining cloud networking constructs such as VPCs, Subnets, etc., in order to ask the system to provision network access to certain groups of servers. + +One way to do that would be to navigate to ``Network->VPC``, ``Network->IPAM``, and ``Services->V-Net`` sections and create these objects, list switch ports, and then Netris will implement the necessary confogurations. + +Before that, there is one more important concept that we want you to learn. Server Cluster and Server Cluster Template. +Server Cluster allows the creation of VPC, IPAM, and V-Net objects by listing server names instead of switch ports -- this is critical for cloud providers because cloud users don't want to deal with switch ports. + +In the web console, navigate to ``Services->Server Cluster Template`` - click ``+Add``, give the template some name 'GPU-Cluster-Template' or something, and copy/paste the below in the JSON area. + +The Template is basically telling the system to what server interfaces should be groupped into what V-Nets. Netris will find out the appropriate switch ports by looking up the topology. + +.. code-block:: shell-session + + [ + { + "postfix": "East-West", + "type": "l3vpn", + "vlan": "untagged", + "vlanID": "auto", + "serverNics": [ + "eth1", + "eth2", + "eth3", + "eth4", + "eth5", + "eth6", + "eth7", + "eth8" + ] + }, + { + "postfix": "North-South-in-band-and-storage", + "type": "l2vpn", + "vlan": "untagged", + "vlanID": "auto", + "serverNics": [ + "eth9", + "eth10" + ], + "ipv4Gateway": "192.168.7.254/21" + }, + { + "postfix": "OOB-Management", + "type": "l2vpn", + "vlan": "untagged", + "vlanID": "auto", + "serverNics": [ + "eth11" + ], + "ipv4Gateway": "192.168.15.254/21" + } + ] + +Server Cluster +============== + +Now, navigate to ``Services->Server Cluster`` and click +Add. Give the new cluster some name, set Admin to Admin (this is related to Netris internal permissions of who can edit/delete this cluster), set the site to your site (Datacenter-1 is the default name), set VPC to 'create new', select the Template (you'll see the Template created in the last step), and click +Add server and include first 10 servers (from 0 to 9). Click Add. + +While the Server Cluster is being provisioned, check out what primitive objects have been created in the Netris controller driven by the Server Cluster and Server Cluster Template constructs. Navigate to ``Network->VPC``, and you'll see a newly created VPC. Navigate to ``Network->IPAM``, then open the VPC filter and make it filter the IPAM by your new VPC, you'll see some subnets created and assigned to that new VPC. Navigate to ``Services->V-Net``, and you'll see some 3 V-Nets created, one for the east-west fabric (L3VPN VXLAN), one for north-south (L2VPN VXLAN - this one will have EVPN-MH bonding enabled, you'll see in next steps), and one for OOB. + +Go ahead, and create another Server Cluster, and include the next 10 servers - or any other servers. The system won't let you "double-book" any server in more than one cluster to avoid conflicts. + +Checking the connectivity +========================= + +SSH to GPU server host 0 SU 0. + +.. code-block:: shell-session + + ubuntu@test-ctl:~$ ssh root@192.168.16.2 + Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64) + + +Cluster-ping the neighboring GPU servers. SU0 host 0-9 + +.. code-block:: shell-session + + root@hgx-pod00-su0-h00:~# ./cluster-ping.sh 0 9 + Usage: ./cluster-ping.sh + + Ping from hgx-pod00-su0-h00 to SU:0 host:9 + + ------ East-West Fabric ------ + ping rail0 (172.0.0.18) : OK + ping rail1 (172.32.0.18) : OK + ping rail2 (172.64.0.18) : OK + ping rail3 (172.96.0.18) : OK + ping rail4 (172.128.0.18) : OK + ping rail5 (172.160.0.18) : OK + ping rail6 (172.192.0.18) : OK + ping rail7 (172.224.0.18) : OK + + ------ North-South Fabric ------ + ping bond0 (192.168.0.10) : OK + ping default GW (192.168.7.254) : OK + + ------ IPMI/BMC ------ + ping eth11 (192.168.8.10) : OK + ping default GW (192.168.15.254) : OK + + + root@hgx-pod00-su0-h00:~# + +Since GPU servers from 0 to 9 are in the same cluster, you should be able to cluster-ping all of them. If you try to cluster-ping other nodes, you will get timeouts because they are not in the same Server Cluster - so the Netris-generated configuration of the switches will contain the access within a single VPC using various configurations throughout the network. + +You can SSH to GPU server SU0 host 10, which belongs in the second cluster, and cluster-ping its neighbors. + +.. code-block:: shell-session + + ubuntu@test-ctl:~$ ssh root@192.168.16.12 + Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64) + + +Cleanup the Controller +====================== + +At this point this Netris Try & Learn scenario has been concluded. You may want to clean up the lab to let your colleagues run through the scenario or if you are working on another one. There is no need to clean up if you are about to return the environment to the Netris team -- we are going to recycle and reinstall the environment anyway. + +1. Delete Server Clusters from the ``Services->Server Cluster`` menu. +2. Delete Server Cluster Profile from the ``Services->Server Cluster Profile`` menu. +3. SSH to the Netris controller server, ``cd /home/ubuntu/netris-cloudsim``, and execute ``pulumi destroy`` to destroy the infrastructure simulation. +4. ``cd /home/ubuntu/netris-init/netris-spectrum-x-init`` and execute tofu-destroy to remove the objects from the Netris controller that were created through the initialization module. + +Please let us know your feedback and questions. + diff --git a/en/latest/_sources/try-learn/using-netris-cloudsim.rst.txt b/en/latest/_sources/try-learn/using-netris-cloudsim.rst.txt new file mode 100644 index 0000000000..f9dcd58bff --- /dev/null +++ b/en/latest/_sources/try-learn/using-netris-cloudsim.rst.txt @@ -0,0 +1,60 @@ +================================================== +Netris Test Controller & Infrastructure Simulation +================================================== + +This document provides general tips and tricks for using the Netris test controller and Netris infrastructure simulation. Please refer to individual scenarios in this section to experiment around a specific use case. + +Once your Try & Learn through Netris Infrastructure Simulation platform is approved you will receive credentials for accessing a Netris controller with a trial license hosted in Netris Infrastructure Simulation platform. + +Controller FQDN: example-ctl.netris.dev +Password: NetrisProvidedPassword + + +Web Console +=========== + +Navigate your browser to the Controller FQDN then use +username: netris +password: + +SSH +=== + +Controller initialization modules and simulation control packages are installed on the Netris controller server. + +Connect to the controller server using ssh. Username: ubuntu Password: + +.. code-block:: shell-session + + ssh ubuntu@ + + +Netris Init Modules +=================== + +Netris init modules are designed to generate Inventory, IPAM, and Topology data based on simple arguments. Most init modules are written using Terraform/HCL. + +Netris test environment may come with a module that is relevant to your use case. If you can't find the right module for your use case, please contact your SA (Solutions Architect) + +Init modules are stored in the ``/home/ubuntu/netris-init/`` directory. + +Each module is stored in its own subdirectory. To use a module 'cd' to the appropriate subdirectory, review ``terraform.tfvars`` file, make changes to the arguments if needed, and save. + +Execute ``tofu-apply`` or ``tofu-destroy`` in the init module subdirectory to apply/destroy the Netris controller configuration. + +Start/Stop a Simulation +======================= + +Once the topology blueprint, Inventory, and IPAM data are created in Netris -- basically, Netris controller is describing an infrastructure - you can start its simulation. + +Go to: + +.. code-block:: shell-session + + cd /home/ubuntu/netris-cloudsim + +Execute ``pulumi up`` or ``pulumi destroy`` to start/stop a simulation of what's described in the Netris Controller + + + + diff --git a/en/latest/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt b/en/latest/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..5d87c5be05 --- /dev/null +++ b/en/latest/_sources/tutorials/activating-bgp-on-equinix-metal-project.rst.txt @@ -0,0 +1,27 @@ +####################################### +Activating BGP on Equinix Metal Project +####################################### + +Why use BGP with Equinix Metal? +SoftGate nodes are like border routers to your VPC, they are routing traffic between hosts inside your project and the Internet. We are going to establish 2 BGP sessions between SoftGate nodes and Equinix Metal. So there will be 4 BGP sessions total. + +We need these BGP sessions for moving further. In the next chapters we are going to request pools of public IP addresses, that Netris will automatically advertise to Equinix Metal, so inbound traffic “knows” how to reach Load Balancer, NAT, and other services that you will use within your VPC. + +.. image:: /tutorials/images/equinix-metal-bgp-diagram.png + :align: center + +You only need to activate BGP on the Equinix Metal Project. Netris will handle the rest. +In the Equinix Metal web console go to Networking → BGP then click Activate BGP on This Project. (see below screenshots) + +.. image:: /tutorials/images/equinix-metal-activate-bgp.png + :align: center + +Netris will handle the rest behind the scenes automatically. Netris will enable BGP peering on the Equinix Metal side, Netris will pull the metadata with the BGP info, and will automatically configure FRR (Free Range Routing BGP daemon) on both SoftGate nodes to bring up the BGP sessions up. + +After a few minutes you should see 4 new BGP sessions in your Netris web console under Net → E-BGP. (example screenshot below). + +.. image:: /tutorials/images/equinix-metal-netris-bgp-up.png + :align: center + + +Now your Netris VPC has established BGP sessions with Equinix Metal Project, and you can proceed to the next step. diff --git a/en/latest/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt b/en/latest/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt new file mode 100644 index 0000000000..17ae00d1c8 --- /dev/null +++ b/en/latest/_sources/tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst.txt @@ -0,0 +1,36 @@ +########################################################### +Provisioning Netris SoftGate nodes in Equinix Metal Project +########################################################### + +For SoftGate nodes you can start with two c3.small.x86 or larger servers. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one. + +Request two servers(c3.small.x86) from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned. + +1) At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory + +.. image:: /tutorials/images/softgate-nodes-created-in-equinix.png + :align: center + +2) When Equinix finishes provisioning of the servers, click on each server name, then click "Tags", and add a tag “netris-softgate”. + + Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location). + +Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + +.. image:: /tutorials/images/softgate-nodes-recognized-in-netris.png + :align: center + +3) Provision SoftGate nodes. + +Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there. + +.. image:: /tutorials/images/softgate-one-liner-provisioning.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/softgate-green.png + :align: center diff --git a/en/latest/_sources/tutorials/aws-concept.rst.txt b/en/latest/_sources/tutorials/aws-concept.rst.txt new file mode 100644 index 0000000000..97af34ccc8 --- /dev/null +++ b/en/latest/_sources/tutorials/aws-concept.rst.txt @@ -0,0 +1,19 @@ +########################### +Site Mesh with AWS Overview +########################### + +Introduction +------------- + +This guide provides a step-by-step process to set up and configure Netris Softgate in AWS for establishing a site mesh network between the user's on-premises, AWS, and other cloud environments. + + +Concept +-------- + +Netris Softgate in AWS is an EC2 instance that runs the Netris software. Therefore, you'll first need to create an EC2 instance for Netris Softgate and install the Netris software on it. Once that's done, you'll need to configure the routes in your AWS VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your AWS VPC to access those destinations through the Netris Softgate EC2 instance. + +.. image:: images/aws-concept-traffic-flows.png + :align: center + +Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way. diff --git a/en/latest/_sources/tutorials/aws-deploy-softgate.rst.txt b/en/latest/_sources/tutorials/aws-deploy-softgate.rst.txt new file mode 100644 index 0000000000..12dbfc4710 --- /dev/null +++ b/en/latest/_sources/tutorials/aws-deploy-softgate.rst.txt @@ -0,0 +1,187 @@ +.. meta:: + :description: Deploy a Softgate in AWS + +######################## +Deploy a Softgate in AWS +######################## + +As stated in the previous section, the following sequence of actions must be taken in order to proceed: create an EC2 instance, add Softgate into the Netris Controller, install Netris Softgate software on the EC2 instance, and configure routes in AWS VPC. Let us commence with these steps in the specified order. + +Create an EC2 instance +====================== + +Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated EC2. To achieve this, create a security group with the "All traffic" type and "Anywhere" source for both inbound and outbound rules. Afterward, an EC2 instance can be created using the security group above. + +.. image:: images/aws-security-group.png + :align: center + +To enable connectivity with other Netris sites, it is essential to create the EC2 instance in the desired VPC. Therefore, deploy a new EC2 instance with the Ubuntu 22.04 operating system installed, utilizing an instance type that meets the minimum hardware requirements of 2 virtual CPUs and 4 GB of RAM, such as t2.medium/t3.medium or any other type that satisfies these specifications. It is also recommended to allocate at least 30 GB of drive space. + + +.. image:: images/aws-softgate-deployed.png + :align: center + +After successfully deploying the EC2 instance, it is crucial to take note of its Public IPv4 address. This address will be required in the upcoming step. + + +Configure Netris Controller +=========================== + +Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed. + +Pre-Requisite Steps +------------------- + +In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below: + +1. Open the Netris Web Console. +2. Navigate to "Net" and select "Sites". +3. Click on the "+ Add" button. +4. Select "Dot1q Trunk" as the "Switch Fabric". +5. Input a descriptive name for the site. +6. Specify ``65500`` in the "Public ASN" field. +7. Click "Add" to create the new site. + +.. image:: images/aws-netris-site-create.png + :align: center + +Subsequently, it is necessary to create a subnet with the "Purpose" of loopback within Netris IPAM. The subnet's "Prefix" will match the External IP of the AWS EC2 instance, with a netmask length of /32. For instance, if the EC2 instance's External IP address is 54.176.11.144, then the "Prefix" for the loopback subnet will be 54.176.11.144/32. It's important to note that an allocation for this "Prefix" must be created prior to creating the loopback subnet. To achieve this, follow the steps below: + +1. Go to the "IPAM" section under the "Net" tab. +2. Click on the "+ Add" button located at the top-right corner. +3. Enter the "Prefix" for the new allocation, such as ``54.176.11.144/32``. +4. Type a descriptive "Name" for the allocation. +5. Select the desired tenant name from the "Tenant" dropdown menu. +6. Click on the "Add" button to create the allocation. + +Once the allocation is created, proceed with the creation of the subnet as follows: + +1. Click on the "+ Add" button located at the top-right corner. +2. Enter the "Prefix" for the new subnet, such as ``54.176.11.144/32``. +3. Type a descriptive "Name" for the subnet. +4. Select the desired tenant name from the "Tenant" dropdown menu. +5. From the "Type" dropdown menu, select "Subnet". +6. Select "loopback" from the "Purpose" dropdown menu. +7. Choose the appropriate site from the "Sites" dropdown menu. +8. Click on the "Add" button to create the subnet. + + +.. image:: images/aws-netris-ipam-lo.png + :align: center + + +Add AWS VPC Subnet into the Netris Controller +--------------------------------------------- + +To register your AWS VPC's entire CIDR block into Netris IPAM, follow these steps: + +1. From the AWS Console, navigate to your VPC and take note of your CIDR blocks. +2. In Netris Controller, go to the "IPAM" section under the "Net" tab. +3. Click the "+ Add" button located at the top-right corner. +4. Enter the VPC CIDR block into the "Prefix" field for the new subnet. For example, if your CIDR block is "172.31.0.0/16", enter that value. +5. Type a descriptive name for the subnet. +6. From the "Tenant" dropdown menu, select the desired tenant name. +7. From the "Type" dropdown menu, select "Subnet". +8. Select "inactive" from the "Purpose" dropdown menu. +9. Choose the appropriate site from the "Sites" dropdown menu. +10. Click the "Add" button. + +.. image:: images/aws-vpc-cidr-to-netris.png + :align: center + + +Static route for AWS VPC Subnet in Netris Controller +---------------------------------------------------- + +We need to create route entry in Netris. The prefix for the route will be the AWS VPC CIDR block, and the next-hop will be the default gateway of Netris Softgate EC2. + +Here are the steps to create the static route: + +1. Securely log in to the Netris Softgate's EC2 instance using SSH. +2. Retrieve the default gateway address by typing the command ``ip route show default``. +3. In Netris Controller, go to the "Routes" section under the "Net" tab. +4. Click on the "+ Add" button located at the top-right corner of the screen. +5. Enter your VPC CIDR block in the "Prefix" field. +6. Enter the default gateway of the Netris Softgate EC2 instance in the "Next-Hop" field. +7. Select the appropriate site from the "Site" dropdown menu. +8. Finally, click on the "Add" button to create the static route. + + +.. image:: images/aws-netris-static-route.png + :align: center + + +Create the Softgate in the Netris Controller +-------------------------------------------- + +After completing all the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide: + +1. Ensure that you have completed all the pre-required steps. +2. Navigate to the "Net" tab in the Netris Controller and select the "Inventory" section. +3. Click on the "+Add" button to create a new Softgate. +4. Provide a descriptive name for the Softgate in the "Name" field. +5. From the "Tenant" dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets. +6. From the "Type" dropdown menu, select "SoftGate". +7. Choose the appropriate site from the "Site" dropdown menu. +8. For the "Main IP address" field, select "Assign automatically". +9. For the "Management IP Address" field, select "None". +10. In the "Description" field, add ``int=eth0`` to specify that Netris should use the softgate's eth0 interface instead of the default bond0 interface that Netris Softgate Agent looks for. +11. Finally, click on the "Add" button to create the Softgate. + +.. image:: images/aws-netris-create-sg.png + :align: center + +Netris SoftGate node provisioning +--------------------------------- + +After creating a softgate resource in Netris Controller, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting "Install Agent". Copy the one-line installer command to your clipboard and connect to your EC2 instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it's done, reboot the server. + +.. image:: images/aws-netris-provision-sg.png + :align: center + + +Configure AWS VPC +================= + +Adding routes +------------- + +To enable specific traffic to be routed to the Netris Softgate EC2 instance in your AWS VPC, you need to modify your VPC's Route Table. To get started, go to the AWS Console and navigate to your VPC's Route Table. From there, click the "Edit routes" button to access the routing table, and then click "Add route" to create a new routing entry. + +In the "Destination" field, enter the subnet CIDR block for the Netris other Sites' subnets you want to access from this VPC. Next, in the "Target" field, select the "Instance" option and then select the Netris Softgate EC2 instance you previously created. This will ensure that traffic for those subnets is directed to the Softgate instance. + + +.. image:: images/aws-vpc-routes-created.png + :align: center + +EC2 Source / destination check +------------------------------ + +To allow Netris SoftGate to work properly, it is necessary to disable the 'Source / destination check' for the SoftGate's EC2 Instance. To do so, follow the steps below: + +1. Go to the AWS Console and navigate to the EC2 service page. +2. Select the SoftGate EC2 instance. +3. Click on the "Actions" button in the "Networking" section. +4. Select "Change Source / destination check". +5. Mark the "Stop" checkbox to disable the feature. +6. Click "Save" to apply the changes. + +.. image:: images/aws-ec2-stop-fwd-check.png + :align: center + + +By disabling the "Source / destination check", the SoftGate EC2 instance can receive and forward traffic between the Netris other Sites and AWS VPC subnets. + + +Enabling Site Mesh +================== + +To enable Site Mesh, in Netris Controller, navigate to the "Net" tab and select "Sites". Next, click on the three vertical dots (⋮) on the right side of the AWS site and select "Edit", and then from the "Site Mesh" dropdown menu, select "Hub". Save the changes. Repeat these steps for all sites that need to have meshed. + +.. image:: images/aws-netris-enable-site-mesh.png + :align: center + +The Site Mesh status can be viewed by navigating to the "Site Mesh" section under the "Net" tab. This will display the current status of Site Mesh for all Sites. + +.. image:: images/aws-netris-site-mesh-status.png + :align: center diff --git a/en/latest/_sources/tutorials/connecting-fabric-to-isp.rst.txt b/en/latest/_sources/tutorials/connecting-fabric-to-isp.rst.txt new file mode 100644 index 0000000000..38318d6463 --- /dev/null +++ b/en/latest/_sources/tutorials/connecting-fabric-to-isp.rst.txt @@ -0,0 +1,17 @@ +########################################## +Connecting Netris managed fabric to an ISP +########################################## + +BGP is used to connect the Netris managed fabric to an ISP for internet access. To do this, connect a cable from the ISP to your switch port. Then, use the information provided by your ISP to configure a BGP session within the Netris Controller. + +To create a BGP session go to Network → E-BGP → +Add + +.. image:: images/create_bgp.png + :align: center + +If everything is correct, State, port and BGP will get green status. + +.. image:: images/bgp_status.png + :align: center + +Check out advanced BGP configuration here, if you require additional features. diff --git a/en/latest/_sources/tutorials/connecting-servers-fabric.rst.txt b/en/latest/_sources/tutorials/connecting-servers-fabric.rst.txt new file mode 100644 index 0000000000..cd12439cef --- /dev/null +++ b/en/latest/_sources/tutorials/connecting-servers-fabric.rst.txt @@ -0,0 +1,11 @@ +=============================================== +Connecting servers to the Netris managed fabric +=============================================== + +To connect servers or other endpoints to the switch fabric, you should use V-Nets. V-Net is a virtual networking service that provides Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports throughout the switch fabric. Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and manages the high availability for the default gateway behind the scenes. Simply add the necessary switch ports to a V-Net, and Netris will handle the rest. +Navigate to Services → V-Net → +Add + +.. image:: images/vnet.png + :align: center + +If you are planning to use Link Aggregation in your setup, please take a look at the options provided by Netris :doc:`here`. diff --git a/en/latest/_sources/tutorials/create-interconnection-to-fabric.rst.txt b/en/latest/_sources/tutorials/create-interconnection-to-fabric.rst.txt new file mode 100644 index 0000000000..8b5bafdb69 --- /dev/null +++ b/en/latest/_sources/tutorials/create-interconnection-to-fabric.rst.txt @@ -0,0 +1,122 @@ +.. meta:: + :description: Creating an Interconnection to Equinix Fabric + +============================================= +Creating an Interconnection to Equinix Fabric +============================================= + +Once you have successfully set up Netris on Equinix Metal, you may want to use it to connect your private networks from Equinix Fabric back to Metal. + +Netris can easily manage the routing and other required VPC services for this. + + +**Create VLANs in Metal** + +To be able to create an interconnect into Equinix Fabric, first you must create a pair of VLANs for the Primary and Secondary connections. + +To do this, go to Equinix Console -> IP&Networks -> Layer 2. +Click "Add Vlan". +Choose the location of your Netris Softgates, then use a couple of VLAN IDs that are unused. + +In the example below, VLAN 1000 is used for Primary, and VLAN 1001 is used for the Secondary. + +.. image:: images/add-new-vlan-equinix.png + :align: center + + + +**Request Interconnection** + +Now we are ready to request the Equinix Interconnection on the Equinix Metal side. + +Go to Equinix Console -> IP&Networks -> Interconnections. + +Click "Request New Interconnection". + +From here, under Fabric VC select "Fabric Billed". + +Choose the Location where your Netris Softgates are located. + +Add an Interconnection name. This can be any name. + +Choose "Redundant interconnection". + +Under "Primary Metal VLAN" select the VLAN created in the previous step, in this example it is VLAN 1000. + +Under "Secondary Metal VLAN" select the VLAN created in the previous step, in this example it is VLAN 1001. + +Then, finalize the request by clicking "Submit Interconnection Request". + +.. image:: images/add-interconnection-request.png + :align: center + + + + +Once this is accomplished, use the token generated to connect your virtual Network Edge device to Equinix Metal. Setting up a connection on the Fabric side is outside the scope of this document. Once that is complete, the Interconnection will show as Active, and you may move onto creating BGP connections with the Network Edge Virtual Device. + + + +**Primary BGP Connection to Fabric Network Edge Device** + +Up to this point, a redundant pair of layer 2 connections have been created and connected back to an Equinix Fabric Network Edge device (or some other port with a router on the other end). To be able to utilize the redundancy of both connections, a BGP peering must be established over the two layer 2 VLANs. + +On the Netris Controller, go to Net -> E-BGP, then click Add. + +Fill in the following information: + +.. code-block:: shell-session + + + Name: Choose any name + Site: Choose the site of your Equinix Project + V-Net: None + BGP Router: Choose the primary Softgate + VLAN ID: Choose the Primary VLAN ID created when creating the Interconnection in the previous steps + Neighbor AS: Choose the ASN configured on the Network Edge router + IP Family: IPv4 + Local IP: Any /30 can be used. Using a Private IP is recommended. + Remote IP: Use the IP of the remote side. + + +.. image:: images/add-equinix-bgp-primary.png + :align: center + + + + +**Secondary BGP Connection to Fabric Network Edge Device** + +Now to create the backup/secondary connection. + + +On the Netris Controller, go to Net -> E-BGP, then click Add. + +Fill in the following information: + +.. code-block:: shell-session + + + Name: Choose any name + Site: Choose the site of your Equinix Project + V-Net: None + BGP Router: Choose the secondary Softgate + VLAN ID: Choose the Secondary VLAN ID created when creating the Interconnection in the previous steps + Neighbor AS: Choose the ASN configured on the Network Edge router + IP Family: IPv4 + Local IP: Any /30 can be used. Using a Private IP is recommended. + Remote IP: Use the IP of the remote side. + + + +Once completed, the EBGP status shoudl look something like this: + +.. image:: images/equinix-ebgp-links-up.png + :align: center + + + + +.. note:: + At this point, you should have both BGP sessions up, but the link may show yellow if the Network Edge device is not advertising any routes back to the Netris Softgates. Once the Network Edge router begins sending routes from other connections, the status should turn green. + diff --git a/en/latest/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt b/en/latest/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..bd09968111 --- /dev/null +++ b/en/latest/_sources/tutorials/enable-services-on-equinix-metal-project.rst.txt @@ -0,0 +1,68 @@ +####################################################### +Enabling services (NAT, V-Net, Load Balancer, IP pools) +####################################################### + +Although bare metal servers in Equinix Metal Project get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities. + +Both NAT and on-demand Load Balancer services need public IP addresses. + + +1) Requesting new Public IP address block +========================================= + +Go to Equinix Metal web console and click on Networking → IPs (see the screenshot below) + +In this example, I’m requesting two IP address blocks, one /30 (4 IPs) for NAT and one /28 (16 IPs) for Load Balancer. + +It’s important to tag IP blocks as “netris”. This is a signal for Netris Controller that this block is intended for Netris VPC. + +You can always request more IP address blocks in the future. Also it is possible to request a large block and then use Netris IPAM for crushing it into smaller blocks. You can read more about Netris IPAM in Netris docs. + +.. image:: /tutorials/images/equinix-metal-request-ip-block.png + :align: center + +Once IP address blocks are provisioned on Equinix Metal Project you should be able to find them automatically replicated in Netris web console under Net → IPAM + +.. image:: /tutorials/images/equinix-metal-netris-ipam-synced.png + :align: center + +You don’t need to worry about advertising them over BGP, Netris will handle that automatically when that makes sense (associated with any service). + + +2) Enable on-demand (elastic) Load Balancer +=========================================== + +To Enable on-demand (elastic) Load Balancer you only need to change the “purpose” field of appropriate IP address block from “common” into “load-balancer” + +Click on the 3 dots menu (in this example of /28 IP address block), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field. + +.. image:: /tutorials/images/netris-enable-elb.png + :align: center + +Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform. + +Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it. + + +3) Enable V-Net +=============== + +V-Net is a service for virtual private networks. You need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources. + +.. image:: /tutorials/images/netris-create-common-subnets.png + :align: center + +4) Enable NAT +============= + +To enable NAT, you need to repurpose a block of IP addresses for NAT. In the below example I’m repurposing the newly requested /30 subnet for NAT. + +.. image:: /tutorials/images/netris-ipam-nat.png + :align: center + +Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. In this example I’m enabling SNAT for the entire 10.0.0.0/8 private network, so basically I just want to ensure that VMs that will get IPs from private networks will get outbound Internet access through NAT. You can always have more granular control either through NAT rule or using Services → ACLs. + +.. image:: /tutorials/images/netris-create-nat-rule.png + :align: center + +At this point the minimal configuration of Netris VPC networking is done, next chapters will describe how to consume the VPC, how to request resources and services. diff --git a/en/latest/_sources/tutorials/enabling-load-balancing-services.rst.txt b/en/latest/_sources/tutorials/enabling-load-balancing-services.rst.txt new file mode 100644 index 0000000000..8742d14791 --- /dev/null +++ b/en/latest/_sources/tutorials/enabling-load-balancing-services.rst.txt @@ -0,0 +1,16 @@ +################################ +Enabling Load-balancing services +################################ + +L4 Load Balancer is an on-demand (elastic) L4 Load Balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +:doc:`Kubernetes integration` + +:doc:`Terraform integration` + +Below is a screenshot of requesting a Load Balancer service through the Netris web console. Navigate to Services → L4 Load Balancer → +Add + +.. image:: images/l4lb_create.png + :align: center diff --git a/en/latest/_sources/tutorials/enabling-nat-services.rst.txt b/en/latest/_sources/tutorials/enabling-nat-services.rst.txt new file mode 100644 index 0000000000..457b667fbf --- /dev/null +++ b/en/latest/_sources/tutorials/enabling-nat-services.rst.txt @@ -0,0 +1,23 @@ +##################### +Enabling NAT services +##################### + +If you utilize private address space for your hosts, you may need a NAT service to enable internet access. +Netris Softgates support SNAT, DNAT and Masquerade features. + +.. note:: + Softgate PRO will support Masquerade in the future releases. + +Navigate to Network → NAT → +Add + +Create a SNAT service to allow connections from your hosts to the Internet. + +.. image:: images/snat_add.png + :align: center + +Selecting a SNAT pool will allocate the entire pool for this service, preventing the use of IP addresses from the pool for DNAT or other SNAT purposes. + +Create a DNAT service to allow connections from the Internet to your internal hosts with private IP. + +.. image:: images/dnat_add.png + :align: center diff --git a/en/latest/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt b/en/latest/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt new file mode 100644 index 0000000000..4493864667 --- /dev/null +++ b/en/latest/_sources/tutorials/equinix-metal-api-integration-enablement.rst.txt @@ -0,0 +1,51 @@ +.. meta:: + :description: Getting Started for Equinix Metal + +#################################### +Enable Equinix Metal API integration +#################################### + + +For each Equinix Metal Project+location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "Equinix Metal" from the dropdown menu. + * - Name + - Type a descriptive name for your Equinix Metal Project+location. + * - Equinix Project ID + - Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID. + * - Equinix Project API key + - Create a new Read/Write API key in Equinix Metal portal under Project Settings → API keys → + Add an API Key. Then copy/paste here. + * - Equinix Location + - Select your equinix location from the dropdown menu. + + +Equinix Metal Project ID + +.. image:: /tutorials/images/equinix-metal-project-id.png + :align: center + + +Equinix Metal Project API key + +.. image:: /tutorials/images/equinix-metal-project-api-keys.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/netris-create-equinix-metal-site.png + :align: center + + diff --git a/en/latest/_sources/tutorials/gcp-concept.rst.txt b/en/latest/_sources/tutorials/gcp-concept.rst.txt new file mode 100644 index 0000000000..c3f4a1e315 --- /dev/null +++ b/en/latest/_sources/tutorials/gcp-concept.rst.txt @@ -0,0 +1,19 @@ +########################### +Site Mesh with GCP Overview +########################### + +Introduction +------------- + +This guide provides a step-by-step process to set up and configure Netris Softgate in GCP for establishing a site mesh network between the user's on-premises, GCP, and other cloud environments. + + +Concept +-------- + +Netris Softgate in GCP is an VM instance that runs the Netris software. Therefore, you'll first need to create an VM instance for Netris Softgate and install the Netris software on it. Once that's done, you'll need to configure the routes in your GCP VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your GCP VPC to access those destinations through the Netris Softgate VM instance. + +.. image:: images/gcp-concept-traffic-flows.png + :align: center + +Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way. diff --git a/en/latest/_sources/tutorials/gcp-deploy-softgate.rst.txt b/en/latest/_sources/tutorials/gcp-deploy-softgate.rst.txt new file mode 100644 index 0000000000..2a6ec89e48 --- /dev/null +++ b/en/latest/_sources/tutorials/gcp-deploy-softgate.rst.txt @@ -0,0 +1,199 @@ +.. meta:: + :description: Deploy a Softgate in GCP + +######################## +Deploy a Softgate in GCP +######################## + +As stated in the previous section, the following sequence of actions must be taken in order to proceed: create a VM instance, add Softgate into the Netris Controller, install Netris Softgate software on the VM instance, and configure routes in GCP VPC. Let's commence with these steps in the specified order. + +Create a VM instance +==================== + +Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated VM instance. To achieve this, create a VPC firewall rule with the following parameters: + +* Name: Any name +* Direction of traffic: "Ingress" +* Action on match: "Allow" +* Targets: "Specified target tags" +* Target tags: ``allow-all`` +* Source filter: "IPv4 ranges" +* Source IPv4 ranges: ``0.0.0.0/0`` +* Protocols and ports: "Allow all" + +Afterward, a VM instance can be created using the network tag above. + +.. image:: images/gcp-firewall-rule.png + :align: center + +To enable connectivity with other Netris sites, creating the VM instance in the desired VPC is essential. Therefore, deploy a new VM instance with the **Ubuntu 22.04** operating system installed, utilizing an instance type that meets the minimum hardware requirements of **2** virtual CPUs, **4 GB** of RAM, and **30 GB** of drive space, such as e2-medium or any other type that satisfies these specifications. From the Advanced Networking options enable ``IP forwarding`` and specify the previously created ``allow-all`` network tag. + +.. image:: images/gcp-softgate-deployed.png + :align: center + +After successfully deploying the VM instance, it is crucial to take note of its External IP. This address will be required in the upcoming step. + + +Configure Netris Controller +=========================== + +Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed. + +Pre-Requisite Steps +------------------- + +In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below: + +1. Open the Netris Web Console. +2. Navigate to "Net" and select "Sites". +3. Click on the "+ Add" button. +4. Select "Dot1q Trunk" as the "Switch Fabric". +5. Input a descriptive name for the site. +6. Specify ``65500`` in the "Public ASN" field. +7. Click "Add" to create the new site. + +.. image:: images/gcp-netris-site-create.png + :align: center + +Subsequently, it is necessary to create a subnet with the "Purpose" of loopback within Netris IPAM. The subnet's "Prefix" will match the External IP of the GCP VM instance, with a netmask length of /32. For instance, if the VM instance's External IP address is 34.85.167.128, then the "Prefix" for the loopback subnet will be 34.85.167.128/32. It's important to note that an allocation for this "Prefix" must be created prior to creating the loopback subnet. To achieve this, follow the steps below: + +1. Go to the "IPAM" section under the "Net" tab. +2. Click on the "+ Add" button located at the top-right corner. +3. Enter the "Prefix" for the new allocation, such as ``34.85.167.128/32``. +4. Type a descriptive "Name" for the allocation. +5. Select the desired tenant name from the "Tenant" dropdown menu. +6. Click on the "Add" button to create the allocation. + +Once the allocation is created, proceed with the creation of the subnet as follows: + +1. Click on the "+ Add" button located at the top-right corner. +2. Enter the "Prefix" for the new subnet, such as ``34.85.167.128/32``. +3. Type a descriptive "Name" for the subnet. +4. Select the desired tenant name from the "Tenant" dropdown menu. +5. From the "Type" dropdown menu, select "Subnet". +6. Select "loopback" from the "Purpose" dropdown menu. +7. Choose the appropriate site from the "Sites" dropdown menu. +8. Click on the "Add" button to create the subnet. + +.. image:: images/gcp-netris-ipam-lo.png + :align: center + +Create the Softgate in the Netris Controller +-------------------------------------------- + +After completing the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide: + +1. Ensure that you have completed all the pre-required steps. +2. Navigate to the "Net" tab in the Netris Controller and select the "Inventory" section. +3. Click on the "+Add" button to create a new Softgate. +4. Provide a descriptive name for the Softgate in the "Name" field. +5. From the "Tenant" dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets. +6. From the "Type" dropdown menu, select "SoftGate". +7. Choose the appropriate site from the "Site" dropdown menu. +8. For the "Main IP address" field select "Assign automatically". +9. For the "Management IP Address" field select "None". +10. In the "Description" field, add ``int=ens4`` to specify that Netris should use the softgate's ens4 interface instead of the default bond0 interface that Netris Softgate Agent looks for. +11. Finally, click on the "Add" button to create the Softgate. + +.. image:: images/gcp-netris-create-sg.png + :align: center + + +Add GCP VPC Subnet(s) into the Netris Controller +------------------------------------------------ + +The next step involves registering into Netris IPAM the subnet of the region where the SoftGate node has been created. Follow the steps below to accomplish this: + +1. Access the GCP Console and navigate to your VPC Subnets. +2. Take note of the Internal IP range associated with the relevant region. +3. In Netris Controller, go to the "IPAM" section under the "Net" tab. +4. Click the "+ Add" button located at the top-right corner. +5. Enter the region's Internal IP range into the "Prefix" field for the new subnet. For example, if your Internal IP range is "10.150.0.0/20", enter that value. +6. Type a descriptive name for the subnet. +7. From the "Tenant" dropdown menu, select the desired tenant name. +8. From the "Type" dropdown menu, select "Subnet". +9. Select "inactive" from the "Purpose" dropdown menu. +10. Choose the appropriate site from the "Sites" dropdown menu. +11. Click the "Add" button. + + +Once the subnet of the region where the SoftGate node was created has been successfully registered in Netris IPAM, you can proceed to create subnets for other necessary regions in a similar manner. + +.. image:: images/gcp-vpc-subnet-to-netris.png + :align: center + + +Static route for GCP VPC Subnet(s) in Netris Controller +------------------------------------------------------- + +We need to create route entries in Netris. The prefix for the route will be the GCP VPC Subnet(s), and the next-hop will be the default gateway of Netris Softgate VM Instance. + +Here are the steps to create the static route: + +1. Securely log in to the Netris Softgate's VM instance using SSH. +2. Retrieve the default gateway address by typing the command ``ip route show default``. +3. In Netris Controller, go to the "Routes" section under the "Net" tab. +4. Click on the "+ Add" button located at the top-right corner of the screen. +5. Enter the subnet of the GCP region in the "Prefix" field. +6. Enter the default gateway of the Netris Softgate VM instance in the "Next-Hop" field. +7. Choose the appropriate site from the "Site" dropdown menu. +8. From the "Apply to" dropdown menu, select the SoftGate. +9. Finally, click on the "Add" button to create the static route. + +Repeat these steps for each GCP VPC Subnet that has been registered in Netris IPAM. + +*Note: Regardless of the GCP region's subnet, the "Next-Hop" field should always contain the default gateway of the Netris SoftGate VM instance.* + +.. image:: images/gcp-netris-static-route.png + :align: center + + +Netris SoftGate node provisioning +--------------------------------- + +After creating a softgate resource in Netris Controller and defining all necessary routes, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting "Install Agent". Copy the one-line installer command to your clipboard and connect to your VM instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it's done, reboot the server. + +.. image:: images/gcp-netris-provision-sg.png + :align: center + + +Configure routing in GCP VPC +============================ + +To ensure that specific traffic is directed to the Netris Softgate VM instance within your GCP VPC, it is necessary to configure new route entries in VPC routes. Follow the step-by-step guide below to add GCP VPC routes: + +1. Access the Google Cloud Platform (GCP) Console. +2. Navigate to the VPC Network page. +3. Click on the "Routes" then switch to the "ROUTE MANAGEMENT" tab to view the existing routes. +4. Click the "+ Create Route" button to create a new route. +5. Provide the following details for the new route: + + * Name: Assign a descriptive name to the route. + * Network: Select the appropriate network for the route. + * Destination IP Range: Specify the subnets of other Netris sites that you wish to access from this VPC. + * Next hop: From the dropdown menu, select the "Specify an instance" + * Next hop instance: From the dropdown menu, select the SoftGate VM Instance + +6. Review the route configuration and ensure all the details are accurate. +7. Click the "Create" button to add the route to the GCP VPC network. +8. Verify that the new route appears in the list of routes for the selected VPC network. + +Repeat these steps for any additional routes you need to add. + + +.. image:: images/gcp-vpc-routes-created.png + :align: center + + +Enabling Site Mesh +================== + +To enable Site Mesh, in Netris Controller, navigate to the "Net" tab and select "Sites". Next, click on the three vertical dots (⋮) on the right side of the GCP site and select "Edit", and then from the "Site Mesh" dropdown menu, select "Hub". Save the changes. Repeat these steps for all sites that need to have meshed. + +.. image:: images/gcp-netris-enable-site-mesh.png + :align: center + +The Site Mesh status can be viewed by navigating to the "Site Mesh" section under the "Net" tab. This will display the current status of Site Mesh for all Sites. + +.. image:: images/gcp-netris-site-mesh-status.png + :align: center diff --git a/en/latest/_sources/tutorials/getting-started-for-equinix-metal.rst.txt b/en/latest/_sources/tutorials/getting-started-for-equinix-metal.rst.txt new file mode 100644 index 0000000000..14d5f88ed9 --- /dev/null +++ b/en/latest/_sources/tutorials/getting-started-for-equinix-metal.rst.txt @@ -0,0 +1,86 @@ +.. meta:: + :description: Getting Started for Equinix Metal + +######################################## +Equinix Metal API integration enablement +######################################## + + +For each Equinix Metal Project+location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "Equinix Metal" from the dropdown menu. + * - Name + - Type a descriptive name for your Equinix Metal Project+location. + * - Equinix Project ID + - Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID. + * - Equinix Project API key + - Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here. + * - Equinix Location + - Select your equinix location from the dropdown menu. + + +Equinix Metal Project ID + +.. image:: /tutorials/images/equinix-metal-project-id.png + :align: center + + +Equinix Metal Project API key + +.. image:: /tutorials/images/equinix-metal-project-api-keys.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/netris-create-equinix-metal-site.png + :align: center + + +Adding Netris SoftGate nodes +============================ + +For SoftGate nodes you can start with two servers of the smallest flavor. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one. + +Request two servers from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned. + +1) At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory + +.. image:: /tutorials/images/softgate-nodes-created-in-equinix.png + :align: center + +2) When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”. + + Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location). + +Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + +.. image:: /tutorials/images/softgate-nodes-recognized-in-netris.png + :align: center + +3) Provision SoftGate nodes. + +Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there. + +.. image:: /tutorials/images/softgate-one-liner-provisioning.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/softgate-green.png + :align: center diff --git a/en/latest/_sources/tutorials/index-equinix.rst.txt b/en/latest/_sources/tutorials/index-equinix.rst.txt new file mode 100644 index 0000000000..d117bbc631 --- /dev/null +++ b/en/latest/_sources/tutorials/index-equinix.rst.txt @@ -0,0 +1,25 @@ +================================================== +Netris VPC for Equinix Metal Getting Started Guide +================================================== + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + +This `video `_ walks through Netris VPC setup steps described below. + + +.. toctree:: + :maxdepth: 2 + :caption: Step-by-step Netris Setup Guide for Equinix Metal + + install-netris-controller-in-equinix-metal + equinix-metal-api-integration-enablement + adding-netris-softgate-nodes-in-equinix-metal + activating-bgp-on-equinix-metal-project + enable-services-on-equinix-metal-project + +.. toctree:: + :maxdepth: 1 + :caption: Examples of Using Netris VPC on Equinix Metal Project + + using-vnet-in-equinix-metal-project + using-l4-load-balancer diff --git a/en/latest/_sources/tutorials/index-vpc.rst.txt b/en/latest/_sources/tutorials/index-vpc.rst.txt new file mode 100644 index 0000000000..e538bfa7f0 --- /dev/null +++ b/en/latest/_sources/tutorials/index-vpc.rst.txt @@ -0,0 +1,13 @@ +====================================== +VPC for Anywhere Getting Started Guide +====================================== + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + + +.. toctree:: + :maxdepth: 2 + + vpc-anywhere-concept + vpc-anywhere-controller-installation + vpc-anywhere-softgate-installation diff --git a/en/latest/_sources/tutorials/index.rst.txt b/en/latest/_sources/tutorials/index.rst.txt new file mode 100644 index 0000000000..46c6f6031b --- /dev/null +++ b/en/latest/_sources/tutorials/index.rst.txt @@ -0,0 +1,12 @@ +======================== +Netris Generic Tutorials +======================== + +You are welcome to join our `Slack channel `_ to get additional support from our engineers and community. + + +.. toctree:: + :maxdepth: 2 + + installing-netris-controller + upgrading-netris diff --git a/en/latest/_sources/tutorials/install-netris-controller-in-equinix-metal.rst.txt b/en/latest/_sources/tutorials/install-netris-controller-in-equinix-metal.rst.txt new file mode 100644 index 0000000000..b0503c08df --- /dev/null +++ b/en/latest/_sources/tutorials/install-netris-controller-in-equinix-metal.rst.txt @@ -0,0 +1,75 @@ +.. meta:: + :description: Installing a Netris Controller in Equinix Metal + +================================================================ +Installing a Netris Controller on Equinix Metal on-demand server +================================================================ + +Installation Steps +------------------ + +You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions). + +**Request a server** + +The smallest flavor, c3.small.x86, is enough for most users. + +.. image:: images/equinix-request-c3-small-server.png + :align: center + +if you decide to use a VM, please see below the minimal VM requirements. + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**DNS record** + +In my example my host got a public IP address 139.178.89.255. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is example using Cloudflare DNS service. (same idea with any DNS software or service) + +.. image:: images/dns-cloudflare-equinix-ip.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 139.178.89.255 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + + +Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center + diff --git a/en/latest/_sources/tutorials/installing-netris-controller.rst.txt b/en/latest/_sources/tutorials/installing-netris-controller.rst.txt new file mode 100644 index 0000000000..6cb57398b9 --- /dev/null +++ b/en/latest/_sources/tutorials/installing-netris-controller.rst.txt @@ -0,0 +1,85 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Installing a Netris Controller +============================== + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there's no need for an individual controller for each deployment. + +It doesn't matter where you host the Netris controller. What matters is that 1) the Netris controller needs to be accessible over the Internet. 2) You can access the web console. 3) Nodes that are going to be managed by Netris have access to the Netris controller through their management network interface. + +**Linux host requirements** + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**DNS record** + +In the example below, the host has a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, it is recommended to use a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is an example using Cloudflare DNS service (may use any DNS service). + +.. image:: images/dns-record-netrisctl.png + :align: center + +Ensure that newly created domain name resolves to the right IP address of the machine that will host the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 54.183.23.201 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + The Netris Controller installer will create a K3s cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let's Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That's why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + + +Once the installation process is finished, you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center + +**Restrict incoming TCP requests to the list below:** + ++----------+--------------------------------+ +| TCP Port | Service | ++==========+================================+ +| 22 | SSH | ++----------+--------------------------------+ +| 80 | HTTP | ++----------+--------------------------------+ +| 443 | Netris Web Console | ++----------+--------------------------------+ +| 2003 | Streaming Telemetry (Collectd) | ++----------+--------------------------------+ +| 3033 | Netris Monitoring (Telescope) | ++----------+--------------------------------+ +| 50051 | Netris Agent (gRPC) | ++----------+--------------------------------+ + diff --git a/en/latest/_sources/tutorials/inventory-setup.rst.txt b/en/latest/_sources/tutorials/inventory-setup.rst.txt new file mode 100644 index 0000000000..48cb9e7b6a --- /dev/null +++ b/en/latest/_sources/tutorials/inventory-setup.rst.txt @@ -0,0 +1,46 @@ +############### +Inventory setup +############### + +The Inventory section includes Netris-managed devices and allows you to add, edit, or delete network switches and SoftGates. +The initial setup of the Netris managed fabric consists of a three-step process: + +* Create Inventory Profiles +* Add Switches +* Add Softgates + +.. note:: + You can also add new devices in the Topology view. + + +**Inventory profiles** + +Inventory profiles enable security hardening for inventory devices. By default, all traffic flow destined for a switch or SoftGate is allowed. However, once an inventory profile is attached to a device, it denies all traffic destined for the device except for Netris-defined and user-defined custom flows. Generated rules include: + +* SSH from user-defined subnets +* NTP from user-defined NTP services +* DNS from user-defined DNS servers +* Custom user-defined rules + +The Netris Controller includes a preconfigured Inventory profile named "default-inventory-profile." You can either edit this profile or create your own. + +**SoftGate creation** + +Each SoftGate node needs to be added to the Netris Controller inventory. +Network → Inventory → +Add + +.. image:: images/inventory_softgate.png + :align: center + +**Switch creation** + +Each Switch node needs to be added to the Netris Controller inventory. +Network → Inventory → +Add + +.. image:: images/inventory_switch.png + :align: center + + + + + diff --git a/en/latest/_sources/tutorials/ipam-setup.rst.txt b/en/latest/_sources/tutorials/ipam-setup.rst.txt new file mode 100644 index 0000000000..99c8ee6528 --- /dev/null +++ b/en/latest/_sources/tutorials/ipam-setup.rst.txt @@ -0,0 +1,53 @@ +########## +IPAM setup +########## + + +Netris IPAM enables users to manage their IP addresses and monitor pool usage effectively. It features a hierarchical view to facilitate various subnetting tasks. +Users must first assign specific roles (purposes) to each subnet or address before they can utilize these subnets in services such as V-Net, NAT, Load Balancing, etc.. +Each VPC has its own IPAM table. + +**Create allocations** + +There are two primary types of IP prefixes: allocations and subnets. Allocations consist of IP ranges assigned to an organization through RIR/LIR or private IP ranges intended for network use. Subnets, on the other hand, are prefixes that will be utilized in various services. Subnets always fall under allocations, while allocations do not have parent subnets. + +In addition to the predefined subnets, the Netris Controller also includes predefined allocations, consisting of private IP addresses defined in RFC 1918 - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. If you intend to create subnets that fall outside of these predefined allocations, you should first create allocations that encompass those subnets. + +.. image:: images/ipam_allocation.png + :align: center + +**Create subnets for devices** + +You will require two subnets for your devices: one for loopback IP addresses and another for the management network. Note that device subnets must reside in the System VPC. + +.. image:: images/ipam_mgmt_subnet.png + :align: center + +.. image:: images/ipam_loopback_subnet.png + :align: center + + +**Create subnets for V-Nets** + +Create at least one subnet with the Common purpose to use it for a new V-Net. IP addresses from this subnet will be assigned to your servers. + +.. image:: images/ipam_common_subnet.png + :align: center + + +**Create subnets for Load-balancing service** + +If you plan to use load-balancing services, you should first define subnet(s) from which IP addresses will be assigned for Virtual IP (frontend). + +.. image:: images/ipam_l4lb_subnet.png + :align: center + + +**Create subnets for NAT service** + +If you plan to perform network address translation (NAT), you must first create subnets for this purpose. + +.. image:: images/ipam_nat_subnet.png + :align: center + + diff --git a/en/latest/_sources/tutorials/more-features.rst.txt b/en/latest/_sources/tutorials/more-features.rst.txt new file mode 100644 index 0000000000..eea8662f9a --- /dev/null +++ b/en/latest/_sources/tutorials/more-features.rst.txt @@ -0,0 +1,8 @@ +============= +More features +============= + +:doc:`Access lists` + +Static routing, Site Mesh (VPN) and Routing on the host features can be found under :doc:`Network Policies`. + diff --git a/en/latest/_sources/tutorials/netris-controller-installation.rst.txt b/en/latest/_sources/tutorials/netris-controller-installation.rst.txt new file mode 100644 index 0000000000..1832a62bc7 --- /dev/null +++ b/en/latest/_sources/tutorials/netris-controller-installation.rst.txt @@ -0,0 +1,88 @@ +########################### +Install a Netris Controller +########################### + +Requirements and Installation steps +----------------------------------- + +You can install the Netris controller on almost any 64-bit Linux host. The Netris Controller may or may not be on the same network as the managed network nodes. Multi-region, multi-site deployments can use a single Netris Controller (+backup). + +The location of the Netris controller host is not critical. However, it is important that the Netris controller is accessible over the network. This is required for two reasons: 1) So you can access the web console, and 2) Nodes that will be managed by Netris need access to the Netris controller through their management network interface. + +If you choose to set up the Netris Controller within your Netris managed network, make sure to provide access to the controller using an out-of-band network connection. + + +**Linux host requirements** + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**DNS record** + +In this example, the controller host has a public IP address 54.183.23.201. While it is acceptable for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record. This approach will make it easier to potentially move the Netris Controller to a different IP address in the future. + +Below is an example using the Cloudflare DNS service. The same concept applies to any DNS software or service. + +.. image:: images/dns-record-netrisctl.png + :align: center + +Ensure that the newly created domain name indeed resolves to the correct IP address of the machine on which you plan to install the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 54.183.23.201 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + The Netris Controller installer will create a K3s cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let's Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That's why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + + +Once the installation process is finished, you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center + +**Restrict incoming TCP requests to the list below:** + ++----------+--------------------------------+ +| TCP Port | Service | ++==========+================================+ +| 22 | SSH | ++----------+--------------------------------+ +| 80 | HTTP | ++----------+--------------------------------+ +| 443 | Netris Web Console | ++----------+--------------------------------+ +| 2003 | Streaming Telemetry (Collectd) | ++----------+--------------------------------+ +| 3033 | Netris Monitoring (Telescope) | ++----------+--------------------------------+ +| 50051 | Netris Agent (gRPC) | ++----------+--------------------------------+ + diff --git a/en/latest/_sources/tutorials/netris-managed-fabric-overview.rst.txt b/en/latest/_sources/tutorials/netris-managed-fabric-overview.rst.txt new file mode 100644 index 0000000000..312c2128f0 --- /dev/null +++ b/en/latest/_sources/tutorials/netris-managed-fabric-overview.rst.txt @@ -0,0 +1,18 @@ +############################## +Netris managed fabric Overview +############################## + +Introduction +------------ + +VPC gateways (SoftGate) with Netris managed fabric is a comprehensive solution for operating physical networks as if they were cloud-based. By abstracting the complexities of granular network configuration, Netris enables you to operate your physical network using a top-down approach, similar to cloud management, instead of the traditional box-by-box operation. + +Concept +------- + +The Netris managed fabric is a switch fabric that employs EVPN technology for enhanced configuration and performance. The Netris Switch Agent should be installed and operational within the user space of the switch's network operating system (NOS). The agent is responsible for automatically generating specific switch configurations based on service requirements and policies defined in the Netris Controller. The Netris managed fabric delivers high capacity, redundancy, resilience, and high availability, along with traffic filtering based on access lists for optimal network performance. + +Netris SoftGate, known as VPC gateway, is a software for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), DHCP, and site-to-site VPN function on a regular x86 server with a SmartNIC card or without it. + +.. image:: /tutorials/images/switch_fabric_vpc.png + :align: center diff --git a/en/latest/_sources/tutorials/netris-switch-agent-installation.rst.txt b/en/latest/_sources/tutorials/netris-switch-agent-installation.rst.txt new file mode 100644 index 0000000000..d40c83a61d --- /dev/null +++ b/en/latest/_sources/tutorials/netris-switch-agent-installation.rst.txt @@ -0,0 +1,5 @@ +################################ +Netris Switch Agent Installation +################################ + +The Netris Switch Agent is compatible with several Network Operating Systems. :doc:`Here` is a list of these systems, along with installation guides for each one. Install the Netris Agent and come back to this tutorial. diff --git a/en/latest/_sources/tutorials/netris-vpc-for-aws.rst.txt b/en/latest/_sources/tutorials/netris-vpc-for-aws.rst.txt new file mode 100644 index 0000000000..97b963522f --- /dev/null +++ b/en/latest/_sources/tutorials/netris-vpc-for-aws.rst.txt @@ -0,0 +1,15 @@ +=========================== +Enabling Site Mesh with AWS +=========================== + + +.. toctree:: + :maxdepth: 2 + + aws-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + aws-deploy-softgate diff --git a/en/latest/_sources/tutorials/netris-vpc-for-equinix-metal.rst.txt b/en/latest/_sources/tutorials/netris-vpc-for-equinix-metal.rst.txt new file mode 100644 index 0000000000..ad1cb39177 --- /dev/null +++ b/en/latest/_sources/tutorials/netris-vpc-for-equinix-metal.rst.txt @@ -0,0 +1,21 @@ +====================================== +Getting Started with Equinix Metal VPC +====================================== + + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + install-netris-controller-in-equinix-metal + equinix-metal-api-integration-enablement + adding-netris-softgate-nodes-in-equinix-metal + activating-bgp-on-equinix-metal-project + enable-services-on-equinix-metal-project + +.. toctree:: + :maxdepth: 1 + :caption: Use + + using-vnet-in-equinix-metal-project + using-l4-load-balancer diff --git a/en/latest/_sources/tutorials/netris-vpc-for-gcp.rst.txt b/en/latest/_sources/tutorials/netris-vpc-for-gcp.rst.txt new file mode 100644 index 0000000000..53cc4f6e06 --- /dev/null +++ b/en/latest/_sources/tutorials/netris-vpc-for-gcp.rst.txt @@ -0,0 +1,15 @@ +=========================== +Enabling Site Mesh with GCP +=========================== + + +.. toctree:: + :maxdepth: 2 + + gcp-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + gcp-deploy-softgate diff --git a/en/latest/_sources/tutorials/netris-vpc-for-phoenixnap-bmc.rst.txt b/en/latest/_sources/tutorials/netris-vpc-for-phoenixnap-bmc.rst.txt new file mode 100644 index 0000000000..732ab6d837 --- /dev/null +++ b/en/latest/_sources/tutorials/netris-vpc-for-phoenixnap-bmc.rst.txt @@ -0,0 +1,24 @@ +=================================== +Getting Started with PhoenixNAP VPC +=================================== + + +.. toctree:: + :maxdepth: 2 + + phoenixnap-bmc-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + phoenixnap-bmc-link-to-installation + phoenixnap-bmc-ipam-setup + +.. toctree:: + :maxdepth: 2 + :caption: Use + + phoenixnap-bmc-using-nat + phoenixnap-bmc-using-vnet + phoenixnap-bmc-using-l4lb diff --git a/en/latest/_sources/tutorials/new-site-setup.rst.txt b/en/latest/_sources/tutorials/new-site-setup.rst.txt new file mode 100644 index 0000000000..af39a1cf97 --- /dev/null +++ b/en/latest/_sources/tutorials/new-site-setup.rst.txt @@ -0,0 +1,19 @@ +############## +New Site setup +############## + +For each individual deployment, region, location, data center, etc. you should define it as a Site. All network components and resources should be associated with their respective Site and VPC. + +Add a new Site. +Network → Sites → +Add + +* Enter the Site name +* Enter your public Autonomous System Number (ASN) if you have one. If not, use a private ASN within the range of 64512 to 65534. +* Choose Switch fabric type “Netris”. + +If you're implementing the Zero Trust security model, you may want to select the ACL Default Policy "Deny." More details can be found +:doc:`here`. + +.. image:: /tutorials/images/site_setup.png + :align: center + diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst.txt new file mode 100644 index 0000000000..e13e2abe1a --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst.txt @@ -0,0 +1,37 @@ +.. meta:: + :description: Provisioning Netris SoftGate nodes in phoenixNAP BMC + +.. _phxnap_sgs: + +#################################################### +Provisioning Netris SoftGate nodes in phoenixNAP BMC +#################################################### + +For SoftGate nodes you can start with two ``s2.c1.small`` or larger servers. In the future, if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration), you can upgrade the servers one-by-one. + +1) Request two servers (s2.c1.small) from phoenixNAP BMC with Ubuntu Jammy OS, type ``netris-softgate`` in the "Server Description" field, +choose a /31 Public IP Allocation and wait until provisioned. + +.. note:: + + It’s required to type the ``netris-softgate`` in the description. This signals Netris Controller that those are not regular bare-metal servers, and they should be synced with the type of SoftGate. + +.. image:: /tutorials/images/phoenixnap-softgate-nodes-creation.png + :align: center + +2) At this point you should see Netris Controller listing newly created servers as “Softgate Softgate1(2)” under Netris Web Console → Net → Inventory. +You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step. + + +3) Provision SoftGate nodes. Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command. + +Then SSH to the corresponding SoftGate server with the ``ubuntu`` user and paste the one-liner there. + +.. image:: /tutorials/images/phoenixnap-softgate-nodes-created.png + :align: center + +When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too. + +.. image:: /tutorials/images/phoenixnap-softgate-nodes-green.png + :align: center diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-api-integration-enablement.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-api-integration-enablement.rst.txt new file mode 100644 index 0000000000..26c9b7ecec --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-api-integration-enablement.rst.txt @@ -0,0 +1,47 @@ +.. meta:: + :description: Enable phoenixNAP BMC API integration + +.. _phxnap_api: + +##################################### +Enable phoenixNAP BMC API integration +##################################### + + +For each phoenixNAP BMC location you need to define an individual Site in Netris Controller. + +Go to Netris Web Console → Net → Sites and click +Add. + +You only need to deal with the below 5 fields. Leave the rest to default values for now. + + +.. list-table:: + :widths: 25 50 + :header-rows: 1 + + * - Netris Parameter + - What to do: + * - Switch Fabric + - Select "PhoenixNAP BMC" from the dropdown menu. + * - Name + - Type a descriptive name for your phoenixNAP BMC location. + * - PhoenixNAP Client ID + - Create a new API Credential with "bmc" scope in phoenixNAP BMC portal under API Credentials → + Create Credentials. Then copy/paste Client ID. + * - PhoenixNAP Client Secret + - Copy/Paste the Client Secret from the already created API Credential. + * - PhoenixNAP Location + - Select your phoenixNAP BMC location from the dropdown menu. + + +phoenixNAP BMC API Credential + +.. image:: /tutorials/images/phoenixnap-api-credential.png + :align: center + + +Netris Create New Site + +.. image:: /tutorials/images/phoenixnap-site-create.png + :align: center + + diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-concept.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-concept.rst.txt new file mode 100644 index 0000000000..1bdf8cc43d --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-concept.rst.txt @@ -0,0 +1,40 @@ +###################################### +Netris VPC for phoenixNAP BMC Overview +###################################### + +Introduction +------------- + +Netris VPC for phoenixNAP BMC is a solution that enables VPC network functionality in phoenixNAP's Bare Metal Cloud. Netris VPC can be used for all or part of the servers/network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures. + + +Concept +-------- + +There is **an agent** inside the Netris VPC controller for phoenixNAP BMC that communicates to phoenixNAP API once the `integration has been enabled `_. That agent is responsible for metadata synchronization between phoenixNAP BMC and Netris VPC controller. Also, it configures the BMC network services based on the services defined in the Netris VPC controller. + +**SoftGate** is a highly optimized automatic Linux gateway, which in turn communicates with the Netris VPC controller via an encrypted protocol. A SoftGate node (typically two of them for redundancy) is a regular BMC server that should be deployed on desired phoenixNAP location. Once the server has been `deployed `_, it starts consuming the BMC public and private networks with various VLANS. + +.. image:: images/phoenixnap-concept-solution-traffic-flows.png + :align: center + +Due to phoenixNAP BMC creating a new VLAN for each server's Public IP Allocation and Public Network/Private Network, we decided to use the upper VLANS Range - 3000-4094 (if you need to change the default range, that can be done in the Netris Site settings). Netris will consume VLANS only from that range. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the BMC network. + +SoftGate node becomes the default gateway for the workloads consuming the VPC network. PhoenixNAP BMC Private Network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers. Netris also creates a Public Network in the phoenixNAP BMC when the IP Addresses from Public IP Allocations (passed to Netris VPC) :ref:`have been used` in any Netris VPC's services. Besides Public/Private Networks creation Netris also attaches all necessary servers to that networks. Therefore, you don’t need to worry about assigning servers to the networks on the phoenixNAP side, Netris will handle that automatically when that makes sense. + +.. image:: images/phoenixnap-concept-public-network.png + :align: center + + + +**What's next** + +* :doc:`Installing Netris on phoenixNAP BMC ` +* :doc:`IPAM Setup for Services ` + + +Usage + +* :doc:`Using NAT services in phoenixNAP BMC ` +* :doc:`Using V-Net (isolated virtual network) services in phoenixNAP BMC ` +* :doc:`Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC ` diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-install-netris-controller.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-install-netris-controller.rst.txt new file mode 100644 index 0000000000..e6bd4faf1f --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-install-netris-controller.rst.txt @@ -0,0 +1,74 @@ +.. meta:: + :description: Installing a Netris Controller in phoenixNAP BMC + +================================================================= +Installing a Netris Controller on phoenixNAP BMC server +================================================================= + +Installation Steps +------------------ + +You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions). + +**Minimum Hardware Requirements** + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**Deploy New Server** + +Deploy a new server in phoenixNAP BMC: select the Location, choose the server flavor (the smallest flavor, s0.d1.small, is enough for most users), type some name for Controller Server, request a new /31 public allocation for this instance. + +.. image:: images/phoenixnap-request-ctl-server.png + :align: center + +**DNS record** + +In my example my host got a public IP address 131.153.154.61. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is example using Cloudflare DNS service. (same idea with any DNS software or service) + +.. image:: images/phoenixnap-dns-cloudflare.png + :align: center + +Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 131.153.154.61 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netrisctl.netris.dev --ctl-ssl-issuer letsencrypt + +.. note:: + Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + + +Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-ipam-setup.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-ipam-setup.rst.txt new file mode 100644 index 0000000000..b30014d49d --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-ipam-setup.rst.txt @@ -0,0 +1,116 @@ +.. meta:: + :description: IPAM Setup for Services + +.. _phxnap_services: + +####################### +IPAM Setup for Services +####################### + +Although bare metal servers in phoenixNAP BMC may get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization, something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities. + +Both NAT and on-demand Load Balancer services need public IP addresses. + +Go to phoenixNAP BMC web console and click on Networking → Public IP Allocations (see the screenshot below) + +Option 1 - Separated allocation for each purpose +================================================ + +In this example, I’m requesting two /29 (5 assignable IPs) IP Allocations, one for NAT and one for Load Balancer. + +It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC. + +You can always request more IP address allocations in the future. + +.. image:: /tutorials/images/phoenixnap-request-ip-allocation.png + :align: center + +Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM + +.. image:: /tutorials/images/phoenixnap-netris-ipam-synced.png + :align: center + +You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service). + + +1. Enable on-demand (elastic) Load Balancer +------------------------------------------- + +To Enable an on-demand (elastic) Load Balancer, you only need to change the “purpose” field of appropriate IP subnet from “inactive” into “load-balancer” + +Click on the 3 dots menu (in this example of first /29 subnet), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-lb-purpose.png + :align: center + +Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform. + +Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it. + + +2. Enable NAT +------------- + +To enable NAT, you need to repurpose a subnet for NAT. In the below example, I’m repurposing the second of the newly requested /29 subnets for NAT. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-nat-purpose.png + :align: center + +Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. + + +Option 2 - Splitting single allocation into different subnets +============================================================= + +In this example, I’m requesting a single /28 (13 assignable IPs) Public IP Allocation, then splitting it into two /29 subnets, one for NAT and one for Load Balancer. + +It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC. + +You can always request more IP address allocations in the future. + +.. image:: /tutorials/images/phoenixnap-request-ip-allocation-slash-28.png + :align: center + +Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM + +.. image:: /tutorials/images/phoenixnap-netris-ipam-synced-slash-28.png + :align: center + +You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service). + + +1. Enable on-demand (elastic) Load Balancer +------------------------------------------- + +In this scenario, to Enable an on-demand (elastic) Load Balancer, you need to create a smaller subnet with the purpose of “load-balancer”. + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-lb-purpose-slash-28.png + :align: center + +Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform. + +Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it. + + +2. Enable NAT +------------- + +To enable NAT, you need to create a smaller subnet with the purpose of “nat”. + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/phoenixnap-netris-ipam-nat-purpose-slash-28.png + :align: center + +Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. + +Note* +===== + +It doesn't matter with what Option (1,2) you will go. Once the Public IP Allocation has been replicated in the Netris IPAM, Netris will automatically reserve the network, first usable, and broadcast IP addresses because they are unusable in this (phoenixNAP BMC) scenario. + +.. image:: /tutorials/images/phoenixnap-reserved-ips.png + :align: center + diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-link-to-installation.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-link-to-installation.rst.txt new file mode 100644 index 0000000000..8be20c8905 --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-link-to-installation.rst.txt @@ -0,0 +1,12 @@ +.. meta:: + :description: Installing Netris on phoenixNAP BMC + +.. _pnap_doc_link: + +################################### +Installing Netris on phoenixNAP BMC +################################### + +To deploy Netris on the phoenixNAP Bare Metal Cloud, please follow the official guide provided by phoenixNAP. This guide will walk you through the step-by-step installation process. + +`Netris for Bare Metal Cloud `_ diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-using-l4lb.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-using-l4lb.rst.txt new file mode 100644 index 0000000000..466bcee6c8 --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-using-l4lb.rst.txt @@ -0,0 +1,18 @@ +.. meta:: + :description: Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC + +.. _phxnap_l4lb: + +#################################################################### +Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC +#################################################################### + +Services --> L4 Load Balancer is an on-demand (elastic) load balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console. + +.. image:: /tutorials/images/phoenixnap-l4lb.png + :align: center + diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-using-nat.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-using-nat.rst.txt new file mode 100644 index 0000000000..735aa2c992 --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-using-nat.rst.txt @@ -0,0 +1,42 @@ +.. meta:: + :description: Using NAT services in phoenixNAP BMC + +.. _phxnap_nat: + +#################################### +Using NAT services in phoenixNAP BMC +#################################### + +NAT, under Net --> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 10.0.0.0/8 to have access to services outside their VPC. The goal is to provide bare-metal servers/VMs with the capability to connect to the Internet through NAT for outbound access. + +Option 1 - MASQUERADE +===================== + +The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn't require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT). + +.. image:: images/phoenixnap-nat-masquerade.png + :align: center + + +Option 2 - SNAT +=============== + +Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a :ref:`NAT purpose`. SNAT replaces the source IP address of the instances with the IP address of the "SNAT to IP". Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity. + +.. image:: images/phoenixnap-nat-snat.png + :align: center + + +You can always have more granular control either through NAT rule or using Services → ACLs. + + +DNAT +==== + +A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a "DNAT to IP/Port". This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database. + +I'm creating a DNAT rule for the ssh port in the example below. It forwards the public IP's 55022 port to the local IP's 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server. + +.. image:: images/phoenixnap-nat-dnat.png + :align: center + diff --git a/en/latest/_sources/tutorials/phoenixnap-bmc-using-vnet.rst.txt b/en/latest/_sources/tutorials/phoenixnap-bmc-using-vnet.rst.txt new file mode 100644 index 0000000000..4f08a07203 --- /dev/null +++ b/en/latest/_sources/tutorials/phoenixnap-bmc-using-vnet.rst.txt @@ -0,0 +1,136 @@ +.. meta:: + :description: Using V-Net (isolated virtual network) services in phoenixNAP BMC + +.. _phxnap_vnet: + +################################################################## +Using V-Net (isolated virtual network) services in phoenixNAP BMC +################################################################## + +V-Net, under Services --> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from phoenixNAP BMC into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network. + +Netris V-Net (in phoenixNAP BMC scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Private Network using phoenixNAP BMC API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Private Network. SoftGate nodes are the default gateway for the V-Net services. + +Adding Subnets for V-Net +======================== + +Before starting to use the V-NETs, you need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources. + +.. image:: /tutorials/images/phoenixnap-netris-create-common-subnets.png + :align: center + +Creating a V-Net +================ + +You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, phoenixNAP BMC, and your Compute. + +.. image:: /tutorials/images/phoenixnap-netris-creating-vnet.png + :align: center + +In this example, the new V-NET has VLAN ID 3000, subnet 10.128.1.0/24, and gateway 10.128.1.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 3000, and they should use IP addresses from 10.128.1.2-254 pointing to 10.128.1.1 as the default gateway or use DHCP. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 3000 will have Internet access over the NAT. + +.. image:: /tutorials/images/phoenixnap-netris-vnet-ready.png + :align: center + + +Deploy a new server w/ dynamic IP into an existing V-Net +-------------------------------------------------------- + +Deploying a new server is significantly simpler when using Netris V-NET compared to the traditional process of setting up a server with a public IP. + +The recommended steps: + +1. **Request a Server:** When ordering a server from phoenixNAP BMC, opt for no public IP address. +2. **Select Netris V-Net:** In the phoenixNAP web console, select a Netris V-Net (previously created in Netris) as your private network. +3. **DHCP:** In the phoenixNAP web console, check the option "**Use your own privately managed DHCP server (Obtain IP address automatically)**". Make sure that DHCP is enabled for the given V-Net through the Netris web console. + +Netris will automatically include the server into the V-Net and will assign an IP configuration through Netris DHCP service. + +.. image:: /tutorials/images/phoenixnap-vnet-import-a-new-server.png + :align: center + +As a result, you will have bare-metal servers in a private subnet that can connect to services outside your VPC (since we have enabled NAT globally in previous chapters), but external services cannot initiate a connection with those servers. Once the servers have been provisioned, they will get a private IP from Netris DHCP, and you can find those IPs by pressing the "**IP/MAC Info**" button on the V-NET. + +.. image:: /tutorials/images/phoenixnap-vnet-imported-new-server.png + :align: center + +In order to connect via SSH to the newly deployed server, you can either create a DNAT rule and connect via Public IP, or if you don't need permanent ssh access to that server, you can simply connect using Softgate as a JumpHost. + +.. code-block:: shell-session + + ssh -o ProxyCommand="ssh -W %h:%p ubuntu@" ubuntu@ + + +.. image:: /tutorials/images/phoenixnap-vnet-ssh-to-server.png + :align: center + + +.. _phxnap_vnet_static_ip: + + +Deploy a new server w/ static IP into an existing V-Net +------------------------------------------------------- + +Deploying a new server is significantly simpler when using Netris V-NET compared to the traditional process of setting up a server with a public IP. + +The recommended steps: + +1. **Request a Server:** When ordering a server from phoenixNAP BMC, opt for no public IP address. +2. **Select Netris V-Net:** In the phoenixNAP web console, select a Netris V-Net (previously created in Netris) as your private network. +3. **IP Configuration:** In the phoenixNAP web console, enter a valid IP address in the 'IP Addresses' text field and specify the Netris V-Net's gateway IP address in the 'Default Gateway' field. + +Netris will automatically include the server into the V-Net. + + +.. image:: /tutorials/images/phoenixnap-vnet-import-a-new-server-with-ip.png + :align: center + +As a result, you will have bare-metal servers in a private subnet that can connect to services outside your VPC (since we have enabled NAT globally in previous chapters), but external services cannot initiate a connection with those servers. + +In order to connect via SSH to the newly deployed server, you can either create a DNAT rule and connect via Public IP, or if you don't need permanent ssh access to that server, you can simply connect using Softgate as a JumpHost. + +.. code-block:: shell-session + + ssh -o ProxyCommand="ssh -W %h:%p ubuntu@" ubuntu@ + + +.. image:: /tutorials/images/phoenixnap-vnet-ssh-to-server.png + :align: center + +Tags +---- + +Tags are used to associate bare-metal servers with V-NET dynamically. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers from the Private Network based on that tag. Thus, you can make flexible V-NETs, and there is no need to include every new server in the V-NET. + +.. image:: /tutorials/images/phoenixnap-vnet-with-tag.png + :align: center + +This feature is even more efficient when you build your infrastructure via Terraform. For example, let's say you've created a V-NET with a tag using Netris Terraform Provider, then order several servers with the same tag using phoenixNAP Terraform Provider. And that's it, when the servers are ready, Netris will detect them and make them part of the V-NET. + +.. image:: /tutorials/images/phoenixnap-vnet-with-tag-terraform.png + :align: center + + +Unmanaged +--------- + +Another option is turning the existing private network into Netris V-Net. All private networks from the allowed VLAN IDs range and in the proper location that Netris has not created are visible as "unmanaged" in the V-Net section. + +.. image:: /tutorials/images/phoenixnap-vnet-unmanaged-vnet.png + :align: center + +The "manage" button will open a dialogue window where it's also possible to add a default gateway for the appropriate VLAN. + + +.. warning:: + Once the private network is being converted into V-Net, it will be managed by Netris and no longer manageable through phoenixNAP BMC console. + +.. image:: /tutorials/images/phoenixnap-vnet-managed-vnet.png + :align: center + +Note that you can use Services --> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites) + + +.. note:: + + Netris creates a private network in phoenixNAP BMC based on declared V-Nets. Besides creation, Netris continuously monitors that private networks. As a result of this continuous monitoring, you can't edit private networks created by Netris from the phoenixNAP BMC console. However, if any modifications are made, Netris will automatically roll everything back to its state. diff --git a/en/latest/_sources/tutorials/softgate-software-provisioning.rst.txt b/en/latest/_sources/tutorials/softgate-software-provisioning.rst.txt new file mode 100644 index 0000000000..578da0dd37 --- /dev/null +++ b/en/latest/_sources/tutorials/softgate-software-provisioning.rst.txt @@ -0,0 +1,121 @@ +============================== +SoftGate software provisioning +============================== + +######################################### +SoftGate PRO (100G) software provisioning +######################################### + +SoftGate PRO is the high performance version that takes advantage of DPDK and SmartNIC technologies. + +Check hardware requirements and BIOS configuration :doc:`here`. + +**Install the Netris Agent** + +A requirement for this setup is a newly installed Ubuntu Linux 20.04 LTS, along with internet connectivity through the management port configured using netplan. + +1. Navigate to Network → Inventory. + +.. image:: images/sg_agent_install.png + :align: center + +Copy the one-liner displayed in the window and execute it as a regular user on the SoftGate. + +2. Upon completing the installation, examine the ifupdown configuration file and ensure that the displayed configuration aligns with what you set up during the OS installation process (the file is generated based on your initial netplan configuration). + +.. note:: + If the Netris Controller is not in the same OOB network, then add a route to the Netris Controller. No default route or other IP addresses should be configured." + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The management network interface + auto ensZ + iface ensZ inet static + address + # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node. + up ip route add via + gateway + + source /etc/network/interfaces.d/* + + +If the Netris agent is able to reach the controller, please remove/comment the Gateway line and save the file. + +.. note:: + Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent. + +3. Reboot the SoftGate. + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up you should see its heartbeat going from Critical to OK in Network → Inventory. + +######################################## +SoftGate (1G, 10G) software provisioning +######################################## + +Use SoftGate non-pro if you are not using SmartNIC cards. +Check hardware requirements here. + +**Install the Netris Agent** + +A requirement for this setup is a newly installed Ubuntu Linux 22.04 LTS, along with internet connectivity through the management port configured using netplan. + +1. Navigate to Network → Inventory. + +.. image:: images/sg_agent_install.png + :align: center + +Copy the one-liner displayed in the window and execute it as a regular user on the SoftGate. + +2. Upon completing the installation, examine the ifupdown configuration file and ensure that the displayed configuration aligns with what you set up during the OS installation process (the file is generated based on your initial netplan configuration). + +.. note:: + If the Netris Controller is not in the same OOB network, then add a route to the Netris Controller. No default route or other IP addresses should be configured. + +.. code-block:: shell-session + + user@host:~$ sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # Physical port on SoftGate node connected to a TRUNK port of your network + auto ens + iface ens inet static + address 0.0.0.0/0 + + + # Bond interface + auto bond0 + iface bond0 inet static + address 0.0.0.0/0 + # Please replace the ensX/Y with the actual interface name(s) below to one(s) present in the OS. + bond-slaves ens + source /etc/network/interfaces.d/* + +If the Netris agent is able to reach the controller, please remove/comment the Gateway line and save the file. + +.. note:: + Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent. + +3. Reboot the SoftGate. + +.. code-block:: shell-session + + user@host:~$ sudo reboot + +Once the server boots up you should see its heartbeat going from Critical to OK in Network → Inventory. diff --git a/en/latest/_sources/tutorials/topology-setup.rst.txt b/en/latest/_sources/tutorials/topology-setup.rst.txt new file mode 100644 index 0000000000..ad564919d5 --- /dev/null +++ b/en/latest/_sources/tutorials/topology-setup.rst.txt @@ -0,0 +1,18 @@ +############## +Topology setup +############## + +The topology manager is used for describing and monitoring the desired network topology. Netris software will configure the underlying network devices according to this topology dynamically and will watch against potential failures. Wire your switches in accordance with the topology view. + +Right-click on the desired device and choose "Create Link". Select switch-ports for current and remote nodes. + +.. image:: images/topology.png + :align: center + +.. image:: images/topology_create_link.png + :align: center + +.. image:: images/topology_completed.png + :align: center + +Once the links have been defined, the network is automatically configured as long as physical connectivity is in place and Netris Agents can communicate with the Netris Controller. diff --git a/en/latest/_sources/tutorials/upgrading-netris.rst.txt b/en/latest/_sources/tutorials/upgrading-netris.rst.txt new file mode 100644 index 0000000000..1810961d0b --- /dev/null +++ b/en/latest/_sources/tutorials/upgrading-netris.rst.txt @@ -0,0 +1,255 @@ +.. meta:: + :description: Upgrading Netris + +.. raw:: html + + + + +.. role:: green + +.. role:: red + +******************** +Upgrade and Rollback +******************** + +Netris upgrade Procedure +======================== + +Backup current database +----------------------- + +Always have a backup, just in case anything hypothetically goes wrong. SSH to the host running the Netris Controller and execute below command. + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot.sql + +Ensure that SQL file ``db-snapshot.sql`` is generated and present in the current directory. + +.. note:: + + An SQL dump is enough for this basic upgrade scenario, however detailed backup & restore procedure is described in :ref:`here`. + +Stop Netris Agents +------------------ + +Stop Netris agents on switches and SoftGate nodes. + +**For Switches:** + +SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sw + +**For SoftGate nodes:** + +SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sg + +Make sure that all devices in the *Network → Inventory* section are ":red:`red`" with the "**check_agent**" status being "**Agent is unavailable**". + + +.. note:: + + A stopped Netris agent has no impact on production traffic through the device. + +.. _upgrade 3: + +Check your current version +-------------------------- + +Before upgrading the Netris Controller, take a note of the "*Netris Version*" by navigating to *Settings → General* in the Controller web interface. The current version number may be used in case of the hypothetical need to perform a rollback procedure. + +.. image:: /tutorials/images/netris_version_example.png + :align: center + :alt: Netris Version Example + +Upgrade the Controller +---------------------- + +SSH to the Controller host and execute the below command. + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh - + +.. note:: + + This process can take up to 5 minutes + + +Afterwards, make sure that all pods have either "*Running*" or "*Completed*" status by executing the following command: + +.. code-block:: shell-session + + kubectl -n netris-controller get pods + + +The output is similar to this: + +.. code-block:: shell-session + + NAME READY STATUS RESTARTS AGE + svclb-netris-controller-haproxy-6tkgj 4/4 Running 0 38d + netris-controller-haproxy-bcb944b7c-qcbf8 1/1 Running 0 13d + netris-controller-squid-7f6fdc6cf9-7fdx8 1/1 Running 0 38d + svclb-netris-controller-squid-58rnp 1/1 Running 0 38d + netris-controller-graphite-0 1/1 Running 0 38d + netris-controller-mongodb-0 1/1 Running 0 38d + netris-controller-redis-master-0 1/1 Running 0 38d + netris-controller-smtp-76778cf85f-lw5v5 1/1 Running 0 10d + netris-controller-mariadb-0 1/1 Running 0 10d + netris-controller-web-session-generator-8b9dbbcd8-8snhd 1/1 Running 0 10d + netris-controller-telescope-notifier-647975848f-fs5dn 1/1 Running 0 10d + netris-controller-app-b9b8d8f8d-4ssqb 1/1 Running 0 10d + netris-controller-grpc-987669fb9-jjskp 1/1 Running 0 10d + netris-controller-telescope-777c98c5d9-mqwl6 1/1 Running 0 10d + helm-install-netris-controller-lqmq7 0/1 Completed 0 20h + + +.. warning:: + + If, after 5 minutes, you see pods with a status other than "*Running*" or "*Completed*", please reach out to us via `Slack `__. + +Check the upgraded version +-------------------------- + +Make sure that the "*Netris Version*" reflects the version change by navigating to *Settings → General* in the Controller web interface. + +Upgrade Switches and SoftGate nodes +----------------------------------- + +Once you have verified that the Netris controller is up-to-date, it is time to update the switch and SoftGate agents. + +Upgrade the switch & SoftGate agents by copying the one-liner from the "*Install Agent*" option of the device’s 3-dot menu found under the *Network → Inventory* section and pasting it into appropriate devices by SSHing to the corresponding device. + +.. note:: + + These one-liners include a unique identifier for binding the physical device with the virtual object in the Controller. Please make sure + to copy/paste into the right devices. + + +.. image:: /tutorials/images/install_agent.gif + :align: center + :alt: Install Agent + +After all the agents have finished the upgrade process, make sure all devices in the *Network → Inventory* section have a ":green:`green`" status and the *Netris version* for each device reflects the version change. + +In the event the "**check_agent**" status is "**Agent is unavailable**" after the agent upgrade has finished, perform agent restart on the affected device(s). + +**For Switches:** + +SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sw + +**For SoftGate nodes:** + +SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sg + +Rollback Procedure +================== + +A rollback procedure can be executed in the event the upgrade introduces any adverse impact on the production traffic. + +Stop Netris Agents +------------------ + +Stop all Netris agents on the devices managed by the controller (switch & SoftGate). + +**For Switches:** + +SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sw + +**For SoftGate nodes:** + +SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl stop netris-sg + +Restore The Database +-------------------- + +Restore the database from the previously taken snapshot. + +Drop the current database and create a new one by running the following command after SSHing to the Controller: + +.. code-block:: shell-session + + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"' + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"' + +While still connected to the Controller, copy the backup file from the controller host system to the MariaDB container and restore the database: + +.. code-block:: shell-session + + kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql + kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql' + +Downgrade the Controller Software +--------------------------------- + +Downgrade Netris Controller application with the following command. + +.. note:: + + For the version number, use the number collected from :ref:`step #3` during the upgrade procedure. + +Example: + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-version 3.0.10-031 + +Afterwards, verify that the version of the "*Netris Version*" reflects the downgraded version by navigating to *Settings → General* in the Netris Controller. + +Downgrade Netris Agent Software +------------------------------- + +Once you have verified that the Netris controller has been downgraded to the correct version, it is time to downgrade the switch and SoftGate agents. + +Install the appropriate version of switch & SoftGate agents by copying the one-liner from the "*Install Agent*" option of the device’s 3-dot menu found under the *Network → Inventory* section and pasting it into appropriate devices by SSHing to the corresponding device. + +.. note:: + + One-liners include a unique identifier for binding the physical device with the virtual object in the Controller. Please make sure to copy/paste into the right devices. + + +After all the switches and SoftGates have been successfully downgraded, make sure all the devices in the *Network → Inventory* section have a ":green:`green`" status and the *Netris version* for each device reflects the version downgrade. + +In case the "**check_agent**" status is "**Agent is unavailable**" after agent downgrade, perform agent restart. + +**For Switches:** + +SSH to the switch and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sw + +**For SoftGate nodes:** + +SSH to the SoftGate and run the following command: + +.. code-block:: shell-session + + sudo systemctl restart netris-sg diff --git a/en/latest/_sources/tutorials/upgrading-sonic-os.rst.txt b/en/latest/_sources/tutorials/upgrading-sonic-os.rst.txt new file mode 100644 index 0000000000..94e9ac04e5 --- /dev/null +++ b/en/latest/_sources/tutorials/upgrading-sonic-os.rst.txt @@ -0,0 +1,63 @@ +.. meta:: + :description: Upgrading SONiC OS + +.. raw:: html + + + + +.. role:: green + +.. role:: red + +************************************** +Upgrading SONiC Operating System +************************************** + +Upgrade Procedure +================= + +The SONiC OS image is being distributed by the Vendor of the switch, you need to obtain one via the support page of the vendor. +The image should be placed on a web server to which switch(es) have access. +In this example we will use a server with an IP address of *10.0.0.36*, the image is located in the */var/www/html/* folder, which is the root of the web server. + +.. image:: /tutorials/images/upgrading_sonic_folder_listing.png + :align: center + +.. hint:: To make sure that the image is available as expected, please try to download the image with browser. + +The upgrade process consists of the following steps: + +1. Login to the switch and execute + +.. code-block:: shell-session + + sudo sonic-installer install -y http://10.0.0.36/Edgecore-SONiC_20220929_052156_ec202012_420.bin + +The process will take some time, after installation complete, reboot is mandatory. + +.. warning:: + + The installation will wipe all the data on the switch, including the admin user password, the authorized keys and ssh identity. After the reboot, a new ssh identity is generated. Because of that, you will be prompted with an identity mismatch message on the first login attempt. Please use your OS-specific procedure to remove the old key. + +2. Login to the switch and verify that the OS version is updated + +.. code-block:: shell-session + + admin@switch15:~$ show version + + SONiC Software Version: SONiC.Edgecore-SONiC_20220929_052156_ec202012_420 + Distribution: Debian 10.13 + Kernel: 4.19.0-12-2-amd64 + Build commit: 895d178f6 + Build date: Thu Sep 29 05:49:16 UTC 2022 + Built by: ubuntu@ip-10-5-1-225 + + Platform: x86_64-accton_as7326_56x-r0 + HwSKU: Accton-AS7326-56X + ASIC: broadcom + ASIC Count: 1 + Serial Number: REDACTED + Uptime: 00:06:00, 1 user, load average: 1.39, 2.08, 2.29 + +3. Perform Netris agent installation step 4 from :ref:`"Install the Netris Agent"` tutorial. diff --git a/en/latest/_sources/tutorials/using-l4-load-balancer.rst.txt b/en/latest/_sources/tutorials/using-l4-load-balancer.rst.txt new file mode 100644 index 0000000000..40b7477c34 --- /dev/null +++ b/en/latest/_sources/tutorials/using-l4-load-balancer.rst.txt @@ -0,0 +1,13 @@ +################################################## +Using on-demand (elastic) L4 Load Balancer service +################################################## + +Services --> L4 Load Balancer is an on-demand (elastic) server load balancer. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console. + +.. image:: /tutorials/images/netris-l4-load-balancer.png + :align: center + diff --git a/en/latest/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt b/en/latest/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt new file mode 100644 index 0000000000..1a7e4fbb21 --- /dev/null +++ b/en/latest/_sources/tutorials/using-vnet-in-equinix-metal-project.rst.txt @@ -0,0 +1,51 @@ +######################################################################## +Using V-Net (isolated virtual network) services in Equinix Metal Project +######################################################################## + +V-Net, under Services --> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from Equinix Metal Project into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network. + +Netris V-Net (in Equinix Metal scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Layer-2 network using Equinix Metal API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Layer-2 network. SoftGate nodes are the default gateway for the V-Net services. + +You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, Equinix, and your Compute. + +.. image:: /tutorials/images/netris-creating-vnet-for-equinix-metal.png + :align: center + +In this example, the new V-NET has VLAN ID 2, subnet 10.0.0.0/24, and gateway 10.0.0.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 2, and they should use IP addresses from 10.0.0.2-254 pointing to 10.0.0.1 as the default gateway. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 2 will have Internet access over the NAT. + +.. image:: /tutorials/images/netris-vnet-ready-in-equinix-metal.png + :align: center + +Tags +==== + +Starting from Netris `v3.3.0 `_, it is possible to dynamically associate bare-metal servers with V-NET. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers to the Layer-2 network based on that tag. Due to this, you can make flexible V-NETs, and there is no need to include every new server in the V-NET. + +.. image:: /tutorials/images/equinix-metal-vnet-with-tag.png + :align: center + +This feature is even more efficient when you build your infrastructure via Terraform. For example, let's say you've created a V-NET with a tag using Netris Terraform Provider, then order several metal servers with the same tag using Equinix Metal Terraform Provider. And that's it, when the servers are ready, Netris will detect them and make them part of the V-NET. + +.. image:: /tutorials/images/equinix-metal-vnet-with-tag-terraform.png + :align: center + +Unmanaged +========= + +Another option is turning the existing Layer-2 network (VLAN) into Netris V-Net. All VLANs in the particular project that aren't used in other services, like an E-BGP, are visible as "unmanaged" in the V-Net section. + +.. image:: /tutorials/images/unmanaged-vlan-equinix.png + :align: center +.. image:: /tutorials/images/unmanaged-vnet.png + :align: center + +The "manage" button will open a dialogue window where it's also possible to add a default gateway for the appropriate VLAN. + + +.. warning:: + Once the VLAN is being converted into V-Net, it will be managed by Netris and no longer manageable through Equinix Metal console. + +.. image:: /tutorials/images/manage-vnet.gif + :align: center + +Note that you can use Services --> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites) diff --git a/en/latest/_sources/tutorials/vpc-anywhere-check-default-site.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-check-default-site.rst.txt new file mode 100644 index 0000000000..9af39953a5 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-check-default-site.rst.txt @@ -0,0 +1,16 @@ +########################### +Check Default Site Settings +########################### + + +The VLAN ID range (under Net-->Site-->Default) defines what VLAN IDs Netris can use. You may want to ensure that this range does not overlap with other VLAN IDs you may have in your network. + + +.. image:: images/vpc-anywhere-check-site-default.png + :align: center + +The default value is 700-900. Feel free to change to any range your network administrators are comfortable. + +.. image:: images/vpc-anywhere-edit-vlan-range-default-site.png + :align: center + diff --git a/en/latest/_sources/tutorials/vpc-anywhere-concept.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-concept.rst.txt new file mode 100644 index 0000000000..ebcf0e4598 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-concept.rst.txt @@ -0,0 +1,25 @@ +######################## +VPC Anywhere Overview +######################## + +Introduction +------------- + +VPC Anywhere is a solution designed by Netris that allows the integration of VPC network functionality into any network. Netris VPC can be used for all or part of the network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures. + + +Concept +--------- + +SoftGate is a highly optimized automatic Linux gateway that communicates with Netris VPC controller through an encrypted management link. + +A SoftGate node (typically two of them for redundancy) connects to the physical network on an 802.1q trunk port (similar to your Vmware or KVM hypervisors). + +Netris needs to know what VLAN IDs it is allowed to utilize. We recommend reserving a range of VLAN IDs dedicated to Netris VPC. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the network. + +SoftGate node becomes the default gateway for the workloads consuming the VPC network. Your existing switch network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers. + + +.. image:: images/vpc-anywhere-solution-traffic-flows.png + :align: center + diff --git a/en/latest/_sources/tutorials/vpc-anywhere-controller-installation.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-controller-installation.rst.txt new file mode 100644 index 0000000000..9da791d671 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-controller-installation.rst.txt @@ -0,0 +1,86 @@ +.. meta:: + :description: Installing a Netris Controller + +============================== +Install a Netris Controller +============================== + +Requirements and Installation steps +----------------------------------- + +You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact, if there are multiple Netris managed deployments there's no need for an individual controller for each deployment. + +It doesn't matter where to host the Netris controller. What does matter is that the Netris controller needs to be accessible over the Internet. 1) So you can access the web console. 2) Nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface. + +**Linux Host requirements** + +* RAM: 8 GB +* CPU: 4 Cores +* Disk: 50GB +* OS: Linux 64-bit + +**DNS record** + +In my example, my host got a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address). + +Below is an example using Cloudflare DNS service. (same idea with any DNS software or service) + +.. image:: images/dns-record-netrisctl.png + :align: center + +Ensure that the newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller. + +.. code-block:: shell-session + + host netrisctl.netris.dev + netrisctl.netris.dev has address 54.183.23.201 + +**Install Netris Controller software and dependencies** + +.. code-block:: shell-session + + curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt + +.. note:: + Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “--ctl-ssl-issuer” will instruct the installer to generate a Let's Encrypt SSL certificate and the "--ctl-hostname" will hint for what domain name the certificate must be generated. That's why it is important to create the DNS record before this step. Detailed info here: `doc `_. + +.. image:: images/netris-controller-installed.png + :align: center + +Once the installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials. + +Security Matters +---------------- + +**Change the default password** + +Setting → My Account → Change Password + +.. image:: images/change-password.png + :align: center + +**Add new admin user** + +Accounts → Users → +Add + +.. image:: images/create-new-admin-user.png + :align: center + +**Restrict incoming TCP requests to the list below:** + ++----------+--------------------------------+ +| TCP Port | Service | ++==========+================================+ +| 22 | SSH | ++----------+--------------------------------+ +| 80 | HTTP | ++----------+--------------------------------+ +| 443 | Netris Web Console | ++----------+--------------------------------+ +| 2003 | Streaming Telemetry (Collectd) | ++----------+--------------------------------+ +| 3033 | Netris Monitoring (Telescope) | ++----------+--------------------------------+ +| 50051 | Netris Agent (gRPC) | ++----------+--------------------------------+ + diff --git a/en/latest/_sources/tutorials/vpc-anywhere-ipam-setup.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-ipam-setup.rst.txt new file mode 100644 index 0000000000..926e261992 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-ipam-setup.rst.txt @@ -0,0 +1,46 @@ +.. meta:: + :description: IPAM Setup for Services + +####################### +IPAM Setup for Services +####################### + +To take advantage of the services provided by Netris, it is imperative to define Allocations/Subnets within the Netris IPAM. The Netris Controller comes with predefined subnets that serve the purpose of "common" and these subnets can be used to create V-Nets. However, to leverage services like NAT/L4 (elastic) Load Balancer, it is essential to create new Allocations and Subnets that align with the appropriate purpose. Alternatively, if an existing predefined subnet satisfies the requirements of your network architecture, it is possible to reconfigure it with the desired purpose. + +Create an allocation +==================== + +In addition to the predefined subnets, the Netris Controller also includes predefined allocations, consisting of private IP addresses defined in RFC 1918 - ``10.0.0.0/8``, ``172.16.0.0/12``, and ``192.168.0.0/16``. If you intend to create subnets that fall outside of these predefined allocations, you must first create allocations that encompass those subnets. + + +.. image:: /tutorials/images/vpc-anywhere-ipam-allocation.png + :align: center + + +Enable on-demand L4 (elastic) Load Balancer +=========================================== + +Once the allocation is created, you can proceed with creating a subnet with the purpose of "load-balancer". + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/vpc-anywhere-ipam-l4lb-subnet.png + :align: center + +The on-demand L4 (elastic) Load Balancer service has been successfully activated and can be accessed through either the web console or Kubernetes using a service of the type load-balancer, or with Terraform. + +It is worth noting that in the example provided, the Tenant field was left at its default setting of "Admin." Tenancy is a feature that enables role-based access control and resource delegation. To illustrate, you may want to establish a user role and tenant for your colleagues who are meant to consume Netris VPC services but are not authorized to administer them. + +.. _vpc_anywhere_ipam_nat: + +Enable Network Address Translation (NAT) +======================================== + +To enable NAT, you need to create a subnet with the purpose of "nat". + +Click the **+ Add** button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu. + +.. image:: /tutorials/images/vpc-anywhere-ipam-nat-subnet.png + :align: center + +The NAT service has been successfully activated. To create a NAT rule, navigate to the Net → NAT section of the Netris web console. Netris supports most standard SNAT and DNAT rules. diff --git a/en/latest/_sources/tutorials/vpc-anywhere-softgate-installation.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-softgate-installation.rst.txt new file mode 100644 index 0000000000..7179c4965e --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-softgate-installation.rst.txt @@ -0,0 +1,90 @@ +.. meta:: + :description: Netris SoftGate Installation + +.. _softgate-installation-vpc_def: + +*************************** +SoftGate Installation +*************************** + +Minimum Hardware Requirements +============================= +* 8 CPU cores +* 16 GB RAM +* 300 GB HDD + + +SoftGate software provisioning +============================== +The SoftGate deployment needs a freshly installed Ubuntu Linux 22.04 LTS. + +.. note:: + Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: :ref:`"Adding SoftGates"`. + +1) Navigate to the **Net-->Inventory** section and click the **three vertical dots (⋮)** on the right side of the SoftGate node you are provisioning. Then click **Install Agent** and copy the one-line installer command to your clipboard. + +.. image:: /images/softgate-install-agent.png + :align: center + +2) Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node) + +.. image:: /images/softgate-provisioning-cli-output.png + :align: center + +.. note:: + Please note that the Netris installation script replaces the default Netplan networking backend with regular ifupdown and attempts to migrate the configuration set during installation to /etc/network/interfaces. + +3) Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as described in /etc/network/interfaces. + +.. code-block:: shell-session + + sudo vim /etc/network/interfaces + +.. code-block:: shell-session + + # The loopback network interface + auto lo + iface lo inet loopback + + # The management network interface + auto ensZ + iface ensZ inet static + address + # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node, otherwise adjust the line + # according to your setup. + up ip route add via + + # Physical port on SoftGate node connected to a TRUNK port of your network + auto ens + iface ens inet static + address 0.0.0.0/0 + + # Optionally you can add more physical interfaces under your bond0, uncomment as needed + #auto ens + #iface ens inet static + # address 0.0.0.0/0 + + # Bond interface + auto bond0 + iface bond0 inet static + address 0.0.0.0/0 + # Please replace/remove the ensX/Y with actual interface name(s) below to one(s) present in the OS. + bond-slaves ens + # Optional, please adjust the bonding mode below according to the desired functionality. + bond-mode active-backup + + source /etc/network/interfaces.d/* + +.. note:: + Ensure that the Management network interface IP address is as expected so the SoftGate node will maintain IP connectivity with Netris Controller after reboot. + +4) Reboot the SoftGate + +.. code-block:: shell-session + + sudo reboot + +Once the server boots up, you should see its heartbeat going from Critical to OK in **Net→Inventory**, **Telescope→Dashboard**, and the SoftGate color will reflect its health in **Net→Topology**. + +.. image:: images/vpc-anywhere-softgates-green.png + :align: center diff --git a/en/latest/_sources/tutorials/vpc-anywhere-upstream-peering.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-upstream-peering.rst.txt new file mode 100644 index 0000000000..0aee7f765a --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-upstream-peering.rst.txt @@ -0,0 +1,65 @@ +.. meta:: + :description: Netris VPC anywhere upstream peering options + +************************************************************** +Connecting VPC to upstream networks (use one of two options) +************************************************************** + +Using a VLAN with public IP addresses (DMZ) +=========================================== +Netris VPC can use a traditional subnet or vlan with public IP addresses (a DMZ network) for upstream network peering. + +**logical diagram** + +.. image:: images/upstream-dmz-logical.png + +**physical diagram** + +.. image:: images/upstream-dmz-physical.png + +BGP Upstream +============ +BGP Upstream: overview +---------------------- +A BGP peering between Netris VPC and upstream routers is the most recommended option, especially for production use. + +In this scenario, SoftGate nodes form BGP peering with upstream routers. Typically each BGP session will use an individual VLAN id and /30 subnet. Upstream BGP router should advertise the default route 0.0.0.0/0 or full Internet table if necessary. Netris SoftGate nodes are designed to handle over 1M routes in the routing table and can perform as border routers for the full-view table. +SoftGate nodes will automatically advertise public IP subnets in the current site as defined under the Net->IPAM section. Alternatively, you can optionally alter the default settings for full granular control over sent and received prefixes. + +**Logical diagram** + +.. image:: images/vpc-anywhere-upstream-bgp-router-logical.png + :align: center + + +**Physical diagram** + +.. image:: images/vpc-anywhere-upstream-bgp-router-physical.png + :align: center + + +BGP Upstream: configuration +--------------------------- + +Your local public AS number is a site attribute located under the Net->Sites section. You can use the default private AS number or change to a real AS number depending on local network architecture needs. + +.. image:: images/local-public-asn.png + :align: center + +BGP peers can be defined under the Net->E-BGP section. + +.. image:: images/add-new-ebgp-form.png + + +Check BGP neighbor statuses under the Net->E-BGP + +.. image:: images/bgp-listing.png + + +To check/trobleshoot BGP use Net->Looking Glass + +.. image:: images/bgp-looking-glass.png + + + + diff --git a/en/latest/_sources/tutorials/vpc-anywhere-using-l4lb.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-using-l4lb.rst.txt new file mode 100644 index 0000000000..00bf7c4497 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-using-l4lb.rst.txt @@ -0,0 +1,16 @@ +.. meta:: + :description: Using on-demand (elastic) L4 Load Balancer service + +################################################## +Using on-demand (elastic) L4 Load Balancer service +################################################## + +Services --> L4 Load Balancer is an on-demand (elastic) L4 Load Balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service. + +Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform. + +Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console. + +.. image:: /tutorials/images/vpc-anywhere-l4lb.png + :align: center + diff --git a/en/latest/_sources/tutorials/vpc-anywhere-using-multi-interface-softgate.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-using-multi-interface-softgate.rst.txt new file mode 100644 index 0000000000..fbf3dfb65a --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-using-multi-interface-softgate.rst.txt @@ -0,0 +1,35 @@ +.. meta:: + :description: Using Multi-interface SoftGate + +############################## +Using Multi-interface SoftGate +############################## + +By default, Netris uses the bond0 interface of SoftGates exclusively. For each V-NET, Netris creates a new subinterface (e.g., bond0.700) with the next available VLAN ID from the :doc:`Site's defined VLAN ID range `. However, a proposed solution aims to modify this behavior. + + + +Creating a V-Net +================ + +If you want to create a V-Net but prefer Netris to use existing interfaces instead of creating a new subinterface on the bond0 interface, you can achieve that by using the "int=" tag. + +example - ``int=eth1`` + +.. image:: /tutorials/images/vpc-anywhere-vnet-experimental.png + :align: center + + +SoftGate to SoftGate Link +========================= + +In addition to creating a subinterface on the bond0 interface for each V-NET, Netris also creates a separate subinterface for SoftGates' internal use. This subinterface enables two SoftGates to communicate and synchronize mission-critical information. By default, Netris uses the last VLAN ID from the :doc:`Site's defined VLAN ID range ` for this subinterface (e.g., bond0.900). + +To change this behavior and use an existing interface instead of creating a new subinterface, navigate to the **Net → Inventory** section of the Netris web console. Click the **three vertical dots (⋮)** on the right side of the SoftGate node and select **Edit**. In the **Description** field of the opened window, type "int=" and save the changes. + +example - ``int=eth2`` + +.. image:: /tutorials/images/vpc-anywhere-sg-to-sg-experimental.png + :align: center + +Repeat this action for the second SoftGate. diff --git a/en/latest/_sources/tutorials/vpc-anywhere-using-nat.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-using-nat.rst.txt new file mode 100644 index 0000000000..83b267eba4 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-using-nat.rst.txt @@ -0,0 +1,41 @@ +.. meta:: + :description: Using NAT services + +################## +Using NAT services +################## + +NAT, under Net --> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 172.24.0.0/16 to have access to services outside their VPC. The goal is to provide instances with the capability to connect to the Internet through NAT for outbound access. + +MASQUERADE +========== + +The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn't require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT). + +The Netris Controller includes a preconfigured MASQUERADE rule that facilitates instances within a private network with a subnet of 172.24.0.0/16 to have immediate access to services outside their VPC environment. + +.. image:: images/vpc-anywhere-nat-masquerade.png + :align: center + + +SNAT +==== + +Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a :ref:`NAT purpose`. SNAT replaces the source IP address of the instances with the IP address of the "SNAT to IP". Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity. + +.. image:: images/vpc-anywhere-nat-snat.png + :align: center + + +You can always have more granular control either through NAT rule or using Services → ACLs. + + +DNAT +==== + +A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a "DNAT to IP/Port". This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database. + +I'm creating a DNAT rule for the ssh port in the example below. It forwards the public IP's 55022 port to the local IP's 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server. + +.. image:: images/vpc-anywhere-nat-dnat.png + :align: center diff --git a/en/latest/_sources/tutorials/vpc-anywhere-using-vnet.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere-using-vnet.rst.txt new file mode 100644 index 0000000000..1aeb0c9b46 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere-using-vnet.rst.txt @@ -0,0 +1,22 @@ +.. meta:: + :description: Using V-Net (isolated virtual network) services + +############################################### +Using V-Net (isolated virtual network) services +############################################### + +V-Net, under Services --> The V-Net service is a virtual network that operates in an isolated environment. In Netris, each V-Net takes the next available VLAN ID from the :doc:`Site's defined VLAN ID range ` and uses it as a global VLAN ID. The SoftGate nodes serve as the default gateway for the V-Net services. + + + +Creating a V-Net +================ + +To ensure proper connectivity, it is recommended to create a corresponding V-Net for each virtual network used in Vmware, KVM, or any other server virtualization platform. The same applies to an isolated group of physical servers. VLAN ID serves as the unique identifier between Netris and the computing platform used. + +.. image:: /tutorials/images/vpc-anywhere-vnet.png + :align: center + +In this example, the new V-NET is assigned VLAN ID 700, subnet 172.24.0.0/20, and gateway 172.24.0.1. As a result, on your Compute side, you can launch VMs or subinterfaces on physical servers with a VLAN ID of 700 and obtain all necessary IP configurations from DHCP. Alternatively, if you prefer not to use DHCP, instances should use IP addresses from the 172.24.0.0/20 subnet and set the default gateway to 172.24.0.1. Netris SoftGate will manage this traffic, and the preconfigured MASQUERADE rule for the 172.24.0.0/16 subnet included in the Netris Controller will enable hosts in VLAN 700 to have internet access through NAT. + +Note that you can use Services --> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites) diff --git a/en/latest/_sources/tutorials/vpc-anywhere.rst.txt b/en/latest/_sources/tutorials/vpc-anywhere.rst.txt new file mode 100644 index 0000000000..2e85e99bd4 --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-anywhere.rst.txt @@ -0,0 +1,28 @@ +================================== +VPC Anywhere Getting Started Guide +================================== + + +.. toctree:: + :maxdepth: 2 + + vpc-anywhere-concept + +.. toctree:: + :maxdepth: 2 + :caption: Setup + + vpc-anywhere-controller-installation + vpc-anywhere-check-default-site + vpc-anywhere-softgate-installation + vpc-anywhere-upstream-peering + vpc-anywhere-ipam-setup + +.. toctree:: + :maxdepth: 2 + :caption: Use + + vpc-anywhere-using-nat + vpc-anywhere-using-vnet + vpc-anywhere-using-l4lb + vpc-anywhere-using-multi-interface-softgate diff --git a/en/latest/_sources/tutorials/vpc-gateways-with-managed-fabric.rst.txt b/en/latest/_sources/tutorials/vpc-gateways-with-managed-fabric.rst.txt new file mode 100644 index 0000000000..651149d46b --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-gateways-with-managed-fabric.rst.txt @@ -0,0 +1,75 @@ +================================================ +Getting Started with Switch-Fabric Manager & VPC +================================================ + +.. toctree:: + :maxdepth: 2 + + netris-managed-fabric-overview + + +.. toctree:: + :maxdepth: 2 + + netris-controller-installation + +.. toctree:: + :maxdepth: 2 + + new-site-setup + +.. toctree:: + :maxdepth: 2 + + vpc-setup + +.. toctree:: + :maxdepth: 2 + + ipam-setup + +.. toctree:: + :maxdepth: 2 + + inventory-setup + +.. toctree:: + :maxdepth: 2 + + topology-setup + + +.. toctree:: + :maxdepth: 2 + + softgate-software-provisioning + +.. toctree:: + :maxdepth: 2 + + netris-switch-agent-installation + +.. toctree:: + :maxdepth: 2 + + connecting-fabric-to-isp + +.. toctree:: + :maxdepth: 2 + + connecting-servers-fabric + +.. toctree:: + :maxdepth: 2 + + enabling-nat-services + +.. toctree:: + :maxdepth: 2 + + enabling-load-balancing-services + +.. toctree:: + :maxdepth: 2 + + more-features diff --git a/en/latest/_sources/tutorials/vpc-setup.rst.txt b/en/latest/_sources/tutorials/vpc-setup.rst.txt new file mode 100644 index 0000000000..ec4ee8fcaf --- /dev/null +++ b/en/latest/_sources/tutorials/vpc-setup.rst.txt @@ -0,0 +1,15 @@ +######### +VPC setup +######### + +The Netris VPC offers you the ability to operate your resources within a logically segregated virtual network. You can create, edit, and remove VPCs as needed. The VPC acts as a VRF in traditional networking, providing the ability to employ overlapping IP ranges across various VPCs while maintaining secure management and operation of resources. + +Netris Controller is preconfigured with a default system VPC-1. Use the default VPC, and create additional VPCs as needed in the future. + +.. image:: images/vpc_concept.png + :align: center + +To create VPC go to Network → VPC → +Add + +.. image:: images/vpc_add.png + :align: center diff --git a/en/latest/_sources/visibility.rst.txt b/en/latest/_sources/visibility.rst.txt new file mode 100644 index 0000000000..cbe72d72e5 --- /dev/null +++ b/en/latest/_sources/visibility.rst.txt @@ -0,0 +1,86 @@ +.. meta:: + :description: Netris System Visibility, Monitoring & Telemetry + +********************** +Visibility (Telescope) +********************** + +Graph Boards +============ + +You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view. + +To start with Graph Boards, first, you need to add a new Graph Board. + +1. Navigate to Telescope → Graph Boards, open the dropdown menu in the top left corner, then click + button. + +.. image:: images/add_board.png + :align: center + +2. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants. + +.. image:: images/create_board.png + :align: center + +Now you can add graphs by clicking +Add graph. + +Description of +Add graph fields: + +- **Title** - Title for the new graph. +- **Type** - Type of data source. + + - Bps - Traffic bits per second. + - Pps - Traffic packets per second. + - Errors - Errors per second. + - Optical - Optical signal statistics/history. + - MAC Count - History of the number of MAC addresses on the port. +- **Function** - Currently, only summing is supported. +- **+Member** - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port. + +Example: Sum of traffic on two ISP uplinks. + +.. image:: images/add_graph.png + :align: center + +Example: Sum of the traffic on all ports under the service called “PC06 V-NET”. + +.. image:: images/add_graph2.png + :align: center + +Screenshot: Listing of a Graph Board with the explanation of the controls. + +.. image:: images/graph_board.png + :align: center + +API Logs +======== +Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type. + +Dashboard +========= +Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems. + +Telescope → Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner. + + +Description of the pie charts. + +* **Hardware Health** - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions. +* **E-BGP** - Statuses of external BGP sessions. +* **LB VIP** - Statuses of Load Balancer frontend / VIP availability. +* **LB Members** - Statuses of Load Balancer backend members. + +By clicking on each title you can see the details of the checks on the right side. + +Screenshot: Dashboard showing details of “Hardware Health.” + +.. image:: images/hardware_health.png + :align: center + +Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state. + +Screenshot: “Save as normal” on selected ports. + +.. image:: images/saveasnormal.png + :align: center + diff --git a/en/latest/_sources/vnet.rst.txt b/en/latest/_sources/vnet.rst.txt new file mode 100644 index 0000000000..99b1ceaa41 --- /dev/null +++ b/en/latest/_sources/vnet.rst.txt @@ -0,0 +1,52 @@ +.. meta:: + :description: V-Net + +.. _v-net_def: + +##### +V-Net +##### +V-Net is a virtual networking service that provides a Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or they can be created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization). +Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes. + + +V-Net Fields +============ + +- **Name** - Unique name for the V-Net +- **Owner** - Tenant, who can make any changes to current V-Net +- **V-Net state** - Active/Disable state for entire V-Net +- **VLAN aware** - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible +- **Guest tenants** - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters +- **Sites** - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites). +- **IPv4 Gateway** - IPv4 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site where V-Net is intended to span. +- **IPv6 Gateway** - IPv6 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site or sites where V-Net is intended to span. +- **Port** - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports. + + - **Enabled** - Enable or disable individual Switch Port under current V-Net + - **Port Name** - Switch Port format: (swp)@ + - **VLAN ID / Untag** - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port's ingress/egress unless VLAN aware mode is used. + - **LAG Mode** - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes). + +.. tip:: Many switches can't autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gbps switch ports, you'll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk. + +.. image:: images/add-vnet.png + :align: center + :class: with-shadow + +.. centered:: + Example: Adding a new V-Net. + +.. image:: images/list-vnet.png + :align: center + :class: with-shadow + +.. centered:: + Example: Listing of V-Nets. + +.. image:: images/list-vnet-expanded.png + :align: center + :class: with-shadow + +.. centered:: + Example: Expanded view of a V-Net listing. \ No newline at end of file diff --git a/en/latest/_sources/vpc.rst.txt b/en/latest/_sources/vpc.rst.txt new file mode 100644 index 0000000000..05eb70cfb5 --- /dev/null +++ b/en/latest/_sources/vpc.rst.txt @@ -0,0 +1,88 @@ +.. meta:: + :description: Netris VPC + +====================== +Netris VPC +====================== + +The Netris VPC offers you the ability to operate your resources within a logically segregated virtual network. You can create, edit, and remove VPCs as needed. The VPC acts as a VRF in traditional networking, providing the flexibility to employ overlapping IP ranges across various VPCs while maintaining secure management and operation of resources. + +Netris Controller is preconfigured with a default system VPC-1. Use the default VPC, and create additional VPCs as needed in the future. + +The following diagram shows a VPC concept in the Netris Controller. + +.. image:: images/vpc_diagram.png + :align: center + :alt: VPC diagram + +VPC is the highest entity in the hierarchy and it spreads over all Sites. +Take a look at the VPC features and services. + +Sites +----- + +For each individual deployment/region, you should define a Site. All network components and resources should be associated with their respective Site and VPC. + +Physically connected sites function like an availability zone. This means that two V-Nets (Subnets) will communicate using the direct link, and even a single V-Net can span within an availability zone. Sites that are not physically connected function like regions. And Netris SiteMesh, a Wireguard-based site-to-site VPN, can be used to enable communication between these regions. + + +IPAM +---- + +You have the ability to create IP Allocations and Subnet assignments for a VPC, and these may overlap between different VPCs. A Subnet can be assigned to multiple sites if you aim to extend your V-Net across various locations. + + +V-Nets +------ + +V-Net is a virtual networking service that provides Layer-2 (unrouted) or Layer-3 (routed) virtual network segments in VPC. V-Net is assigned to one VPC and one or multiple sites. Your endpoints (servers, VMs) are connected to V-Nets. + +**Note: multisite feature requires a direct physical link between Sites.** + +External connections +-------------------- + +You can connect your VPC to ISP providers or other segments of your network using Netris E-BGP service, or statically by defining a V-Net and using Net->Routes (for natively integrated Bare Metal Cloud Providers please refer to the provider-specific tutorial, as external connections usually establish automatically + +SiteMesh connections +-------------------- + +Connect your VPCs over an automatic Wireguard-based Site-to-Site VPN across multiple Sites if you don't have a direct link between those Sites. + +**Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).** + + +NAT services +------------ +SNAT allows your endpoints to communicate with the Internet. DNAT is also available. + +**Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).** + +Load-balancing service +---------------------- + +Use L4LB service to share the load between your endpoints. + +**Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).** + + +Access lists +------------ + +ACLs provide a layer of security that acts as a firewall for controlling traffic in and out of one or more subnets. + +**Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).** + +Working with Netris VPC +----------------------- + +You have the flexibility to create and manage your VPCs using any of the interfaces listed below: + +* Web interface of the Netris Controller +* Terraform Netris provider +* Kubernetes Integration +* REST API + + + + diff --git a/en/latest/_static/_sphinx_javascript_frameworks_compat.js b/en/latest/_static/_sphinx_javascript_frameworks_compat.js new file mode 100644 index 0000000000..8549469dc2 --- /dev/null +++ b/en/latest/_static/_sphinx_javascript_frameworks_compat.js @@ -0,0 +1,134 @@ +/* + * _sphinx_javascript_frameworks_compat.js + * ~~~~~~~~~~ + * + * Compatability shim for jQuery and underscores.js. + * + * WILL BE REMOVED IN Sphinx 6.0 + * xref RemovedInSphinx60Warning + * + */ + +/** + * select a different prefix for underscore + */ +$u = _.noConflict(); + + +/** + * small helper function to urldecode strings + * + * See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/decodeURIComponent#Decoding_query_parameters_from_a_URL + */ +jQuery.urldecode = function(x) { + if (!x) { + return x + } + return decodeURIComponent(x.replace(/\+/g, ' ')); +}; + +/** + * small helper function to urlencode strings + */ +jQuery.urlencode = encodeURIComponent; + +/** + * This function returns the parsed url parameters of the + * current request. Multiple values per key are supported, + * it will always return arrays of strings for the value parts. + */ +jQuery.getQueryParameters = function(s) { + if (typeof s === 'undefined') + s = document.location.search; + var parts = s.substr(s.indexOf('?') + 1).split('&'); + var result = {}; + for (var i = 0; i < parts.length; i++) { + var tmp = parts[i].split('=', 2); + var key = jQuery.urldecode(tmp[0]); + var value = jQuery.urldecode(tmp[1]); + if (key in result) + result[key].push(value); + else + result[key] = [value]; + } + return result; +}; + +/** + * highlight a given string on a jquery object by wrapping it in + * span elements with the given class name. + */ +jQuery.fn.highlightText = function(text, className) { + function highlight(node, addItems) { + if (node.nodeType === 3) { + var val = node.nodeValue; + var pos = val.toLowerCase().indexOf(text); + if (pos >= 0 && + !jQuery(node.parentNode).hasClass(className) && + !jQuery(node.parentNode).hasClass("nohighlight")) { + var span; + var isInSVG = jQuery(node).closest("body, svg, foreignObject").is("svg"); + if (isInSVG) { + span = document.createElementNS("http://www.w3.org/2000/svg", "tspan"); + } else { + span = document.createElement("span"); + span.className = className; + } + span.appendChild(document.createTextNode(val.substr(pos, text.length))); + node.parentNode.insertBefore(span, node.parentNode.insertBefore( + document.createTextNode(val.substr(pos + text.length)), + node.nextSibling)); + node.nodeValue = val.substr(0, pos); + if (isInSVG) { + var rect = document.createElementNS("http://www.w3.org/2000/svg", "rect"); + var bbox = node.parentElement.getBBox(); + rect.x.baseVal.value = bbox.x; + rect.y.baseVal.value = bbox.y; + rect.width.baseVal.value = bbox.width; + rect.height.baseVal.value = bbox.height; + rect.setAttribute('class', className); + addItems.push({ + "parent": node.parentNode, + "target": rect}); + } + } + } + else if (!jQuery(node).is("button, select, textarea")) { + jQuery.each(node.childNodes, function() { + highlight(this, addItems); + }); + } + } + var addItems = []; + var result = this.each(function() { + highlight(this, addItems); + }); + for (var i = 0; i < addItems.length; ++i) { + jQuery(addItems[i].parent).before(addItems[i].target); + } + return result; +}; + +/* + * backward compatibility for jQuery.browser + * This will be supported until firefox bug is fixed. + */ +if (!jQuery.browser) { + jQuery.uaMatch = function(ua) { + ua = ua.toLowerCase(); + + var match = /(chrome)[ \/]([\w.]+)/.exec(ua) || + /(webkit)[ \/]([\w.]+)/.exec(ua) || + /(opera)(?:.*version|)[ \/]([\w.]+)/.exec(ua) || + /(msie) ([\w.]+)/.exec(ua) || + ua.indexOf("compatible") < 0 && /(mozilla)(?:.*? rv:([\w.]+)|)/.exec(ua) || + []; + + return { + browser: match[ 1 ] || "", + version: match[ 2 ] || "0" + }; + }; + jQuery.browser = {}; + jQuery.browser[jQuery.uaMatch(navigator.userAgent).browser] = true; +} diff --git a/en/latest/_static/basic.css b/en/latest/_static/basic.css new file mode 100644 index 0000000000..eeb0519a69 --- /dev/null +++ b/en/latest/_static/basic.css @@ -0,0 +1,899 @@ +/* + * basic.css + * ~~~~~~~~~ + * + * Sphinx stylesheet -- basic theme. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +div.section::after { + display: block; + content: ''; + clear: left; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; + word-wrap: break-word; + overflow-wrap : break-word; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar #searchbox form.search { + overflow: hidden; +} + +div.sphinxsidebar #searchbox input[type="text"] { + float: left; + width: 80%; + padding: 0.25em; + box-sizing: border-box; +} + +div.sphinxsidebar #searchbox input[type="submit"] { + float: left; + width: 20%; + border-left: none; + padding: 0.25em; + box-sizing: border-box; +} + + +img { + border: 0; + max-width: 100%; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin: 10px 0 0 20px; + padding: 0; +} + +ul.search li { + padding: 5px 0 5px 20px; + background-image: url(file.png); + background-repeat: no-repeat; + background-position: 0 7px; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li p.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; + margin-left: auto; + margin-right: auto; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable ul { + margin-top: 0; + margin-bottom: 0; + list-style-type: none; +} + +table.indextable > tbody > tr > td > ul { + padding-left: 0em; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- domain module index --------------------------------------------------- */ + +table.modindextable td { + padding: 2px; + border-collapse: collapse; +} + +/* -- general body styles --------------------------------------------------- */ + +div.body { + min-width: 360px; + max-width: 800px; +} + +div.body p, div.body dd, div.body li, div.body blockquote { + -moz-hyphens: auto; + -ms-hyphens: auto; + -webkit-hyphens: auto; + hyphens: auto; +} + +a.headerlink { + visibility: hidden; +} +a.brackets:before, +span.brackets > a:before{ + content: "["; +} + +a.brackets:after, +span.brackets > a:after { + content: "]"; +} + + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink, +caption:hover > a.headerlink, +p.caption:hover > a.headerlink, +div.code-block-caption:hover > a.headerlink { + visibility: visible; +} + +div.body p.caption { + text-align: inherit; +} + +div.body td { + text-align: left; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +img.align-left, figure.align-left, .figure.align-left, object.align-left { + clear: left; + float: left; + margin-right: 1em; +} + +img.align-right, figure.align-right, .figure.align-right, object.align-right { + clear: right; + float: right; + margin-left: 1em; +} + +img.align-center, figure.align-center, .figure.align-center, object.align-center { + display: block; + margin-left: auto; + margin-right: auto; +} + +img.align-default, figure.align-default, .figure.align-default { + display: block; + margin-left: auto; + margin-right: auto; +} + +.align-left { + text-align: left; +} + +.align-center { + text-align: center; +} + +.align-default { + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar, +aside.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px; + background-color: #ffe; + width: 40%; + float: right; + clear: right; + overflow-x: auto; +} + +p.sidebar-title { + font-weight: bold; +} +div.admonition, div.topic, blockquote { + clear: left; +} + +/* -- topics ---------------------------------------------------------------- */ +div.topic { + border: 1px solid #ccc; + padding: 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.body p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- content of sidebars/topics/admonitions -------------------------------- */ + +div.sidebar > :last-child, +aside.sidebar > :last-child, +div.topic > :last-child, +div.admonition > :last-child { + margin-bottom: 0; +} + +div.sidebar::after, +aside.sidebar::after, +div.topic::after, +div.admonition::after, +blockquote::after { + display: block; + content: ''; + clear: both; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + margin-top: 10px; + margin-bottom: 10px; + border: 0; + border-collapse: collapse; +} + +table.align-center { + margin-left: auto; + margin-right: auto; +} + +table.align-default { + margin-left: auto; + margin-right: auto; +} + +table caption span.caption-number { + font-style: italic; +} + +table caption span.caption-text { +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +th > :first-child, +td > :first-child { + margin-top: 0px; +} + +th > :last-child, +td > :last-child { + margin-bottom: 0px; +} + +/* -- figures --------------------------------------------------------------- */ + +div.figure, figure { + margin: 0.5em; + padding: 0.5em; +} + +div.figure p.caption, figcaption { + padding: 0.3em; +} + +div.figure p.caption span.caption-number, +figcaption span.caption-number { + font-style: italic; +} + +div.figure p.caption span.caption-text, +figcaption span.caption-text { +} + +/* -- field list styles ----------------------------------------------------- */ + +table.field-list td, table.field-list th { + border: 0 !important; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.field-name { + -moz-hyphens: manual; + -ms-hyphens: manual; + -webkit-hyphens: manual; + hyphens: manual; +} + +/* -- hlist styles ---------------------------------------------------------- */ + +table.hlist { + margin: 1em 0; +} + +table.hlist td { + vertical-align: top; +} + +/* -- object description styles --------------------------------------------- */ + +.sig { + font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace; +} + +.sig-name, code.descname { + background-color: transparent; + font-weight: bold; +} + +.sig-name { + font-size: 1.1em; +} + +code.descname { + font-size: 1.2em; +} + +.sig-prename, code.descclassname { + background-color: transparent; +} + +.optional { + font-size: 1.3em; +} + +.sig-paren { + font-size: larger; +} + +.sig-param.n { + font-style: italic; +} + +/* C++ specific styling */ + +.sig-inline.c-texpr, +.sig-inline.cpp-texpr { + font-family: unset; +} + +.sig.c .k, .sig.c .kt, +.sig.cpp .k, .sig.cpp .kt { + color: #0033B3; +} + +.sig.c .m, +.sig.cpp .m { + color: #1750EB; +} + +.sig.c .s, .sig.c .sc, +.sig.cpp .s, .sig.cpp .sc { + color: #067D17; +} + + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +:not(li) > ol > li:first-child > :first-child, +:not(li) > ul > li:first-child > :first-child { + margin-top: 0px; +} + +:not(li) > ol > li:last-child > :last-child, +:not(li) > ul > li:last-child > :last-child { + margin-bottom: 0px; +} + +ol.simple ol p, +ol.simple ul p, +ul.simple ol p, +ul.simple ul p { + margin-top: 0; +} + +ol.simple > li:not(:first-child) > p, +ul.simple > li:not(:first-child) > p { + margin-top: 0; +} + +ol.simple p, +ul.simple p { + margin-bottom: 0; +} +dl.footnote > dt, +dl.citation > dt { + float: left; + margin-right: 0.5em; +} + +dl.footnote > dd, +dl.citation > dd { + margin-bottom: 0em; +} + +dl.footnote > dd:after, +dl.citation > dd:after { + content: ""; + clear: both; +} + +dl.field-list { + display: grid; + grid-template-columns: fit-content(30%) auto; +} + +dl.field-list > dt { + font-weight: bold; + word-break: break-word; + padding-left: 0.5em; + padding-right: 5px; +} +dl.field-list > dt:after { + content: ":"; +} + + +dl.field-list > dd { + padding-left: 0.5em; + margin-top: 0em; + margin-left: 0em; + margin-bottom: 0em; +} + +dl { + margin-bottom: 15px; +} + +dd > :first-child { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +dl > dd:last-child, +dl > dd:last-child > :last-child { + margin-bottom: 0; +} + +dt:target, span.highlighted { + background-color: #fbe54e; +} + +rect.highlighted { + fill: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa; +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +.classifier:before { + font-style: normal; + margin: 0 0.5em; + content: ":"; + display: inline-block; +} + +abbr, acronym { + border-bottom: dotted 1px; + cursor: help; +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; + overflow-y: hidden; /* fixes display issues on Chrome browsers */ +} + +pre, div[class*="highlight-"] { + clear: both; +} + +span.pre { + -moz-hyphens: none; + -ms-hyphens: none; + -webkit-hyphens: none; + hyphens: none; + white-space: nowrap; +} + +div[class*="highlight-"] { + margin: 1em 0; +} + +td.linenos pre { + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + display: block; +} + +table.highlighttable tbody { + display: block; +} + +table.highlighttable tr { + display: flex; +} + +table.highlighttable td { + margin: 0; + padding: 0; +} + +table.highlighttable td.linenos { + padding-right: 0.5em; +} + +table.highlighttable td.code { + flex: 1; + overflow: hidden; +} + +.highlight .hll { + display: block; +} + +div.highlight pre, +table.highlighttable pre { + margin: 0; +} + +div.code-block-caption + div { + margin-top: 0; +} + +div.code-block-caption { + margin-top: 1em; + padding: 2px 5px; + font-size: small; +} + +div.code-block-caption code { + background-color: transparent; +} + +table.highlighttable td.linenos, +span.linenos, +div.highlight span.gp { /* gp: Generic.Prompt */ + user-select: none; + -webkit-user-select: text; /* Safari fallback only */ + -webkit-user-select: none; /* Chrome/Safari */ + -moz-user-select: none; /* Firefox */ + -ms-user-select: none; /* IE10+ */ +} + +div.code-block-caption span.caption-number { + padding: 0.1em 0.3em; + font-style: italic; +} + +div.code-block-caption span.caption-text { +} + +div.literal-block-wrapper { + margin: 1em 0; +} + +code.xref, a code { + background-color: transparent; + font-weight: bold; +} + +h1 code, h2 code, h3 code, h4 code, h5 code, h6 code { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.body div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +span.eqno a.headerlink { + position: absolute; + z-index: 1; +} + +div.math:hover a.headerlink { + visibility: visible; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} \ No newline at end of file diff --git a/en/latest/_static/check-solid.svg b/en/latest/_static/check-solid.svg new file mode 100644 index 0000000000..92fad4b5c0 --- /dev/null +++ b/en/latest/_static/check-solid.svg @@ -0,0 +1,4 @@ + + + + diff --git a/en/latest/_static/clipboard.min.js b/en/latest/_static/clipboard.min.js new file mode 100644 index 0000000000..54b3c46381 --- /dev/null +++ b/en/latest/_static/clipboard.min.js @@ -0,0 +1,7 @@ +/*! + * clipboard.js v2.0.8 + * https://clipboardjs.com/ + * + * Licensed MIT © Zeno Rocha + */ +!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1 + + + + diff --git a/en/latest/_static/copybutton.css b/en/latest/_static/copybutton.css new file mode 100644 index 0000000000..f1916ec7d1 --- /dev/null +++ b/en/latest/_static/copybutton.css @@ -0,0 +1,94 @@ +/* Copy buttons */ +button.copybtn { + position: absolute; + display: flex; + top: .3em; + right: .3em; + width: 1.7em; + height: 1.7em; + opacity: 0; + transition: opacity 0.3s, border .3s, background-color .3s; + user-select: none; + padding: 0; + border: none; + outline: none; + border-radius: 0.4em; + /* The colors that GitHub uses */ + border: #1b1f2426 1px solid; + background-color: #f6f8fa; + color: #57606a; +} + +button.copybtn.success { + border-color: #22863a; + color: #22863a; +} + +button.copybtn svg { + stroke: currentColor; + width: 1.5em; + height: 1.5em; + padding: 0.1em; +} + +div.highlight { + position: relative; +} + +/* Show the copybutton */ +.highlight:hover button.copybtn, button.copybtn.success { + opacity: 1; +} + +.highlight button.copybtn:hover { + background-color: rgb(235, 235, 235); +} + +.highlight button.copybtn:active { + background-color: rgb(187, 187, 187); +} + +/** + * A minimal CSS-only tooltip copied from: + * https://codepen.io/mildrenben/pen/rVBrpK + * + * To use, write HTML like the following: + * + *

Short

+ */ + .o-tooltip--left { + position: relative; + } + + .o-tooltip--left:after { + opacity: 0; + visibility: hidden; + position: absolute; + content: attr(data-tooltip); + padding: .2em; + font-size: .8em; + left: -.2em; + background: grey; + color: white; + white-space: nowrap; + z-index: 2; + border-radius: 2px; + transform: translateX(-102%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); +} + +.o-tooltip--left:hover:after { + display: block; + opacity: 1; + visibility: visible; + transform: translateX(-100%) translateY(0); + transition: opacity 0.2s cubic-bezier(0.64, 0.09, 0.08, 1), transform 0.2s cubic-bezier(0.64, 0.09, 0.08, 1); + transition-delay: .5s; +} + +/* By default the copy button shouldn't show up when printing a page */ +@media print { + button.copybtn { + display: none; + } +} diff --git a/en/latest/_static/copybutton.js b/en/latest/_static/copybutton.js new file mode 100644 index 0000000000..2ea7ff3e21 --- /dev/null +++ b/en/latest/_static/copybutton.js @@ -0,0 +1,248 @@ +// Localization support +const messages = { + 'en': { + 'copy': 'Copy', + 'copy_to_clipboard': 'Copy to clipboard', + 'copy_success': 'Copied!', + 'copy_failure': 'Failed to copy', + }, + 'es' : { + 'copy': 'Copiar', + 'copy_to_clipboard': 'Copiar al portapapeles', + 'copy_success': '¡Copiado!', + 'copy_failure': 'Error al copiar', + }, + 'de' : { + 'copy': 'Kopieren', + 'copy_to_clipboard': 'In die Zwischenablage kopieren', + 'copy_success': 'Kopiert!', + 'copy_failure': 'Fehler beim Kopieren', + }, + 'fr' : { + 'copy': 'Copier', + 'copy_to_clipboard': 'Copier dans le presse-papier', + 'copy_success': 'Copié !', + 'copy_failure': 'Échec de la copie', + }, + 'ru': { + 'copy': 'Скопировать', + 'copy_to_clipboard': 'Скопировать в буфер', + 'copy_success': 'Скопировано!', + 'copy_failure': 'Не удалось скопировать', + }, + 'zh-CN': { + 'copy': '复制', + 'copy_to_clipboard': '复制到剪贴板', + 'copy_success': '复制成功!', + 'copy_failure': '复制失败', + }, + 'it' : { + 'copy': 'Copiare', + 'copy_to_clipboard': 'Copiato negli appunti', + 'copy_success': 'Copiato!', + 'copy_failure': 'Errore durante la copia', + } +} + +let locale = 'en' +if( document.documentElement.lang !== undefined + && messages[document.documentElement.lang] !== undefined ) { + locale = document.documentElement.lang +} + +let doc_url_root = DOCUMENTATION_OPTIONS.URL_ROOT; +if (doc_url_root == '#') { + doc_url_root = ''; +} + +/** + * SVG files for our copy buttons + */ +let iconCheck = ` + ${messages[locale]['copy_success']} + + +` + +// If the user specified their own SVG use that, otherwise use the default +let iconCopy = ``; +if (!iconCopy) { + iconCopy = ` + ${messages[locale]['copy_to_clipboard']} + + + +` +} + +/** + * Set up copy/paste for code blocks + */ + +const runWhenDOMLoaded = cb => { + if (document.readyState != 'loading') { + cb() + } else if (document.addEventListener) { + document.addEventListener('DOMContentLoaded', cb) + } else { + document.attachEvent('onreadystatechange', function() { + if (document.readyState == 'complete') cb() + }) + } +} + +const codeCellId = index => `codecell${index}` + +// Clears selected text since ClipboardJS will select the text when copying +const clearSelection = () => { + if (window.getSelection) { + window.getSelection().removeAllRanges() + } else if (document.selection) { + document.selection.empty() + } +} + +// Changes tooltip text for a moment, then changes it back +// We want the timeout of our `success` class to be a bit shorter than the +// tooltip and icon change, so that we can hide the icon before changing back. +var timeoutIcon = 2000; +var timeoutSuccessClass = 1500; + +const temporarilyChangeTooltip = (el, oldText, newText) => { + el.setAttribute('data-tooltip', newText) + el.classList.add('success') + // Remove success a little bit sooner than we change the tooltip + // So that we can use CSS to hide the copybutton first + setTimeout(() => el.classList.remove('success'), timeoutSuccessClass) + setTimeout(() => el.setAttribute('data-tooltip', oldText), timeoutIcon) +} + +// Changes the copy button icon for two seconds, then changes it back +const temporarilyChangeIcon = (el) => { + el.innerHTML = iconCheck; + setTimeout(() => {el.innerHTML = iconCopy}, timeoutIcon) +} + +const addCopyButtonToCodeCells = () => { + // If ClipboardJS hasn't loaded, wait a bit and try again. This + // happens because we load ClipboardJS asynchronously. + if (window.ClipboardJS === undefined) { + setTimeout(addCopyButtonToCodeCells, 250) + return + } + + // Add copybuttons to all of our code cells + const COPYBUTTON_SELECTOR = 'div.highlight pre'; + const codeCells = document.querySelectorAll(COPYBUTTON_SELECTOR) + codeCells.forEach((codeCell, index) => { + const id = codeCellId(index) + codeCell.setAttribute('id', id) + + const clipboardButton = id => + `` + codeCell.insertAdjacentHTML('afterend', clipboardButton(id)) + }) + +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} + + +var copyTargetText = (trigger) => { + var target = document.querySelector(trigger.attributes['data-clipboard-target'].value); + + // get filtered text + let exclude = '.linenos'; + + let text = filterText(target, exclude); + return formatCopyText(text, '', false, true, true, true, '', '') +} + + // Initialize with a callback so we can modify the text before copy + const clipboard = new ClipboardJS('.copybtn', {text: copyTargetText}) + + // Update UI with error/success messages + clipboard.on('success', event => { + clearSelection() + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_success']) + temporarilyChangeIcon(event.trigger) + }) + + clipboard.on('error', event => { + temporarilyChangeTooltip(event.trigger, messages[locale]['copy'], messages[locale]['copy_failure']) + }) +} + +runWhenDOMLoaded(addCopyButtonToCodeCells) \ No newline at end of file diff --git a/en/latest/_static/copybutton_funcs.js b/en/latest/_static/copybutton_funcs.js new file mode 100644 index 0000000000..dbe1aaad79 --- /dev/null +++ b/en/latest/_static/copybutton_funcs.js @@ -0,0 +1,73 @@ +function escapeRegExp(string) { + return string.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); // $& means the whole matched string +} + +/** + * Removes excluded text from a Node. + * + * @param {Node} target Node to filter. + * @param {string} exclude CSS selector of nodes to exclude. + * @returns {DOMString} Text from `target` with text removed. + */ +export function filterText(target, exclude) { + const clone = target.cloneNode(true); // clone as to not modify the live DOM + if (exclude) { + // remove excluded nodes + clone.querySelectorAll(exclude).forEach(node => node.remove()); + } + return clone.innerText; +} + +// Callback when a copy button is clicked. Will be passed the node that was clicked +// should then grab the text and replace pieces of text that shouldn't be used in output +export function formatCopyText(textContent, copybuttonPromptText, isRegexp = false, onlyCopyPromptLines = true, removePrompts = true, copyEmptyLines = true, lineContinuationChar = "", hereDocDelim = "") { + var regexp; + var match; + + // Do we check for line continuation characters and "HERE-documents"? + var useLineCont = !!lineContinuationChar + var useHereDoc = !!hereDocDelim + + // create regexp to capture prompt and remaining line + if (isRegexp) { + regexp = new RegExp('^(' + copybuttonPromptText + ')(.*)') + } else { + regexp = new RegExp('^(' + escapeRegExp(copybuttonPromptText) + ')(.*)') + } + + const outputLines = []; + var promptFound = false; + var gotLineCont = false; + var gotHereDoc = false; + const lineGotPrompt = []; + for (const line of textContent.split('\n')) { + match = line.match(regexp) + if (match || gotLineCont || gotHereDoc) { + promptFound = regexp.test(line) + lineGotPrompt.push(promptFound) + if (removePrompts && promptFound) { + outputLines.push(match[2]) + } else { + outputLines.push(line) + } + gotLineCont = line.endsWith(lineContinuationChar) & useLineCont + if (line.includes(hereDocDelim) & useHereDoc) + gotHereDoc = !gotHereDoc + } else if (!onlyCopyPromptLines) { + outputLines.push(line) + } else if (copyEmptyLines && line.trim() === '') { + outputLines.push(line) + } + } + + // If no lines with the prompt were found then just use original lines + if (lineGotPrompt.some(v => v === true)) { + textContent = outputLines.join('\n'); + } + + // Remove a trailing newline to avoid auto-running when pasting + if (textContent.endsWith("\n")) { + textContent = textContent.slice(0, -1) + } + return textContent +} diff --git a/en/latest/_static/css/badge_only.css b/en/latest/_static/css/badge_only.css new file mode 100644 index 0000000000..e380325bc6 --- /dev/null +++ b/en/latest/_static/css/badge_only.css @@ -0,0 +1 @@ +.fa:before{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}@font-face{font-family:FontAwesome;font-style:normal;font-weight:400;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#FontAwesome) format("svg")}.fa:before{font-family:FontAwesome;font-style:normal;font-weight:400;line-height:1}.fa:before,a .fa{text-decoration:inherit}.fa:before,a .fa,li .fa{display:inline-block}li .fa-large:before{width:1.875em}ul.fas{list-style-type:none;margin-left:2em;text-indent:-.8em}ul.fas li .fa{width:.8em}ul.fas li .fa-large:before{vertical-align:baseline}.fa-book:before,.icon-book:before{content:"\f02d"}.fa-caret-down:before,.icon-caret-down:before{content:"\f0d7"}.fa-caret-up:before,.icon-caret-up:before{content:"\f0d8"}.fa-caret-left:before,.icon-caret-left:before{content:"\f0d9"}.fa-caret-right:before,.icon-caret-right:before{content:"\f0da"}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60}.rst-versions .rst-current-version:after{clear:both;content:"";display:block}.rst-versions .rst-current-version .fa{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}} \ No newline at end of file diff --git a/en/latest/_static/css/fonts/Roboto-Slab-Bold.woff b/en/latest/_static/css/fonts/Roboto-Slab-Bold.woff new file mode 100644 index 0000000000..6cb6000018 Binary files /dev/null and b/en/latest/_static/css/fonts/Roboto-Slab-Bold.woff differ diff --git a/en/latest/_static/css/fonts/Roboto-Slab-Bold.woff2 b/en/latest/_static/css/fonts/Roboto-Slab-Bold.woff2 new file mode 100644 index 0000000000..7059e23142 Binary files /dev/null and b/en/latest/_static/css/fonts/Roboto-Slab-Bold.woff2 differ diff --git a/en/latest/_static/css/fonts/Roboto-Slab-Regular.woff b/en/latest/_static/css/fonts/Roboto-Slab-Regular.woff new file mode 100644 index 0000000000..f815f63f99 Binary files /dev/null and b/en/latest/_static/css/fonts/Roboto-Slab-Regular.woff differ diff --git a/en/latest/_static/css/fonts/Roboto-Slab-Regular.woff2 b/en/latest/_static/css/fonts/Roboto-Slab-Regular.woff2 new file mode 100644 index 0000000000..f2c76e5bda Binary files /dev/null and b/en/latest/_static/css/fonts/Roboto-Slab-Regular.woff2 differ diff --git a/en/latest/_static/css/fonts/fontawesome-webfont.eot b/en/latest/_static/css/fonts/fontawesome-webfont.eot new file mode 100644 index 0000000000..e9f60ca953 Binary files /dev/null and b/en/latest/_static/css/fonts/fontawesome-webfont.eot differ diff --git a/en/latest/_static/css/fonts/fontawesome-webfont.svg b/en/latest/_static/css/fonts/fontawesome-webfont.svg new file mode 100644 index 0000000000..855c845e53 --- /dev/null +++ b/en/latest/_static/css/fonts/fontawesome-webfont.svg @@ -0,0 +1,2671 @@ + + + + +Created by FontForge 20120731 at Mon Oct 24 17:37:40 2016 + By ,,, +Copyright Dave Gandy 2016. All rights reserved. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/en/latest/_static/css/fonts/fontawesome-webfont.ttf b/en/latest/_static/css/fonts/fontawesome-webfont.ttf new file mode 100644 index 0000000000..35acda2fa1 Binary files /dev/null and b/en/latest/_static/css/fonts/fontawesome-webfont.ttf differ diff --git a/en/latest/_static/css/fonts/fontawesome-webfont.woff b/en/latest/_static/css/fonts/fontawesome-webfont.woff new file mode 100644 index 0000000000..400014a4b0 Binary files /dev/null and b/en/latest/_static/css/fonts/fontawesome-webfont.woff differ diff --git a/en/latest/_static/css/fonts/fontawesome-webfont.woff2 b/en/latest/_static/css/fonts/fontawesome-webfont.woff2 new file mode 100644 index 0000000000..4d13fc6040 Binary files /dev/null and b/en/latest/_static/css/fonts/fontawesome-webfont.woff2 differ diff --git a/en/latest/_static/css/fonts/lato-bold-italic.woff b/en/latest/_static/css/fonts/lato-bold-italic.woff new file mode 100644 index 0000000000..88ad05b9ff Binary files /dev/null and b/en/latest/_static/css/fonts/lato-bold-italic.woff differ diff --git a/en/latest/_static/css/fonts/lato-bold-italic.woff2 b/en/latest/_static/css/fonts/lato-bold-italic.woff2 new file mode 100644 index 0000000000..c4e3d804b5 Binary files /dev/null and b/en/latest/_static/css/fonts/lato-bold-italic.woff2 differ diff --git a/en/latest/_static/css/fonts/lato-bold.woff b/en/latest/_static/css/fonts/lato-bold.woff new file mode 100644 index 0000000000..c6dff51f06 Binary files /dev/null and b/en/latest/_static/css/fonts/lato-bold.woff differ diff --git a/en/latest/_static/css/fonts/lato-bold.woff2 b/en/latest/_static/css/fonts/lato-bold.woff2 new file mode 100644 index 0000000000..bb195043cf Binary files /dev/null and b/en/latest/_static/css/fonts/lato-bold.woff2 differ diff --git a/en/latest/_static/css/fonts/lato-normal-italic.woff b/en/latest/_static/css/fonts/lato-normal-italic.woff new file mode 100644 index 0000000000..76114bc033 Binary files /dev/null and b/en/latest/_static/css/fonts/lato-normal-italic.woff differ diff --git a/en/latest/_static/css/fonts/lato-normal-italic.woff2 b/en/latest/_static/css/fonts/lato-normal-italic.woff2 new file mode 100644 index 0000000000..3404f37e2e Binary files /dev/null and b/en/latest/_static/css/fonts/lato-normal-italic.woff2 differ diff --git a/en/latest/_static/css/fonts/lato-normal.woff b/en/latest/_static/css/fonts/lato-normal.woff new file mode 100644 index 0000000000..ae1307ff5f Binary files /dev/null and b/en/latest/_static/css/fonts/lato-normal.woff differ diff --git a/en/latest/_static/css/fonts/lato-normal.woff2 b/en/latest/_static/css/fonts/lato-normal.woff2 new file mode 100644 index 0000000000..3bf9843328 Binary files /dev/null and b/en/latest/_static/css/fonts/lato-normal.woff2 differ diff --git a/en/latest/_static/css/theme.css b/en/latest/_static/css/theme.css new file mode 100644 index 0000000000..8cd4f101a9 --- /dev/null +++ b/en/latest/_static/css/theme.css @@ -0,0 +1,4 @@ +html{box-sizing:border-box}*,:after,:before{box-sizing:inherit}article,aside,details,figcaption,figure,footer,header,hgroup,nav,section{display:block}audio,canvas,video{display:inline-block;*display:inline;*zoom:1}[hidden],audio:not([controls]){display:none}*{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}html{font-size:100%;-webkit-text-size-adjust:100%;-ms-text-size-adjust:100%}body{margin:0}a:active,a:hover{outline:0}abbr[title]{border-bottom:1px dotted}b,strong{font-weight:700}blockquote{margin:0}dfn{font-style:italic}ins{background:#ff9;text-decoration:none}ins,mark{color:#000}mark{background:#ff0;font-style:italic;font-weight:700}.rst-content code,.rst-content tt,code,kbd,pre,samp{font-family:monospace,serif;_font-family:courier new,monospace;font-size:1em}pre{white-space:pre}q{quotes:none}q:after,q:before{content:"";content:none}small{font-size:85%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sup{top:-.5em}sub{bottom:-.25em}dl,ol,ul{margin:0;padding:0;list-style:none;list-style-image:none}li{list-style:none}dd{margin:0}img{border:0;-ms-interpolation-mode:bicubic;vertical-align:middle;max-width:100%}svg:not(:root){overflow:hidden}figure,form{margin:0}label{cursor:pointer}button,input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}button,input{line-height:normal}button,input[type=button],input[type=reset],input[type=submit]{cursor:pointer;-webkit-appearance:button;*overflow:visible}button[disabled],input[disabled]{cursor:default}input[type=search]{-webkit-appearance:textfield;-moz-box-sizing:content-box;-webkit-box-sizing:content-box;box-sizing:content-box}textarea{resize:vertical}table{border-collapse:collapse;border-spacing:0}td{vertical-align:top}.chromeframe{margin:.2em 0;background:#ccc;color:#000;padding:.2em 0}.ir{display:block;border:0;text-indent:-999em;overflow:hidden;background-color:transparent;background-repeat:no-repeat;text-align:left;direction:ltr;*line-height:0}.ir br{display:none}.hidden{display:none!important;visibility:hidden}.visuallyhidden{border:0;clip:rect(0 0 0 0);height:1px;margin:-1px;overflow:hidden;padding:0;position:absolute;width:1px}.visuallyhidden.focusable:active,.visuallyhidden.focusable:focus{clip:auto;height:auto;margin:0;overflow:visible;position:static;width:auto}.invisible{visibility:hidden}.relative{position:relative}big,small{font-size:100%}@media print{body,html,section{background:none!important}*{box-shadow:none!important;text-shadow:none!important;filter:none!important;-ms-filter:none!important}a,a:visited{text-decoration:underline}.ir a:after,a[href^="#"]:after,a[href^="javascript:"]:after{content:""}blockquote,pre{page-break-inside:avoid}thead{display:table-header-group}img,tr{page-break-inside:avoid}img{max-width:100%!important}@page{margin:.5cm}.rst-content .toctree-wrapper>p.caption,h2,h3,p{orphans:3;widows:3}.rst-content .toctree-wrapper>p.caption,h2,h3{page-break-after:avoid}}.btn,.fa:before,.icon:before,.rst-content .admonition,.rst-content .admonition-title:before,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .code-block-caption .headerlink:before,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-alert,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before,.wy-nav-top a,.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a,input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week],select,textarea{-webkit-font-smoothing:antialiased}.clearfix{*zoom:1}.clearfix:after,.clearfix:before{display:table;content:""}.clearfix:after{clear:both}/*! + * Font Awesome 4.7.0 by @davegandy - http://fontawesome.io - @fontawesome + * License - http://fontawesome.io/license (Font: SIL OFL 1.1, CSS: MIT License) + */@font-face{font-family:FontAwesome;src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713);src:url(fonts/fontawesome-webfont.eot?674f50d287a8c48dc19ba404d20fe713?#iefix&v=4.7.0) format("embedded-opentype"),url(fonts/fontawesome-webfont.woff2?af7ae505a9eed503f8b8e6982036873e) format("woff2"),url(fonts/fontawesome-webfont.woff?fee66e712a8a08eef5805a46892932ad) format("woff"),url(fonts/fontawesome-webfont.ttf?b06871f281fee6b241d60582ae9369b9) format("truetype"),url(fonts/fontawesome-webfont.svg?912ec66d7572ff821749319396470bde#fontawesomeregular) format("svg");font-weight:400;font-style:normal}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{display:inline-block;font:normal normal normal 14px/1 FontAwesome;font-size:inherit;text-rendering:auto;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}.fa-lg{font-size:1.33333em;line-height:.75em;vertical-align:-15%}.fa-2x{font-size:2em}.fa-3x{font-size:3em}.fa-4x{font-size:4em}.fa-5x{font-size:5em}.fa-fw{width:1.28571em;text-align:center}.fa-ul{padding-left:0;margin-left:2.14286em;list-style-type:none}.fa-ul>li{position:relative}.fa-li{position:absolute;left:-2.14286em;width:2.14286em;top:.14286em;text-align:center}.fa-li.fa-lg{left:-1.85714em}.fa-border{padding:.2em .25em .15em;border:.08em solid #eee;border-radius:.1em}.fa-pull-left{float:left}.fa-pull-right{float:right}.fa-pull-left.icon,.fa.fa-pull-left,.rst-content .code-block-caption .fa-pull-left.headerlink,.rst-content .fa-pull-left.admonition-title,.rst-content code.download span.fa-pull-left:first-child,.rst-content dl dt .fa-pull-left.headerlink,.rst-content h1 .fa-pull-left.headerlink,.rst-content h2 .fa-pull-left.headerlink,.rst-content h3 .fa-pull-left.headerlink,.rst-content h4 .fa-pull-left.headerlink,.rst-content h5 .fa-pull-left.headerlink,.rst-content h6 .fa-pull-left.headerlink,.rst-content p.caption .fa-pull-left.headerlink,.rst-content table>caption .fa-pull-left.headerlink,.rst-content tt.download span.fa-pull-left:first-child,.wy-menu-vertical li.current>a span.fa-pull-left.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-left.toctree-expand,.wy-menu-vertical li span.fa-pull-left.toctree-expand{margin-right:.3em}.fa-pull-right.icon,.fa.fa-pull-right,.rst-content .code-block-caption .fa-pull-right.headerlink,.rst-content .fa-pull-right.admonition-title,.rst-content code.download span.fa-pull-right:first-child,.rst-content dl dt .fa-pull-right.headerlink,.rst-content h1 .fa-pull-right.headerlink,.rst-content h2 .fa-pull-right.headerlink,.rst-content h3 .fa-pull-right.headerlink,.rst-content h4 .fa-pull-right.headerlink,.rst-content h5 .fa-pull-right.headerlink,.rst-content h6 .fa-pull-right.headerlink,.rst-content p.caption .fa-pull-right.headerlink,.rst-content table>caption .fa-pull-right.headerlink,.rst-content tt.download span.fa-pull-right:first-child,.wy-menu-vertical li.current>a span.fa-pull-right.toctree-expand,.wy-menu-vertical li.on a span.fa-pull-right.toctree-expand,.wy-menu-vertical li span.fa-pull-right.toctree-expand{margin-left:.3em}.pull-right{float:right}.pull-left{float:left}.fa.pull-left,.pull-left.icon,.rst-content .code-block-caption .pull-left.headerlink,.rst-content .pull-left.admonition-title,.rst-content code.download span.pull-left:first-child,.rst-content dl dt .pull-left.headerlink,.rst-content h1 .pull-left.headerlink,.rst-content h2 .pull-left.headerlink,.rst-content h3 .pull-left.headerlink,.rst-content h4 .pull-left.headerlink,.rst-content h5 .pull-left.headerlink,.rst-content h6 .pull-left.headerlink,.rst-content p.caption .pull-left.headerlink,.rst-content table>caption .pull-left.headerlink,.rst-content tt.download span.pull-left:first-child,.wy-menu-vertical li.current>a span.pull-left.toctree-expand,.wy-menu-vertical li.on a span.pull-left.toctree-expand,.wy-menu-vertical li span.pull-left.toctree-expand{margin-right:.3em}.fa.pull-right,.pull-right.icon,.rst-content .code-block-caption .pull-right.headerlink,.rst-content .pull-right.admonition-title,.rst-content code.download span.pull-right:first-child,.rst-content dl dt .pull-right.headerlink,.rst-content h1 .pull-right.headerlink,.rst-content h2 .pull-right.headerlink,.rst-content h3 .pull-right.headerlink,.rst-content h4 .pull-right.headerlink,.rst-content h5 .pull-right.headerlink,.rst-content h6 .pull-right.headerlink,.rst-content p.caption .pull-right.headerlink,.rst-content table>caption .pull-right.headerlink,.rst-content tt.download span.pull-right:first-child,.wy-menu-vertical li.current>a span.pull-right.toctree-expand,.wy-menu-vertical li.on a span.pull-right.toctree-expand,.wy-menu-vertical li span.pull-right.toctree-expand{margin-left:.3em}.fa-spin{-webkit-animation:fa-spin 2s linear infinite;animation:fa-spin 2s linear infinite}.fa-pulse{-webkit-animation:fa-spin 1s steps(8) infinite;animation:fa-spin 1s steps(8) infinite}@-webkit-keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}@keyframes fa-spin{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}to{-webkit-transform:rotate(359deg);transform:rotate(359deg)}}.fa-rotate-90{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=1)";-webkit-transform:rotate(90deg);-ms-transform:rotate(90deg);transform:rotate(90deg)}.fa-rotate-180{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2)";-webkit-transform:rotate(180deg);-ms-transform:rotate(180deg);transform:rotate(180deg)}.fa-rotate-270{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=3)";-webkit-transform:rotate(270deg);-ms-transform:rotate(270deg);transform:rotate(270deg)}.fa-flip-horizontal{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=0, mirror=1)";-webkit-transform:scaleX(-1);-ms-transform:scaleX(-1);transform:scaleX(-1)}.fa-flip-vertical{-ms-filter:"progid:DXImageTransform.Microsoft.BasicImage(rotation=2, mirror=1)";-webkit-transform:scaleY(-1);-ms-transform:scaleY(-1);transform:scaleY(-1)}:root .fa-flip-horizontal,:root .fa-flip-vertical,:root .fa-rotate-90,:root .fa-rotate-180,:root .fa-rotate-270{filter:none}.fa-stack{position:relative;display:inline-block;width:2em;height:2em;line-height:2em;vertical-align:middle}.fa-stack-1x,.fa-stack-2x{position:absolute;left:0;width:100%;text-align:center}.fa-stack-1x{line-height:inherit}.fa-stack-2x{font-size:2em}.fa-inverse{color:#fff}.fa-glass:before{content:""}.fa-music:before{content:""}.fa-search:before,.icon-search:before{content:""}.fa-envelope-o:before{content:""}.fa-heart:before{content:""}.fa-star:before{content:""}.fa-star-o:before{content:""}.fa-user:before{content:""}.fa-film:before{content:""}.fa-th-large:before{content:""}.fa-th:before{content:""}.fa-th-list:before{content:""}.fa-check:before{content:""}.fa-close:before,.fa-remove:before,.fa-times:before{content:""}.fa-search-plus:before{content:""}.fa-search-minus:before{content:""}.fa-power-off:before{content:""}.fa-signal:before{content:""}.fa-cog:before,.fa-gear:before{content:""}.fa-trash-o:before{content:""}.fa-home:before,.icon-home:before{content:""}.fa-file-o:before{content:""}.fa-clock-o:before{content:""}.fa-road:before{content:""}.fa-download:before,.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{content:""}.fa-arrow-circle-o-down:before{content:""}.fa-arrow-circle-o-up:before{content:""}.fa-inbox:before{content:""}.fa-play-circle-o:before{content:""}.fa-repeat:before,.fa-rotate-right:before{content:""}.fa-refresh:before{content:""}.fa-list-alt:before{content:""}.fa-lock:before{content:""}.fa-flag:before{content:""}.fa-headphones:before{content:""}.fa-volume-off:before{content:""}.fa-volume-down:before{content:""}.fa-volume-up:before{content:""}.fa-qrcode:before{content:""}.fa-barcode:before{content:""}.fa-tag:before{content:""}.fa-tags:before{content:""}.fa-book:before,.icon-book:before{content:""}.fa-bookmark:before{content:""}.fa-print:before{content:""}.fa-camera:before{content:""}.fa-font:before{content:""}.fa-bold:before{content:""}.fa-italic:before{content:""}.fa-text-height:before{content:""}.fa-text-width:before{content:""}.fa-align-left:before{content:""}.fa-align-center:before{content:""}.fa-align-right:before{content:""}.fa-align-justify:before{content:""}.fa-list:before{content:""}.fa-dedent:before,.fa-outdent:before{content:""}.fa-indent:before{content:""}.fa-video-camera:before{content:""}.fa-image:before,.fa-photo:before,.fa-picture-o:before{content:""}.fa-pencil:before{content:""}.fa-map-marker:before{content:""}.fa-adjust:before{content:""}.fa-tint:before{content:""}.fa-edit:before,.fa-pencil-square-o:before{content:""}.fa-share-square-o:before{content:""}.fa-check-square-o:before{content:""}.fa-arrows:before{content:""}.fa-step-backward:before{content:""}.fa-fast-backward:before{content:""}.fa-backward:before{content:""}.fa-play:before{content:""}.fa-pause:before{content:""}.fa-stop:before{content:""}.fa-forward:before{content:""}.fa-fast-forward:before{content:""}.fa-step-forward:before{content:""}.fa-eject:before{content:""}.fa-chevron-left:before{content:""}.fa-chevron-right:before{content:""}.fa-plus-circle:before{content:""}.fa-minus-circle:before{content:""}.fa-times-circle:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before{content:""}.fa-check-circle:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before{content:""}.fa-question-circle:before{content:""}.fa-info-circle:before{content:""}.fa-crosshairs:before{content:""}.fa-times-circle-o:before{content:""}.fa-check-circle-o:before{content:""}.fa-ban:before{content:""}.fa-arrow-left:before{content:""}.fa-arrow-right:before{content:""}.fa-arrow-up:before{content:""}.fa-arrow-down:before{content:""}.fa-mail-forward:before,.fa-share:before{content:""}.fa-expand:before{content:""}.fa-compress:before{content:""}.fa-plus:before{content:""}.fa-minus:before{content:""}.fa-asterisk:before{content:""}.fa-exclamation-circle:before,.rst-content .admonition-title:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before{content:""}.fa-gift:before{content:""}.fa-leaf:before{content:""}.fa-fire:before,.icon-fire:before{content:""}.fa-eye:before{content:""}.fa-eye-slash:before{content:""}.fa-exclamation-triangle:before,.fa-warning:before{content:""}.fa-plane:before{content:""}.fa-calendar:before{content:""}.fa-random:before{content:""}.fa-comment:before{content:""}.fa-magnet:before{content:""}.fa-chevron-up:before{content:""}.fa-chevron-down:before{content:""}.fa-retweet:before{content:""}.fa-shopping-cart:before{content:""}.fa-folder:before{content:""}.fa-folder-open:before{content:""}.fa-arrows-v:before{content:""}.fa-arrows-h:before{content:""}.fa-bar-chart-o:before,.fa-bar-chart:before{content:""}.fa-twitter-square:before{content:""}.fa-facebook-square:before{content:""}.fa-camera-retro:before{content:""}.fa-key:before{content:""}.fa-cogs:before,.fa-gears:before{content:""}.fa-comments:before{content:""}.fa-thumbs-o-up:before{content:""}.fa-thumbs-o-down:before{content:""}.fa-star-half:before{content:""}.fa-heart-o:before{content:""}.fa-sign-out:before{content:""}.fa-linkedin-square:before{content:""}.fa-thumb-tack:before{content:""}.fa-external-link:before{content:""}.fa-sign-in:before{content:""}.fa-trophy:before{content:""}.fa-github-square:before{content:""}.fa-upload:before{content:""}.fa-lemon-o:before{content:""}.fa-phone:before{content:""}.fa-square-o:before{content:""}.fa-bookmark-o:before{content:""}.fa-phone-square:before{content:""}.fa-twitter:before{content:""}.fa-facebook-f:before,.fa-facebook:before{content:""}.fa-github:before,.icon-github:before{content:""}.fa-unlock:before{content:""}.fa-credit-card:before{content:""}.fa-feed:before,.fa-rss:before{content:""}.fa-hdd-o:before{content:""}.fa-bullhorn:before{content:""}.fa-bell:before{content:""}.fa-certificate:before{content:""}.fa-hand-o-right:before{content:""}.fa-hand-o-left:before{content:""}.fa-hand-o-up:before{content:""}.fa-hand-o-down:before{content:""}.fa-arrow-circle-left:before,.icon-circle-arrow-left:before{content:""}.fa-arrow-circle-right:before,.icon-circle-arrow-right:before{content:""}.fa-arrow-circle-up:before{content:""}.fa-arrow-circle-down:before{content:""}.fa-globe:before{content:""}.fa-wrench:before{content:""}.fa-tasks:before{content:""}.fa-filter:before{content:""}.fa-briefcase:before{content:""}.fa-arrows-alt:before{content:""}.fa-group:before,.fa-users:before{content:""}.fa-chain:before,.fa-link:before,.icon-link:before{content:""}.fa-cloud:before{content:""}.fa-flask:before{content:""}.fa-cut:before,.fa-scissors:before{content:""}.fa-copy:before,.fa-files-o:before{content:""}.fa-paperclip:before{content:""}.fa-floppy-o:before,.fa-save:before{content:""}.fa-square:before{content:""}.fa-bars:before,.fa-navicon:before,.fa-reorder:before{content:""}.fa-list-ul:before{content:""}.fa-list-ol:before{content:""}.fa-strikethrough:before{content:""}.fa-underline:before{content:""}.fa-table:before{content:""}.fa-magic:before{content:""}.fa-truck:before{content:""}.fa-pinterest:before{content:""}.fa-pinterest-square:before{content:""}.fa-google-plus-square:before{content:""}.fa-google-plus:before{content:""}.fa-money:before{content:""}.fa-caret-down:before,.icon-caret-down:before,.wy-dropdown .caret:before{content:""}.fa-caret-up:before{content:""}.fa-caret-left:before{content:""}.fa-caret-right:before{content:""}.fa-columns:before{content:""}.fa-sort:before,.fa-unsorted:before{content:""}.fa-sort-desc:before,.fa-sort-down:before{content:""}.fa-sort-asc:before,.fa-sort-up:before{content:""}.fa-envelope:before{content:""}.fa-linkedin:before{content:""}.fa-rotate-left:before,.fa-undo:before{content:""}.fa-gavel:before,.fa-legal:before{content:""}.fa-dashboard:before,.fa-tachometer:before{content:""}.fa-comment-o:before{content:""}.fa-comments-o:before{content:""}.fa-bolt:before,.fa-flash:before{content:""}.fa-sitemap:before{content:""}.fa-umbrella:before{content:""}.fa-clipboard:before,.fa-paste:before{content:""}.fa-lightbulb-o:before{content:""}.fa-exchange:before{content:""}.fa-cloud-download:before{content:""}.fa-cloud-upload:before{content:""}.fa-user-md:before{content:""}.fa-stethoscope:before{content:""}.fa-suitcase:before{content:""}.fa-bell-o:before{content:""}.fa-coffee:before{content:""}.fa-cutlery:before{content:""}.fa-file-text-o:before{content:""}.fa-building-o:before{content:""}.fa-hospital-o:before{content:""}.fa-ambulance:before{content:""}.fa-medkit:before{content:""}.fa-fighter-jet:before{content:""}.fa-beer:before{content:""}.fa-h-square:before{content:""}.fa-plus-square:before{content:""}.fa-angle-double-left:before{content:""}.fa-angle-double-right:before{content:""}.fa-angle-double-up:before{content:""}.fa-angle-double-down:before{content:""}.fa-angle-left:before{content:""}.fa-angle-right:before{content:""}.fa-angle-up:before{content:""}.fa-angle-down:before{content:""}.fa-desktop:before{content:""}.fa-laptop:before{content:""}.fa-tablet:before{content:""}.fa-mobile-phone:before,.fa-mobile:before{content:""}.fa-circle-o:before{content:""}.fa-quote-left:before{content:""}.fa-quote-right:before{content:""}.fa-spinner:before{content:""}.fa-circle:before{content:""}.fa-mail-reply:before,.fa-reply:before{content:""}.fa-github-alt:before{content:""}.fa-folder-o:before{content:""}.fa-folder-open-o:before{content:""}.fa-smile-o:before{content:""}.fa-frown-o:before{content:""}.fa-meh-o:before{content:""}.fa-gamepad:before{content:""}.fa-keyboard-o:before{content:""}.fa-flag-o:before{content:""}.fa-flag-checkered:before{content:""}.fa-terminal:before{content:""}.fa-code:before{content:""}.fa-mail-reply-all:before,.fa-reply-all:before{content:""}.fa-star-half-empty:before,.fa-star-half-full:before,.fa-star-half-o:before{content:""}.fa-location-arrow:before{content:""}.fa-crop:before{content:""}.fa-code-fork:before{content:""}.fa-chain-broken:before,.fa-unlink:before{content:""}.fa-question:before{content:""}.fa-info:before{content:""}.fa-exclamation:before{content:""}.fa-superscript:before{content:""}.fa-subscript:before{content:""}.fa-eraser:before{content:""}.fa-puzzle-piece:before{content:""}.fa-microphone:before{content:""}.fa-microphone-slash:before{content:""}.fa-shield:before{content:""}.fa-calendar-o:before{content:""}.fa-fire-extinguisher:before{content:""}.fa-rocket:before{content:""}.fa-maxcdn:before{content:""}.fa-chevron-circle-left:before{content:""}.fa-chevron-circle-right:before{content:""}.fa-chevron-circle-up:before{content:""}.fa-chevron-circle-down:before{content:""}.fa-html5:before{content:""}.fa-css3:before{content:""}.fa-anchor:before{content:""}.fa-unlock-alt:before{content:""}.fa-bullseye:before{content:""}.fa-ellipsis-h:before{content:""}.fa-ellipsis-v:before{content:""}.fa-rss-square:before{content:""}.fa-play-circle:before{content:""}.fa-ticket:before{content:""}.fa-minus-square:before{content:""}.fa-minus-square-o:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before{content:""}.fa-level-up:before{content:""}.fa-level-down:before{content:""}.fa-check-square:before{content:""}.fa-pencil-square:before{content:""}.fa-external-link-square:before{content:""}.fa-share-square:before{content:""}.fa-compass:before{content:""}.fa-caret-square-o-down:before,.fa-toggle-down:before{content:""}.fa-caret-square-o-up:before,.fa-toggle-up:before{content:""}.fa-caret-square-o-right:before,.fa-toggle-right:before{content:""}.fa-eur:before,.fa-euro:before{content:""}.fa-gbp:before{content:""}.fa-dollar:before,.fa-usd:before{content:""}.fa-inr:before,.fa-rupee:before{content:""}.fa-cny:before,.fa-jpy:before,.fa-rmb:before,.fa-yen:before{content:""}.fa-rouble:before,.fa-rub:before,.fa-ruble:before{content:""}.fa-krw:before,.fa-won:before{content:""}.fa-bitcoin:before,.fa-btc:before{content:""}.fa-file:before{content:""}.fa-file-text:before{content:""}.fa-sort-alpha-asc:before{content:""}.fa-sort-alpha-desc:before{content:""}.fa-sort-amount-asc:before{content:""}.fa-sort-amount-desc:before{content:""}.fa-sort-numeric-asc:before{content:""}.fa-sort-numeric-desc:before{content:""}.fa-thumbs-up:before{content:""}.fa-thumbs-down:before{content:""}.fa-youtube-square:before{content:""}.fa-youtube:before{content:""}.fa-xing:before{content:""}.fa-xing-square:before{content:""}.fa-youtube-play:before{content:""}.fa-dropbox:before{content:""}.fa-stack-overflow:before{content:""}.fa-instagram:before{content:""}.fa-flickr:before{content:""}.fa-adn:before{content:""}.fa-bitbucket:before,.icon-bitbucket:before{content:""}.fa-bitbucket-square:before{content:""}.fa-tumblr:before{content:""}.fa-tumblr-square:before{content:""}.fa-long-arrow-down:before{content:""}.fa-long-arrow-up:before{content:""}.fa-long-arrow-left:before{content:""}.fa-long-arrow-right:before{content:""}.fa-apple:before{content:""}.fa-windows:before{content:""}.fa-android:before{content:""}.fa-linux:before{content:""}.fa-dribbble:before{content:""}.fa-skype:before{content:""}.fa-foursquare:before{content:""}.fa-trello:before{content:""}.fa-female:before{content:""}.fa-male:before{content:""}.fa-gittip:before,.fa-gratipay:before{content:""}.fa-sun-o:before{content:""}.fa-moon-o:before{content:""}.fa-archive:before{content:""}.fa-bug:before{content:""}.fa-vk:before{content:""}.fa-weibo:before{content:""}.fa-renren:before{content:""}.fa-pagelines:before{content:""}.fa-stack-exchange:before{content:""}.fa-arrow-circle-o-right:before{content:""}.fa-arrow-circle-o-left:before{content:""}.fa-caret-square-o-left:before,.fa-toggle-left:before{content:""}.fa-dot-circle-o:before{content:""}.fa-wheelchair:before{content:""}.fa-vimeo-square:before{content:""}.fa-try:before,.fa-turkish-lira:before{content:""}.fa-plus-square-o:before,.wy-menu-vertical li span.toctree-expand:before{content:""}.fa-space-shuttle:before{content:""}.fa-slack:before{content:""}.fa-envelope-square:before{content:""}.fa-wordpress:before{content:""}.fa-openid:before{content:""}.fa-bank:before,.fa-institution:before,.fa-university:before{content:""}.fa-graduation-cap:before,.fa-mortar-board:before{content:""}.fa-yahoo:before{content:""}.fa-google:before{content:""}.fa-reddit:before{content:""}.fa-reddit-square:before{content:""}.fa-stumbleupon-circle:before{content:""}.fa-stumbleupon:before{content:""}.fa-delicious:before{content:""}.fa-digg:before{content:""}.fa-pied-piper-pp:before{content:""}.fa-pied-piper-alt:before{content:""}.fa-drupal:before{content:""}.fa-joomla:before{content:""}.fa-language:before{content:""}.fa-fax:before{content:""}.fa-building:before{content:""}.fa-child:before{content:""}.fa-paw:before{content:""}.fa-spoon:before{content:""}.fa-cube:before{content:""}.fa-cubes:before{content:""}.fa-behance:before{content:""}.fa-behance-square:before{content:""}.fa-steam:before{content:""}.fa-steam-square:before{content:""}.fa-recycle:before{content:""}.fa-automobile:before,.fa-car:before{content:""}.fa-cab:before,.fa-taxi:before{content:""}.fa-tree:before{content:""}.fa-spotify:before{content:""}.fa-deviantart:before{content:""}.fa-soundcloud:before{content:""}.fa-database:before{content:""}.fa-file-pdf-o:before{content:""}.fa-file-word-o:before{content:""}.fa-file-excel-o:before{content:""}.fa-file-powerpoint-o:before{content:""}.fa-file-image-o:before,.fa-file-photo-o:before,.fa-file-picture-o:before{content:""}.fa-file-archive-o:before,.fa-file-zip-o:before{content:""}.fa-file-audio-o:before,.fa-file-sound-o:before{content:""}.fa-file-movie-o:before,.fa-file-video-o:before{content:""}.fa-file-code-o:before{content:""}.fa-vine:before{content:""}.fa-codepen:before{content:""}.fa-jsfiddle:before{content:""}.fa-life-bouy:before,.fa-life-buoy:before,.fa-life-ring:before,.fa-life-saver:before,.fa-support:before{content:""}.fa-circle-o-notch:before{content:""}.fa-ra:before,.fa-rebel:before,.fa-resistance:before{content:""}.fa-empire:before,.fa-ge:before{content:""}.fa-git-square:before{content:""}.fa-git:before{content:""}.fa-hacker-news:before,.fa-y-combinator-square:before,.fa-yc-square:before{content:""}.fa-tencent-weibo:before{content:""}.fa-qq:before{content:""}.fa-wechat:before,.fa-weixin:before{content:""}.fa-paper-plane:before,.fa-send:before{content:""}.fa-paper-plane-o:before,.fa-send-o:before{content:""}.fa-history:before{content:""}.fa-circle-thin:before{content:""}.fa-header:before{content:""}.fa-paragraph:before{content:""}.fa-sliders:before{content:""}.fa-share-alt:before{content:""}.fa-share-alt-square:before{content:""}.fa-bomb:before{content:""}.fa-futbol-o:before,.fa-soccer-ball-o:before{content:""}.fa-tty:before{content:""}.fa-binoculars:before{content:""}.fa-plug:before{content:""}.fa-slideshare:before{content:""}.fa-twitch:before{content:""}.fa-yelp:before{content:""}.fa-newspaper-o:before{content:""}.fa-wifi:before{content:""}.fa-calculator:before{content:""}.fa-paypal:before{content:""}.fa-google-wallet:before{content:""}.fa-cc-visa:before{content:""}.fa-cc-mastercard:before{content:""}.fa-cc-discover:before{content:""}.fa-cc-amex:before{content:""}.fa-cc-paypal:before{content:""}.fa-cc-stripe:before{content:""}.fa-bell-slash:before{content:""}.fa-bell-slash-o:before{content:""}.fa-trash:before{content:""}.fa-copyright:before{content:""}.fa-at:before{content:""}.fa-eyedropper:before{content:""}.fa-paint-brush:before{content:""}.fa-birthday-cake:before{content:""}.fa-area-chart:before{content:""}.fa-pie-chart:before{content:""}.fa-line-chart:before{content:""}.fa-lastfm:before{content:""}.fa-lastfm-square:before{content:""}.fa-toggle-off:before{content:""}.fa-toggle-on:before{content:""}.fa-bicycle:before{content:""}.fa-bus:before{content:""}.fa-ioxhost:before{content:""}.fa-angellist:before{content:""}.fa-cc:before{content:""}.fa-ils:before,.fa-shekel:before,.fa-sheqel:before{content:""}.fa-meanpath:before{content:""}.fa-buysellads:before{content:""}.fa-connectdevelop:before{content:""}.fa-dashcube:before{content:""}.fa-forumbee:before{content:""}.fa-leanpub:before{content:""}.fa-sellsy:before{content:""}.fa-shirtsinbulk:before{content:""}.fa-simplybuilt:before{content:""}.fa-skyatlas:before{content:""}.fa-cart-plus:before{content:""}.fa-cart-arrow-down:before{content:""}.fa-diamond:before{content:""}.fa-ship:before{content:""}.fa-user-secret:before{content:""}.fa-motorcycle:before{content:""}.fa-street-view:before{content:""}.fa-heartbeat:before{content:""}.fa-venus:before{content:""}.fa-mars:before{content:""}.fa-mercury:before{content:""}.fa-intersex:before,.fa-transgender:before{content:""}.fa-transgender-alt:before{content:""}.fa-venus-double:before{content:""}.fa-mars-double:before{content:""}.fa-venus-mars:before{content:""}.fa-mars-stroke:before{content:""}.fa-mars-stroke-v:before{content:""}.fa-mars-stroke-h:before{content:""}.fa-neuter:before{content:""}.fa-genderless:before{content:""}.fa-facebook-official:before{content:""}.fa-pinterest-p:before{content:""}.fa-whatsapp:before{content:""}.fa-server:before{content:""}.fa-user-plus:before{content:""}.fa-user-times:before{content:""}.fa-bed:before,.fa-hotel:before{content:""}.fa-viacoin:before{content:""}.fa-train:before{content:""}.fa-subway:before{content:""}.fa-medium:before{content:""}.fa-y-combinator:before,.fa-yc:before{content:""}.fa-optin-monster:before{content:""}.fa-opencart:before{content:""}.fa-expeditedssl:before{content:""}.fa-battery-4:before,.fa-battery-full:before,.fa-battery:before{content:""}.fa-battery-3:before,.fa-battery-three-quarters:before{content:""}.fa-battery-2:before,.fa-battery-half:before{content:""}.fa-battery-1:before,.fa-battery-quarter:before{content:""}.fa-battery-0:before,.fa-battery-empty:before{content:""}.fa-mouse-pointer:before{content:""}.fa-i-cursor:before{content:""}.fa-object-group:before{content:""}.fa-object-ungroup:before{content:""}.fa-sticky-note:before{content:""}.fa-sticky-note-o:before{content:""}.fa-cc-jcb:before{content:""}.fa-cc-diners-club:before{content:""}.fa-clone:before{content:""}.fa-balance-scale:before{content:""}.fa-hourglass-o:before{content:""}.fa-hourglass-1:before,.fa-hourglass-start:before{content:""}.fa-hourglass-2:before,.fa-hourglass-half:before{content:""}.fa-hourglass-3:before,.fa-hourglass-end:before{content:""}.fa-hourglass:before{content:""}.fa-hand-grab-o:before,.fa-hand-rock-o:before{content:""}.fa-hand-paper-o:before,.fa-hand-stop-o:before{content:""}.fa-hand-scissors-o:before{content:""}.fa-hand-lizard-o:before{content:""}.fa-hand-spock-o:before{content:""}.fa-hand-pointer-o:before{content:""}.fa-hand-peace-o:before{content:""}.fa-trademark:before{content:""}.fa-registered:before{content:""}.fa-creative-commons:before{content:""}.fa-gg:before{content:""}.fa-gg-circle:before{content:""}.fa-tripadvisor:before{content:""}.fa-odnoklassniki:before{content:""}.fa-odnoklassniki-square:before{content:""}.fa-get-pocket:before{content:""}.fa-wikipedia-w:before{content:""}.fa-safari:before{content:""}.fa-chrome:before{content:""}.fa-firefox:before{content:""}.fa-opera:before{content:""}.fa-internet-explorer:before{content:""}.fa-television:before,.fa-tv:before{content:""}.fa-contao:before{content:""}.fa-500px:before{content:""}.fa-amazon:before{content:""}.fa-calendar-plus-o:before{content:""}.fa-calendar-minus-o:before{content:""}.fa-calendar-times-o:before{content:""}.fa-calendar-check-o:before{content:""}.fa-industry:before{content:""}.fa-map-pin:before{content:""}.fa-map-signs:before{content:""}.fa-map-o:before{content:""}.fa-map:before{content:""}.fa-commenting:before{content:""}.fa-commenting-o:before{content:""}.fa-houzz:before{content:""}.fa-vimeo:before{content:""}.fa-black-tie:before{content:""}.fa-fonticons:before{content:""}.fa-reddit-alien:before{content:""}.fa-edge:before{content:""}.fa-credit-card-alt:before{content:""}.fa-codiepie:before{content:""}.fa-modx:before{content:""}.fa-fort-awesome:before{content:""}.fa-usb:before{content:""}.fa-product-hunt:before{content:""}.fa-mixcloud:before{content:""}.fa-scribd:before{content:""}.fa-pause-circle:before{content:""}.fa-pause-circle-o:before{content:""}.fa-stop-circle:before{content:""}.fa-stop-circle-o:before{content:""}.fa-shopping-bag:before{content:""}.fa-shopping-basket:before{content:""}.fa-hashtag:before{content:""}.fa-bluetooth:before{content:""}.fa-bluetooth-b:before{content:""}.fa-percent:before{content:""}.fa-gitlab:before,.icon-gitlab:before{content:""}.fa-wpbeginner:before{content:""}.fa-wpforms:before{content:""}.fa-envira:before{content:""}.fa-universal-access:before{content:""}.fa-wheelchair-alt:before{content:""}.fa-question-circle-o:before{content:""}.fa-blind:before{content:""}.fa-audio-description:before{content:""}.fa-volume-control-phone:before{content:""}.fa-braille:before{content:""}.fa-assistive-listening-systems:before{content:""}.fa-american-sign-language-interpreting:before,.fa-asl-interpreting:before{content:""}.fa-deaf:before,.fa-deafness:before,.fa-hard-of-hearing:before{content:""}.fa-glide:before{content:""}.fa-glide-g:before{content:""}.fa-sign-language:before,.fa-signing:before{content:""}.fa-low-vision:before{content:""}.fa-viadeo:before{content:""}.fa-viadeo-square:before{content:""}.fa-snapchat:before{content:""}.fa-snapchat-ghost:before{content:""}.fa-snapchat-square:before{content:""}.fa-pied-piper:before{content:""}.fa-first-order:before{content:""}.fa-yoast:before{content:""}.fa-themeisle:before{content:""}.fa-google-plus-circle:before,.fa-google-plus-official:before{content:""}.fa-fa:before,.fa-font-awesome:before{content:""}.fa-handshake-o:before{content:""}.fa-envelope-open:before{content:""}.fa-envelope-open-o:before{content:""}.fa-linode:before{content:""}.fa-address-book:before{content:""}.fa-address-book-o:before{content:""}.fa-address-card:before,.fa-vcard:before{content:""}.fa-address-card-o:before,.fa-vcard-o:before{content:""}.fa-user-circle:before{content:""}.fa-user-circle-o:before{content:""}.fa-user-o:before{content:""}.fa-id-badge:before{content:""}.fa-drivers-license:before,.fa-id-card:before{content:""}.fa-drivers-license-o:before,.fa-id-card-o:before{content:""}.fa-quora:before{content:""}.fa-free-code-camp:before{content:""}.fa-telegram:before{content:""}.fa-thermometer-4:before,.fa-thermometer-full:before,.fa-thermometer:before{content:""}.fa-thermometer-3:before,.fa-thermometer-three-quarters:before{content:""}.fa-thermometer-2:before,.fa-thermometer-half:before{content:""}.fa-thermometer-1:before,.fa-thermometer-quarter:before{content:""}.fa-thermometer-0:before,.fa-thermometer-empty:before{content:""}.fa-shower:before{content:""}.fa-bath:before,.fa-bathtub:before,.fa-s15:before{content:""}.fa-podcast:before{content:""}.fa-window-maximize:before{content:""}.fa-window-minimize:before{content:""}.fa-window-restore:before{content:""}.fa-times-rectangle:before,.fa-window-close:before{content:""}.fa-times-rectangle-o:before,.fa-window-close-o:before{content:""}.fa-bandcamp:before{content:""}.fa-grav:before{content:""}.fa-etsy:before{content:""}.fa-imdb:before{content:""}.fa-ravelry:before{content:""}.fa-eercast:before{content:""}.fa-microchip:before{content:""}.fa-snowflake-o:before{content:""}.fa-superpowers:before{content:""}.fa-wpexplorer:before{content:""}.fa-meetup:before{content:""}.sr-only{position:absolute;width:1px;height:1px;padding:0;margin:-1px;overflow:hidden;clip:rect(0,0,0,0);border:0}.sr-only-focusable:active,.sr-only-focusable:focus{position:static;width:auto;height:auto;margin:0;overflow:visible;clip:auto}.fa,.icon,.rst-content .admonition-title,.rst-content .code-block-caption .headerlink,.rst-content code.download span:first-child,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink,.rst-content tt.download span:first-child,.wy-dropdown .caret,.wy-inline-validate.wy-inline-validate-danger .wy-input-context,.wy-inline-validate.wy-inline-validate-info .wy-input-context,.wy-inline-validate.wy-inline-validate-success .wy-input-context,.wy-inline-validate.wy-inline-validate-warning .wy-input-context,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li span.toctree-expand{font-family:inherit}.fa:before,.icon:before,.rst-content .admonition-title:before,.rst-content .code-block-caption .headerlink:before,.rst-content code.download span:first-child:before,.rst-content dl dt .headerlink:before,.rst-content h1 .headerlink:before,.rst-content h2 .headerlink:before,.rst-content h3 .headerlink:before,.rst-content h4 .headerlink:before,.rst-content h5 .headerlink:before,.rst-content h6 .headerlink:before,.rst-content p.caption .headerlink:before,.rst-content table>caption .headerlink:before,.rst-content tt.download span:first-child:before,.wy-dropdown .caret:before,.wy-inline-validate.wy-inline-validate-danger .wy-input-context:before,.wy-inline-validate.wy-inline-validate-info .wy-input-context:before,.wy-inline-validate.wy-inline-validate-success .wy-input-context:before,.wy-inline-validate.wy-inline-validate-warning .wy-input-context:before,.wy-menu-vertical li.current>a span.toctree-expand:before,.wy-menu-vertical li.on a span.toctree-expand:before,.wy-menu-vertical li span.toctree-expand:before{font-family:FontAwesome;display:inline-block;font-style:normal;font-weight:400;line-height:1;text-decoration:inherit}.rst-content .code-block-caption a .headerlink,.rst-content a .admonition-title,.rst-content code.download a span:first-child,.rst-content dl dt a .headerlink,.rst-content h1 a .headerlink,.rst-content h2 a .headerlink,.rst-content h3 a .headerlink,.rst-content h4 a .headerlink,.rst-content h5 a .headerlink,.rst-content h6 a .headerlink,.rst-content p.caption a .headerlink,.rst-content table>caption a .headerlink,.rst-content tt.download a span:first-child,.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand,.wy-menu-vertical li a span.toctree-expand,a .fa,a .icon,a .rst-content .admonition-title,a .rst-content .code-block-caption .headerlink,a .rst-content code.download span:first-child,a .rst-content dl dt .headerlink,a .rst-content h1 .headerlink,a .rst-content h2 .headerlink,a .rst-content h3 .headerlink,a .rst-content h4 .headerlink,a .rst-content h5 .headerlink,a .rst-content h6 .headerlink,a .rst-content p.caption .headerlink,a .rst-content table>caption .headerlink,a .rst-content tt.download span:first-child,a .wy-menu-vertical li span.toctree-expand{display:inline-block;text-decoration:inherit}.btn .fa,.btn .icon,.btn .rst-content .admonition-title,.btn .rst-content .code-block-caption .headerlink,.btn .rst-content code.download span:first-child,.btn .rst-content dl dt .headerlink,.btn .rst-content h1 .headerlink,.btn .rst-content h2 .headerlink,.btn .rst-content h3 .headerlink,.btn .rst-content h4 .headerlink,.btn .rst-content h5 .headerlink,.btn .rst-content h6 .headerlink,.btn .rst-content p.caption .headerlink,.btn .rst-content table>caption .headerlink,.btn .rst-content tt.download span:first-child,.btn .wy-menu-vertical li.current>a span.toctree-expand,.btn .wy-menu-vertical li.on a span.toctree-expand,.btn .wy-menu-vertical li span.toctree-expand,.nav .fa,.nav .icon,.nav .rst-content .admonition-title,.nav .rst-content .code-block-caption .headerlink,.nav .rst-content code.download span:first-child,.nav .rst-content dl dt .headerlink,.nav .rst-content h1 .headerlink,.nav .rst-content h2 .headerlink,.nav .rst-content h3 .headerlink,.nav .rst-content h4 .headerlink,.nav .rst-content h5 .headerlink,.nav .rst-content h6 .headerlink,.nav .rst-content p.caption .headerlink,.nav .rst-content table>caption .headerlink,.nav .rst-content tt.download span:first-child,.nav .wy-menu-vertical li.current>a span.toctree-expand,.nav .wy-menu-vertical li.on a span.toctree-expand,.nav .wy-menu-vertical li span.toctree-expand,.rst-content .btn .admonition-title,.rst-content .code-block-caption .btn .headerlink,.rst-content .code-block-caption .nav .headerlink,.rst-content .nav .admonition-title,.rst-content code.download .btn span:first-child,.rst-content code.download .nav span:first-child,.rst-content dl dt .btn .headerlink,.rst-content dl dt .nav .headerlink,.rst-content h1 .btn .headerlink,.rst-content h1 .nav .headerlink,.rst-content h2 .btn .headerlink,.rst-content h2 .nav .headerlink,.rst-content h3 .btn .headerlink,.rst-content h3 .nav .headerlink,.rst-content h4 .btn .headerlink,.rst-content h4 .nav .headerlink,.rst-content h5 .btn .headerlink,.rst-content h5 .nav .headerlink,.rst-content h6 .btn .headerlink,.rst-content h6 .nav .headerlink,.rst-content p.caption .btn .headerlink,.rst-content p.caption .nav .headerlink,.rst-content table>caption .btn .headerlink,.rst-content table>caption .nav .headerlink,.rst-content tt.download .btn span:first-child,.rst-content tt.download .nav span:first-child,.wy-menu-vertical li .btn span.toctree-expand,.wy-menu-vertical li.current>a .btn span.toctree-expand,.wy-menu-vertical li.current>a .nav span.toctree-expand,.wy-menu-vertical li .nav span.toctree-expand,.wy-menu-vertical li.on a .btn span.toctree-expand,.wy-menu-vertical li.on a .nav span.toctree-expand{display:inline}.btn .fa-large.icon,.btn .fa.fa-large,.btn .rst-content .code-block-caption .fa-large.headerlink,.btn .rst-content .fa-large.admonition-title,.btn .rst-content code.download span.fa-large:first-child,.btn .rst-content dl dt .fa-large.headerlink,.btn .rst-content h1 .fa-large.headerlink,.btn .rst-content h2 .fa-large.headerlink,.btn .rst-content h3 .fa-large.headerlink,.btn .rst-content h4 .fa-large.headerlink,.btn .rst-content h5 .fa-large.headerlink,.btn .rst-content h6 .fa-large.headerlink,.btn .rst-content p.caption .fa-large.headerlink,.btn .rst-content table>caption .fa-large.headerlink,.btn .rst-content tt.download span.fa-large:first-child,.btn .wy-menu-vertical li span.fa-large.toctree-expand,.nav .fa-large.icon,.nav .fa.fa-large,.nav .rst-content .code-block-caption .fa-large.headerlink,.nav .rst-content .fa-large.admonition-title,.nav .rst-content code.download span.fa-large:first-child,.nav .rst-content dl dt .fa-large.headerlink,.nav .rst-content h1 .fa-large.headerlink,.nav .rst-content h2 .fa-large.headerlink,.nav .rst-content h3 .fa-large.headerlink,.nav .rst-content h4 .fa-large.headerlink,.nav .rst-content h5 .fa-large.headerlink,.nav .rst-content h6 .fa-large.headerlink,.nav .rst-content p.caption .fa-large.headerlink,.nav .rst-content table>caption .fa-large.headerlink,.nav .rst-content tt.download span.fa-large:first-child,.nav .wy-menu-vertical li span.fa-large.toctree-expand,.rst-content .btn .fa-large.admonition-title,.rst-content .code-block-caption .btn .fa-large.headerlink,.rst-content .code-block-caption .nav .fa-large.headerlink,.rst-content .nav .fa-large.admonition-title,.rst-content code.download .btn span.fa-large:first-child,.rst-content code.download .nav span.fa-large:first-child,.rst-content dl dt .btn .fa-large.headerlink,.rst-content dl dt .nav .fa-large.headerlink,.rst-content h1 .btn .fa-large.headerlink,.rst-content h1 .nav .fa-large.headerlink,.rst-content h2 .btn .fa-large.headerlink,.rst-content h2 .nav .fa-large.headerlink,.rst-content h3 .btn .fa-large.headerlink,.rst-content h3 .nav .fa-large.headerlink,.rst-content h4 .btn .fa-large.headerlink,.rst-content h4 .nav .fa-large.headerlink,.rst-content h5 .btn .fa-large.headerlink,.rst-content h5 .nav .fa-large.headerlink,.rst-content h6 .btn .fa-large.headerlink,.rst-content h6 .nav .fa-large.headerlink,.rst-content p.caption .btn .fa-large.headerlink,.rst-content p.caption .nav .fa-large.headerlink,.rst-content table>caption .btn .fa-large.headerlink,.rst-content table>caption .nav .fa-large.headerlink,.rst-content tt.download .btn span.fa-large:first-child,.rst-content tt.download .nav span.fa-large:first-child,.wy-menu-vertical li .btn span.fa-large.toctree-expand,.wy-menu-vertical li .nav span.fa-large.toctree-expand{line-height:.9em}.btn .fa-spin.icon,.btn .fa.fa-spin,.btn .rst-content .code-block-caption .fa-spin.headerlink,.btn .rst-content .fa-spin.admonition-title,.btn .rst-content code.download span.fa-spin:first-child,.btn .rst-content dl dt .fa-spin.headerlink,.btn .rst-content h1 .fa-spin.headerlink,.btn .rst-content h2 .fa-spin.headerlink,.btn .rst-content h3 .fa-spin.headerlink,.btn .rst-content h4 .fa-spin.headerlink,.btn .rst-content h5 .fa-spin.headerlink,.btn .rst-content h6 .fa-spin.headerlink,.btn .rst-content p.caption .fa-spin.headerlink,.btn .rst-content table>caption .fa-spin.headerlink,.btn .rst-content tt.download span.fa-spin:first-child,.btn .wy-menu-vertical li span.fa-spin.toctree-expand,.nav .fa-spin.icon,.nav .fa.fa-spin,.nav .rst-content .code-block-caption .fa-spin.headerlink,.nav .rst-content .fa-spin.admonition-title,.nav .rst-content code.download span.fa-spin:first-child,.nav .rst-content dl dt .fa-spin.headerlink,.nav .rst-content h1 .fa-spin.headerlink,.nav .rst-content h2 .fa-spin.headerlink,.nav .rst-content h3 .fa-spin.headerlink,.nav .rst-content h4 .fa-spin.headerlink,.nav .rst-content h5 .fa-spin.headerlink,.nav .rst-content h6 .fa-spin.headerlink,.nav .rst-content p.caption .fa-spin.headerlink,.nav .rst-content table>caption .fa-spin.headerlink,.nav .rst-content tt.download span.fa-spin:first-child,.nav .wy-menu-vertical li span.fa-spin.toctree-expand,.rst-content .btn .fa-spin.admonition-title,.rst-content .code-block-caption .btn .fa-spin.headerlink,.rst-content .code-block-caption .nav .fa-spin.headerlink,.rst-content .nav .fa-spin.admonition-title,.rst-content code.download .btn span.fa-spin:first-child,.rst-content code.download .nav span.fa-spin:first-child,.rst-content dl dt .btn .fa-spin.headerlink,.rst-content dl dt .nav .fa-spin.headerlink,.rst-content h1 .btn .fa-spin.headerlink,.rst-content h1 .nav .fa-spin.headerlink,.rst-content h2 .btn .fa-spin.headerlink,.rst-content h2 .nav .fa-spin.headerlink,.rst-content h3 .btn .fa-spin.headerlink,.rst-content h3 .nav .fa-spin.headerlink,.rst-content h4 .btn .fa-spin.headerlink,.rst-content h4 .nav .fa-spin.headerlink,.rst-content h5 .btn .fa-spin.headerlink,.rst-content h5 .nav .fa-spin.headerlink,.rst-content h6 .btn .fa-spin.headerlink,.rst-content h6 .nav .fa-spin.headerlink,.rst-content p.caption .btn .fa-spin.headerlink,.rst-content p.caption .nav .fa-spin.headerlink,.rst-content table>caption .btn .fa-spin.headerlink,.rst-content table>caption .nav .fa-spin.headerlink,.rst-content tt.download .btn span.fa-spin:first-child,.rst-content tt.download .nav span.fa-spin:first-child,.wy-menu-vertical li .btn span.fa-spin.toctree-expand,.wy-menu-vertical li .nav span.fa-spin.toctree-expand{display:inline-block}.btn.fa:before,.btn.icon:before,.rst-content .btn.admonition-title:before,.rst-content .code-block-caption .btn.headerlink:before,.rst-content code.download span.btn:first-child:before,.rst-content dl dt .btn.headerlink:before,.rst-content h1 .btn.headerlink:before,.rst-content h2 .btn.headerlink:before,.rst-content h3 .btn.headerlink:before,.rst-content h4 .btn.headerlink:before,.rst-content h5 .btn.headerlink:before,.rst-content h6 .btn.headerlink:before,.rst-content p.caption .btn.headerlink:before,.rst-content table>caption .btn.headerlink:before,.rst-content tt.download span.btn:first-child:before,.wy-menu-vertical li span.btn.toctree-expand:before{opacity:.5;-webkit-transition:opacity .05s ease-in;-moz-transition:opacity .05s ease-in;transition:opacity .05s ease-in}.btn.fa:hover:before,.btn.icon:hover:before,.rst-content .btn.admonition-title:hover:before,.rst-content .code-block-caption .btn.headerlink:hover:before,.rst-content code.download span.btn:first-child:hover:before,.rst-content dl dt .btn.headerlink:hover:before,.rst-content h1 .btn.headerlink:hover:before,.rst-content h2 .btn.headerlink:hover:before,.rst-content h3 .btn.headerlink:hover:before,.rst-content h4 .btn.headerlink:hover:before,.rst-content h5 .btn.headerlink:hover:before,.rst-content h6 .btn.headerlink:hover:before,.rst-content p.caption .btn.headerlink:hover:before,.rst-content table>caption .btn.headerlink:hover:before,.rst-content tt.download span.btn:first-child:hover:before,.wy-menu-vertical li span.btn.toctree-expand:hover:before{opacity:1}.btn-mini .fa:before,.btn-mini .icon:before,.btn-mini .rst-content .admonition-title:before,.btn-mini .rst-content .code-block-caption .headerlink:before,.btn-mini .rst-content code.download span:first-child:before,.btn-mini .rst-content dl dt .headerlink:before,.btn-mini .rst-content h1 .headerlink:before,.btn-mini .rst-content h2 .headerlink:before,.btn-mini .rst-content h3 .headerlink:before,.btn-mini .rst-content h4 .headerlink:before,.btn-mini .rst-content h5 .headerlink:before,.btn-mini .rst-content h6 .headerlink:before,.btn-mini .rst-content p.caption .headerlink:before,.btn-mini .rst-content table>caption .headerlink:before,.btn-mini .rst-content tt.download span:first-child:before,.btn-mini .wy-menu-vertical li span.toctree-expand:before,.rst-content .btn-mini .admonition-title:before,.rst-content .code-block-caption .btn-mini .headerlink:before,.rst-content code.download .btn-mini span:first-child:before,.rst-content dl dt .btn-mini .headerlink:before,.rst-content h1 .btn-mini .headerlink:before,.rst-content h2 .btn-mini .headerlink:before,.rst-content h3 .btn-mini .headerlink:before,.rst-content h4 .btn-mini .headerlink:before,.rst-content h5 .btn-mini .headerlink:before,.rst-content h6 .btn-mini .headerlink:before,.rst-content p.caption .btn-mini .headerlink:before,.rst-content table>caption .btn-mini .headerlink:before,.rst-content tt.download .btn-mini span:first-child:before,.wy-menu-vertical li .btn-mini span.toctree-expand:before{font-size:14px;vertical-align:-15%}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning,.wy-alert{padding:12px;line-height:24px;margin-bottom:24px;background:#e7f2fa}.rst-content .admonition-title,.wy-alert-title{font-weight:700;display:block;color:#fff;background:#6ab0de;padding:6px 12px;margin:-12px -12px 12px}.rst-content .danger,.rst-content .error,.rst-content .wy-alert-danger.admonition,.rst-content .wy-alert-danger.admonition-todo,.rst-content .wy-alert-danger.attention,.rst-content .wy-alert-danger.caution,.rst-content .wy-alert-danger.hint,.rst-content .wy-alert-danger.important,.rst-content .wy-alert-danger.note,.rst-content .wy-alert-danger.seealso,.rst-content .wy-alert-danger.tip,.rst-content .wy-alert-danger.warning,.wy-alert.wy-alert-danger{background:#fdf3f2}.rst-content .danger .admonition-title,.rst-content .danger .wy-alert-title,.rst-content .error .admonition-title,.rst-content .error .wy-alert-title,.rst-content .wy-alert-danger.admonition-todo .admonition-title,.rst-content .wy-alert-danger.admonition-todo .wy-alert-title,.rst-content .wy-alert-danger.admonition .admonition-title,.rst-content .wy-alert-danger.admonition .wy-alert-title,.rst-content .wy-alert-danger.attention .admonition-title,.rst-content .wy-alert-danger.attention .wy-alert-title,.rst-content .wy-alert-danger.caution .admonition-title,.rst-content .wy-alert-danger.caution .wy-alert-title,.rst-content .wy-alert-danger.hint .admonition-title,.rst-content .wy-alert-danger.hint .wy-alert-title,.rst-content .wy-alert-danger.important .admonition-title,.rst-content .wy-alert-danger.important .wy-alert-title,.rst-content .wy-alert-danger.note .admonition-title,.rst-content .wy-alert-danger.note .wy-alert-title,.rst-content .wy-alert-danger.seealso .admonition-title,.rst-content .wy-alert-danger.seealso .wy-alert-title,.rst-content .wy-alert-danger.tip .admonition-title,.rst-content .wy-alert-danger.tip .wy-alert-title,.rst-content .wy-alert-danger.warning .admonition-title,.rst-content .wy-alert-danger.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-danger .admonition-title,.wy-alert.wy-alert-danger .rst-content .admonition-title,.wy-alert.wy-alert-danger .wy-alert-title{background:#f29f97}.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .warning,.rst-content .wy-alert-warning.admonition,.rst-content .wy-alert-warning.danger,.rst-content .wy-alert-warning.error,.rst-content .wy-alert-warning.hint,.rst-content .wy-alert-warning.important,.rst-content .wy-alert-warning.note,.rst-content .wy-alert-warning.seealso,.rst-content .wy-alert-warning.tip,.wy-alert.wy-alert-warning{background:#ffedcc}.rst-content .admonition-todo .admonition-title,.rst-content .admonition-todo .wy-alert-title,.rst-content .attention .admonition-title,.rst-content .attention .wy-alert-title,.rst-content .caution .admonition-title,.rst-content .caution .wy-alert-title,.rst-content .warning .admonition-title,.rst-content .warning .wy-alert-title,.rst-content .wy-alert-warning.admonition .admonition-title,.rst-content .wy-alert-warning.admonition .wy-alert-title,.rst-content .wy-alert-warning.danger .admonition-title,.rst-content .wy-alert-warning.danger .wy-alert-title,.rst-content .wy-alert-warning.error .admonition-title,.rst-content .wy-alert-warning.error .wy-alert-title,.rst-content .wy-alert-warning.hint .admonition-title,.rst-content .wy-alert-warning.hint .wy-alert-title,.rst-content .wy-alert-warning.important .admonition-title,.rst-content .wy-alert-warning.important .wy-alert-title,.rst-content .wy-alert-warning.note .admonition-title,.rst-content .wy-alert-warning.note .wy-alert-title,.rst-content .wy-alert-warning.seealso .admonition-title,.rst-content .wy-alert-warning.seealso .wy-alert-title,.rst-content .wy-alert-warning.tip .admonition-title,.rst-content .wy-alert-warning.tip .wy-alert-title,.rst-content .wy-alert.wy-alert-warning .admonition-title,.wy-alert.wy-alert-warning .rst-content .admonition-title,.wy-alert.wy-alert-warning .wy-alert-title{background:#f0b37e}.rst-content .note,.rst-content .seealso,.rst-content .wy-alert-info.admonition,.rst-content .wy-alert-info.admonition-todo,.rst-content .wy-alert-info.attention,.rst-content .wy-alert-info.caution,.rst-content .wy-alert-info.danger,.rst-content .wy-alert-info.error,.rst-content .wy-alert-info.hint,.rst-content .wy-alert-info.important,.rst-content .wy-alert-info.tip,.rst-content .wy-alert-info.warning,.wy-alert.wy-alert-info{background:#e7f2fa}.rst-content .note .admonition-title,.rst-content .note .wy-alert-title,.rst-content .seealso .admonition-title,.rst-content .seealso .wy-alert-title,.rst-content .wy-alert-info.admonition-todo .admonition-title,.rst-content .wy-alert-info.admonition-todo .wy-alert-title,.rst-content .wy-alert-info.admonition .admonition-title,.rst-content .wy-alert-info.admonition .wy-alert-title,.rst-content .wy-alert-info.attention .admonition-title,.rst-content .wy-alert-info.attention .wy-alert-title,.rst-content .wy-alert-info.caution .admonition-title,.rst-content .wy-alert-info.caution .wy-alert-title,.rst-content .wy-alert-info.danger .admonition-title,.rst-content .wy-alert-info.danger .wy-alert-title,.rst-content .wy-alert-info.error .admonition-title,.rst-content .wy-alert-info.error .wy-alert-title,.rst-content .wy-alert-info.hint .admonition-title,.rst-content .wy-alert-info.hint .wy-alert-title,.rst-content .wy-alert-info.important .admonition-title,.rst-content .wy-alert-info.important .wy-alert-title,.rst-content .wy-alert-info.tip .admonition-title,.rst-content .wy-alert-info.tip .wy-alert-title,.rst-content .wy-alert-info.warning .admonition-title,.rst-content .wy-alert-info.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-info .admonition-title,.wy-alert.wy-alert-info .rst-content .admonition-title,.wy-alert.wy-alert-info .wy-alert-title{background:#6ab0de}.rst-content .hint,.rst-content .important,.rst-content .tip,.rst-content .wy-alert-success.admonition,.rst-content .wy-alert-success.admonition-todo,.rst-content .wy-alert-success.attention,.rst-content .wy-alert-success.caution,.rst-content .wy-alert-success.danger,.rst-content .wy-alert-success.error,.rst-content .wy-alert-success.note,.rst-content .wy-alert-success.seealso,.rst-content .wy-alert-success.warning,.wy-alert.wy-alert-success{background:#dbfaf4}.rst-content .hint .admonition-title,.rst-content .hint .wy-alert-title,.rst-content .important .admonition-title,.rst-content .important .wy-alert-title,.rst-content .tip .admonition-title,.rst-content .tip .wy-alert-title,.rst-content .wy-alert-success.admonition-todo .admonition-title,.rst-content .wy-alert-success.admonition-todo .wy-alert-title,.rst-content .wy-alert-success.admonition .admonition-title,.rst-content .wy-alert-success.admonition .wy-alert-title,.rst-content .wy-alert-success.attention .admonition-title,.rst-content .wy-alert-success.attention .wy-alert-title,.rst-content .wy-alert-success.caution .admonition-title,.rst-content .wy-alert-success.caution .wy-alert-title,.rst-content .wy-alert-success.danger .admonition-title,.rst-content .wy-alert-success.danger .wy-alert-title,.rst-content .wy-alert-success.error .admonition-title,.rst-content .wy-alert-success.error .wy-alert-title,.rst-content .wy-alert-success.note .admonition-title,.rst-content .wy-alert-success.note .wy-alert-title,.rst-content .wy-alert-success.seealso .admonition-title,.rst-content .wy-alert-success.seealso .wy-alert-title,.rst-content .wy-alert-success.warning .admonition-title,.rst-content .wy-alert-success.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-success .admonition-title,.wy-alert.wy-alert-success .rst-content .admonition-title,.wy-alert.wy-alert-success .wy-alert-title{background:#1abc9c}.rst-content .wy-alert-neutral.admonition,.rst-content .wy-alert-neutral.admonition-todo,.rst-content .wy-alert-neutral.attention,.rst-content .wy-alert-neutral.caution,.rst-content .wy-alert-neutral.danger,.rst-content .wy-alert-neutral.error,.rst-content .wy-alert-neutral.hint,.rst-content .wy-alert-neutral.important,.rst-content .wy-alert-neutral.note,.rst-content .wy-alert-neutral.seealso,.rst-content .wy-alert-neutral.tip,.rst-content .wy-alert-neutral.warning,.wy-alert.wy-alert-neutral{background:#f3f6f6}.rst-content .wy-alert-neutral.admonition-todo .admonition-title,.rst-content .wy-alert-neutral.admonition-todo .wy-alert-title,.rst-content .wy-alert-neutral.admonition .admonition-title,.rst-content .wy-alert-neutral.admonition .wy-alert-title,.rst-content .wy-alert-neutral.attention .admonition-title,.rst-content .wy-alert-neutral.attention .wy-alert-title,.rst-content .wy-alert-neutral.caution .admonition-title,.rst-content .wy-alert-neutral.caution .wy-alert-title,.rst-content .wy-alert-neutral.danger .admonition-title,.rst-content .wy-alert-neutral.danger .wy-alert-title,.rst-content .wy-alert-neutral.error .admonition-title,.rst-content .wy-alert-neutral.error .wy-alert-title,.rst-content .wy-alert-neutral.hint .admonition-title,.rst-content .wy-alert-neutral.hint .wy-alert-title,.rst-content .wy-alert-neutral.important .admonition-title,.rst-content .wy-alert-neutral.important .wy-alert-title,.rst-content .wy-alert-neutral.note .admonition-title,.rst-content .wy-alert-neutral.note .wy-alert-title,.rst-content .wy-alert-neutral.seealso .admonition-title,.rst-content .wy-alert-neutral.seealso .wy-alert-title,.rst-content .wy-alert-neutral.tip .admonition-title,.rst-content .wy-alert-neutral.tip .wy-alert-title,.rst-content .wy-alert-neutral.warning .admonition-title,.rst-content .wy-alert-neutral.warning .wy-alert-title,.rst-content .wy-alert.wy-alert-neutral .admonition-title,.wy-alert.wy-alert-neutral .rst-content .admonition-title,.wy-alert.wy-alert-neutral .wy-alert-title{color:#404040;background:#e1e4e5}.rst-content .wy-alert-neutral.admonition-todo a,.rst-content .wy-alert-neutral.admonition a,.rst-content .wy-alert-neutral.attention a,.rst-content .wy-alert-neutral.caution a,.rst-content .wy-alert-neutral.danger a,.rst-content .wy-alert-neutral.error a,.rst-content .wy-alert-neutral.hint a,.rst-content .wy-alert-neutral.important a,.rst-content .wy-alert-neutral.note a,.rst-content .wy-alert-neutral.seealso a,.rst-content .wy-alert-neutral.tip a,.rst-content .wy-alert-neutral.warning a,.wy-alert.wy-alert-neutral a{color:#2980b9}.rst-content .admonition-todo p:last-child,.rst-content .admonition p:last-child,.rst-content .attention p:last-child,.rst-content .caution p:last-child,.rst-content .danger p:last-child,.rst-content .error p:last-child,.rst-content .hint p:last-child,.rst-content .important p:last-child,.rst-content .note p:last-child,.rst-content .seealso p:last-child,.rst-content .tip p:last-child,.rst-content .warning p:last-child,.wy-alert p:last-child{margin-bottom:0}.wy-tray-container{position:fixed;bottom:0;left:0;z-index:600}.wy-tray-container li{display:block;width:300px;background:transparent;color:#fff;text-align:center;box-shadow:0 5px 5px 0 rgba(0,0,0,.1);padding:0 24px;min-width:20%;opacity:0;height:0;line-height:56px;overflow:hidden;-webkit-transition:all .3s ease-in;-moz-transition:all .3s ease-in;transition:all .3s ease-in}.wy-tray-container li.wy-tray-item-success{background:#27ae60}.wy-tray-container li.wy-tray-item-info{background:#2980b9}.wy-tray-container li.wy-tray-item-warning{background:#e67e22}.wy-tray-container li.wy-tray-item-danger{background:#e74c3c}.wy-tray-container li.on{opacity:1;height:56px}@media screen and (max-width:768px){.wy-tray-container{bottom:auto;top:0;width:100%}.wy-tray-container li{width:100%}}button{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle;cursor:pointer;line-height:normal;-webkit-appearance:button;*overflow:visible}button::-moz-focus-inner,input::-moz-focus-inner{border:0;padding:0}button[disabled]{cursor:default}.btn{display:inline-block;border-radius:2px;line-height:normal;white-space:nowrap;text-align:center;cursor:pointer;font-size:100%;padding:6px 12px 8px;color:#fff;border:1px solid rgba(0,0,0,.1);background-color:#27ae60;text-decoration:none;font-weight:400;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 2px -1px hsla(0,0%,100%,.5),inset 0 -2px 0 0 rgba(0,0,0,.1);outline-none:false;vertical-align:middle;*display:inline;zoom:1;-webkit-user-drag:none;-webkit-user-select:none;-moz-user-select:none;-ms-user-select:none;user-select:none;-webkit-transition:all .1s linear;-moz-transition:all .1s linear;transition:all .1s linear}.btn-hover{background:#2e8ece;color:#fff}.btn:hover{background:#2cc36b;color:#fff}.btn:focus{background:#2cc36b;outline:0}.btn:active{box-shadow:inset 0 -1px 0 0 rgba(0,0,0,.05),inset 0 2px 0 0 rgba(0,0,0,.1);padding:8px 12px 6px}.btn:visited{color:#fff}.btn-disabled,.btn-disabled:active,.btn-disabled:focus,.btn-disabled:hover,.btn:disabled{background-image:none;filter:progid:DXImageTransform.Microsoft.gradient(enabled = false);filter:alpha(opacity=40);opacity:.4;cursor:not-allowed;box-shadow:none}.btn::-moz-focus-inner{padding:0;border:0}.btn-small{font-size:80%}.btn-info{background-color:#2980b9!important}.btn-info:hover{background-color:#2e8ece!important}.btn-neutral{background-color:#f3f6f6!important;color:#404040!important}.btn-neutral:hover{background-color:#e5ebeb!important;color:#404040}.btn-neutral:visited{color:#404040!important}.btn-success{background-color:#27ae60!important}.btn-success:hover{background-color:#295!important}.btn-danger{background-color:#e74c3c!important}.btn-danger:hover{background-color:#ea6153!important}.btn-warning{background-color:#e67e22!important}.btn-warning:hover{background-color:#e98b39!important}.btn-invert{background-color:#222}.btn-invert:hover{background-color:#2f2f2f!important}.btn-link{background-color:transparent!important;color:#2980b9;box-shadow:none;border-color:transparent!important}.btn-link:active,.btn-link:hover{background-color:transparent!important;color:#409ad5!important;box-shadow:none}.btn-link:visited{color:#9b59b6}.wy-btn-group .btn,.wy-control .btn{vertical-align:middle}.wy-btn-group{margin-bottom:24px;*zoom:1}.wy-btn-group:after,.wy-btn-group:before{display:table;content:""}.wy-btn-group:after{clear:both}.wy-dropdown{position:relative;display:inline-block}.wy-dropdown-active .wy-dropdown-menu{display:block}.wy-dropdown-menu{position:absolute;left:0;display:none;float:left;top:100%;min-width:100%;background:#fcfcfc;z-index:100;border:1px solid #cfd7dd;box-shadow:0 2px 2px 0 rgba(0,0,0,.1);padding:12px}.wy-dropdown-menu>dd>a{display:block;clear:both;color:#404040;white-space:nowrap;font-size:90%;padding:0 12px;cursor:pointer}.wy-dropdown-menu>dd>a:hover{background:#2980b9;color:#fff}.wy-dropdown-menu>dd.divider{border-top:1px solid #cfd7dd;margin:6px 0}.wy-dropdown-menu>dd.search{padding-bottom:12px}.wy-dropdown-menu>dd.search input[type=search]{width:100%}.wy-dropdown-menu>dd.call-to-action{background:#e3e3e3;text-transform:uppercase;font-weight:500;font-size:80%}.wy-dropdown-menu>dd.call-to-action:hover{background:#e3e3e3}.wy-dropdown-menu>dd.call-to-action .btn{color:#fff}.wy-dropdown.wy-dropdown-up .wy-dropdown-menu{bottom:100%;top:auto;left:auto;right:0}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu{background:#fcfcfc;margin-top:2px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a{padding:6px 12px}.wy-dropdown.wy-dropdown-bubble .wy-dropdown-menu a:hover{background:#2980b9;color:#fff}.wy-dropdown.wy-dropdown-left .wy-dropdown-menu{right:0;left:auto;text-align:right}.wy-dropdown-arrow:before{content:" ";border-bottom:5px solid #f5f5f5;border-left:5px solid transparent;border-right:5px solid transparent;position:absolute;display:block;top:-4px;left:50%;margin-left:-3px}.wy-dropdown-arrow.wy-dropdown-arrow-left:before{left:11px}.wy-form-stacked select{display:block}.wy-form-aligned .wy-help-inline,.wy-form-aligned input,.wy-form-aligned label,.wy-form-aligned select,.wy-form-aligned textarea{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-form-aligned .wy-control-group>label{display:inline-block;vertical-align:middle;width:10em;margin:6px 12px 0 0;float:left}.wy-form-aligned .wy-control{float:left}.wy-form-aligned .wy-control label{display:block}.wy-form-aligned .wy-control select{margin-top:6px}fieldset{margin:0}fieldset,legend{border:0;padding:0}legend{width:100%;white-space:normal;margin-bottom:24px;font-size:150%;*margin-left:-7px}label,legend{display:block}label{margin:0 0 .3125em;color:#333;font-size:90%}input,select,textarea{font-size:100%;margin:0;vertical-align:baseline;*vertical-align:middle}.wy-control-group{margin-bottom:24px;max-width:1200px;margin-left:auto;margin-right:auto;*zoom:1}.wy-control-group:after,.wy-control-group:before{display:table;content:""}.wy-control-group:after{clear:both}.wy-control-group.wy-control-group-required>label:after{content:" *";color:#e74c3c}.wy-control-group .wy-form-full,.wy-control-group .wy-form-halves,.wy-control-group .wy-form-thirds{padding-bottom:12px}.wy-control-group .wy-form-full input[type=color],.wy-control-group .wy-form-full input[type=date],.wy-control-group .wy-form-full input[type=datetime-local],.wy-control-group .wy-form-full input[type=datetime],.wy-control-group .wy-form-full input[type=email],.wy-control-group .wy-form-full input[type=month],.wy-control-group .wy-form-full input[type=number],.wy-control-group .wy-form-full input[type=password],.wy-control-group .wy-form-full input[type=search],.wy-control-group .wy-form-full input[type=tel],.wy-control-group .wy-form-full input[type=text],.wy-control-group .wy-form-full input[type=time],.wy-control-group .wy-form-full input[type=url],.wy-control-group .wy-form-full input[type=week],.wy-control-group .wy-form-full select,.wy-control-group .wy-form-halves input[type=color],.wy-control-group .wy-form-halves input[type=date],.wy-control-group .wy-form-halves input[type=datetime-local],.wy-control-group .wy-form-halves input[type=datetime],.wy-control-group .wy-form-halves input[type=email],.wy-control-group .wy-form-halves input[type=month],.wy-control-group .wy-form-halves input[type=number],.wy-control-group .wy-form-halves input[type=password],.wy-control-group .wy-form-halves input[type=search],.wy-control-group .wy-form-halves input[type=tel],.wy-control-group .wy-form-halves input[type=text],.wy-control-group .wy-form-halves input[type=time],.wy-control-group .wy-form-halves input[type=url],.wy-control-group .wy-form-halves input[type=week],.wy-control-group .wy-form-halves select,.wy-control-group .wy-form-thirds input[type=color],.wy-control-group .wy-form-thirds input[type=date],.wy-control-group .wy-form-thirds input[type=datetime-local],.wy-control-group .wy-form-thirds input[type=datetime],.wy-control-group .wy-form-thirds input[type=email],.wy-control-group .wy-form-thirds input[type=month],.wy-control-group .wy-form-thirds input[type=number],.wy-control-group .wy-form-thirds input[type=password],.wy-control-group .wy-form-thirds input[type=search],.wy-control-group .wy-form-thirds input[type=tel],.wy-control-group .wy-form-thirds input[type=text],.wy-control-group .wy-form-thirds input[type=time],.wy-control-group .wy-form-thirds input[type=url],.wy-control-group .wy-form-thirds input[type=week],.wy-control-group .wy-form-thirds select{width:100%}.wy-control-group .wy-form-full{float:left;display:block;width:100%;margin-right:0}.wy-control-group .wy-form-full:last-child{margin-right:0}.wy-control-group .wy-form-halves{float:left;display:block;margin-right:2.35765%;width:48.82117%}.wy-control-group .wy-form-halves:last-child,.wy-control-group .wy-form-halves:nth-of-type(2n){margin-right:0}.wy-control-group .wy-form-halves:nth-of-type(odd){clear:left}.wy-control-group .wy-form-thirds{float:left;display:block;margin-right:2.35765%;width:31.76157%}.wy-control-group .wy-form-thirds:last-child,.wy-control-group .wy-form-thirds:nth-of-type(3n){margin-right:0}.wy-control-group .wy-form-thirds:nth-of-type(3n+1){clear:left}.wy-control-group.wy-control-group-no-input .wy-control,.wy-control-no-input{margin:6px 0 0;font-size:90%}.wy-control-no-input{display:inline-block}.wy-control-group.fluid-input input[type=color],.wy-control-group.fluid-input input[type=date],.wy-control-group.fluid-input input[type=datetime-local],.wy-control-group.fluid-input input[type=datetime],.wy-control-group.fluid-input input[type=email],.wy-control-group.fluid-input input[type=month],.wy-control-group.fluid-input input[type=number],.wy-control-group.fluid-input input[type=password],.wy-control-group.fluid-input input[type=search],.wy-control-group.fluid-input input[type=tel],.wy-control-group.fluid-input input[type=text],.wy-control-group.fluid-input input[type=time],.wy-control-group.fluid-input input[type=url],.wy-control-group.fluid-input input[type=week]{width:100%}.wy-form-message-inline{padding-left:.3em;color:#666;font-size:90%}.wy-form-message{display:block;color:#999;font-size:70%;margin-top:.3125em;font-style:italic}.wy-form-message p{font-size:inherit;font-style:italic;margin-bottom:6px}.wy-form-message p:last-child{margin-bottom:0}input{line-height:normal}input[type=button],input[type=reset],input[type=submit]{-webkit-appearance:button;cursor:pointer;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;*overflow:visible}input[type=color],input[type=date],input[type=datetime-local],input[type=datetime],input[type=email],input[type=month],input[type=number],input[type=password],input[type=search],input[type=tel],input[type=text],input[type=time],input[type=url],input[type=week]{-webkit-appearance:none;padding:6px;display:inline-block;border:1px solid #ccc;font-size:80%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;box-shadow:inset 0 1px 3px #ddd;border-radius:0;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}input[type=datetime-local]{padding:.34375em .625em}input[disabled]{cursor:default}input[type=checkbox],input[type=radio]{padding:0;margin-right:.3125em;*height:13px;*width:13px}input[type=checkbox],input[type=radio],input[type=search]{-webkit-box-sizing:border-box;-moz-box-sizing:border-box;box-sizing:border-box}input[type=search]::-webkit-search-cancel-button,input[type=search]::-webkit-search-decoration{-webkit-appearance:none}input[type=color]:focus,input[type=date]:focus,input[type=datetime-local]:focus,input[type=datetime]:focus,input[type=email]:focus,input[type=month]:focus,input[type=number]:focus,input[type=password]:focus,input[type=search]:focus,input[type=tel]:focus,input[type=text]:focus,input[type=time]:focus,input[type=url]:focus,input[type=week]:focus{outline:0;outline:thin dotted\9;border-color:#333}input.no-focus:focus{border-color:#ccc!important}input[type=checkbox]:focus,input[type=file]:focus,input[type=radio]:focus{outline:thin dotted #333;outline:1px auto #129fea}input[type=color][disabled],input[type=date][disabled],input[type=datetime-local][disabled],input[type=datetime][disabled],input[type=email][disabled],input[type=month][disabled],input[type=number][disabled],input[type=password][disabled],input[type=search][disabled],input[type=tel][disabled],input[type=text][disabled],input[type=time][disabled],input[type=url][disabled],input[type=week][disabled]{cursor:not-allowed;background-color:#fafafa}input:focus:invalid,select:focus:invalid,textarea:focus:invalid{color:#e74c3c;border:1px solid #e74c3c}input:focus:invalid:focus,select:focus:invalid:focus,textarea:focus:invalid:focus{border-color:#e74c3c}input[type=checkbox]:focus:invalid:focus,input[type=file]:focus:invalid:focus,input[type=radio]:focus:invalid:focus{outline-color:#e74c3c}input.wy-input-large{padding:12px;font-size:100%}textarea{overflow:auto;vertical-align:top;width:100%;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif}select,textarea{padding:.5em .625em;display:inline-block;border:1px solid #ccc;font-size:80%;box-shadow:inset 0 1px 3px #ddd;-webkit-transition:border .3s linear;-moz-transition:border .3s linear;transition:border .3s linear}select{border:1px solid #ccc;background-color:#fff}select[multiple]{height:auto}select:focus,textarea:focus{outline:0}input[readonly],select[disabled],select[readonly],textarea[disabled],textarea[readonly]{cursor:not-allowed;background-color:#fafafa}input[type=checkbox][disabled],input[type=radio][disabled]{cursor:not-allowed}.wy-checkbox,.wy-radio{margin:6px 0;color:#404040;display:block}.wy-checkbox input,.wy-radio input{vertical-align:baseline}.wy-form-message-inline{display:inline-block;*display:inline;*zoom:1;vertical-align:middle}.wy-input-prefix,.wy-input-suffix{white-space:nowrap;padding:6px}.wy-input-prefix .wy-input-context,.wy-input-suffix .wy-input-context{line-height:27px;padding:0 8px;display:inline-block;font-size:80%;background-color:#f3f6f6;border:1px solid #ccc;color:#999}.wy-input-suffix .wy-input-context{border-left:0}.wy-input-prefix .wy-input-context{border-right:0}.wy-switch{position:relative;display:block;height:24px;margin-top:12px;cursor:pointer}.wy-switch:before{left:0;top:0;width:36px;height:12px;background:#ccc}.wy-switch:after,.wy-switch:before{position:absolute;content:"";display:block;border-radius:4px;-webkit-transition:all .2s ease-in-out;-moz-transition:all .2s ease-in-out;transition:all .2s ease-in-out}.wy-switch:after{width:18px;height:18px;background:#999;left:-3px;top:-3px}.wy-switch span{position:absolute;left:48px;display:block;font-size:12px;color:#ccc;line-height:1}.wy-switch.active:before{background:#1e8449}.wy-switch.active:after{left:24px;background:#27ae60}.wy-switch.disabled{cursor:not-allowed;opacity:.8}.wy-control-group.wy-control-group-error .wy-form-message,.wy-control-group.wy-control-group-error>label{color:#e74c3c}.wy-control-group.wy-control-group-error input[type=color],.wy-control-group.wy-control-group-error input[type=date],.wy-control-group.wy-control-group-error input[type=datetime-local],.wy-control-group.wy-control-group-error input[type=datetime],.wy-control-group.wy-control-group-error input[type=email],.wy-control-group.wy-control-group-error input[type=month],.wy-control-group.wy-control-group-error input[type=number],.wy-control-group.wy-control-group-error input[type=password],.wy-control-group.wy-control-group-error input[type=search],.wy-control-group.wy-control-group-error input[type=tel],.wy-control-group.wy-control-group-error input[type=text],.wy-control-group.wy-control-group-error input[type=time],.wy-control-group.wy-control-group-error input[type=url],.wy-control-group.wy-control-group-error input[type=week],.wy-control-group.wy-control-group-error textarea{border:1px solid #e74c3c}.wy-inline-validate{white-space:nowrap}.wy-inline-validate .wy-input-context{padding:.5em .625em;display:inline-block;font-size:80%}.wy-inline-validate.wy-inline-validate-success .wy-input-context{color:#27ae60}.wy-inline-validate.wy-inline-validate-danger .wy-input-context{color:#e74c3c}.wy-inline-validate.wy-inline-validate-warning .wy-input-context{color:#e67e22}.wy-inline-validate.wy-inline-validate-info .wy-input-context{color:#2980b9}.rotate-90{-webkit-transform:rotate(90deg);-moz-transform:rotate(90deg);-ms-transform:rotate(90deg);-o-transform:rotate(90deg);transform:rotate(90deg)}.rotate-180{-webkit-transform:rotate(180deg);-moz-transform:rotate(180deg);-ms-transform:rotate(180deg);-o-transform:rotate(180deg);transform:rotate(180deg)}.rotate-270{-webkit-transform:rotate(270deg);-moz-transform:rotate(270deg);-ms-transform:rotate(270deg);-o-transform:rotate(270deg);transform:rotate(270deg)}.mirror{-webkit-transform:scaleX(-1);-moz-transform:scaleX(-1);-ms-transform:scaleX(-1);-o-transform:scaleX(-1);transform:scaleX(-1)}.mirror.rotate-90{-webkit-transform:scaleX(-1) rotate(90deg);-moz-transform:scaleX(-1) rotate(90deg);-ms-transform:scaleX(-1) rotate(90deg);-o-transform:scaleX(-1) rotate(90deg);transform:scaleX(-1) rotate(90deg)}.mirror.rotate-180{-webkit-transform:scaleX(-1) rotate(180deg);-moz-transform:scaleX(-1) rotate(180deg);-ms-transform:scaleX(-1) rotate(180deg);-o-transform:scaleX(-1) rotate(180deg);transform:scaleX(-1) rotate(180deg)}.mirror.rotate-270{-webkit-transform:scaleX(-1) rotate(270deg);-moz-transform:scaleX(-1) rotate(270deg);-ms-transform:scaleX(-1) rotate(270deg);-o-transform:scaleX(-1) rotate(270deg);transform:scaleX(-1) rotate(270deg)}@media only screen and (max-width:480px){.wy-form button[type=submit]{margin:.7em 0 0}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=text],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week],.wy-form label{margin-bottom:.3em;display:block}.wy-form input[type=color],.wy-form input[type=date],.wy-form input[type=datetime-local],.wy-form input[type=datetime],.wy-form input[type=email],.wy-form input[type=month],.wy-form input[type=number],.wy-form input[type=password],.wy-form input[type=search],.wy-form input[type=tel],.wy-form input[type=time],.wy-form input[type=url],.wy-form input[type=week]{margin-bottom:0}.wy-form-aligned .wy-control-group label{margin-bottom:.3em;text-align:left;display:block;width:100%}.wy-form-aligned .wy-control{margin:1.5em 0 0}.wy-form-message,.wy-form-message-inline,.wy-form .wy-help-inline{display:block;font-size:80%;padding:6px 0}}@media screen and (max-width:768px){.tablet-hide{display:none}}@media screen and (max-width:480px){.mobile-hide{display:none}}.float-left{float:left}.float-right{float:right}.full-width{width:100%}.rst-content table.docutils,.rst-content table.field-list,.wy-table{border-collapse:collapse;border-spacing:0;empty-cells:show;margin-bottom:24px}.rst-content table.docutils caption,.rst-content table.field-list caption,.wy-table caption{color:#000;font:italic 85%/1 arial,sans-serif;padding:1em 0;text-align:center}.rst-content table.docutils td,.rst-content table.docutils th,.rst-content table.field-list td,.rst-content table.field-list th,.wy-table td,.wy-table th{font-size:90%;margin:0;overflow:visible;padding:8px 16px}.rst-content table.docutils td:first-child,.rst-content table.docutils th:first-child,.rst-content table.field-list td:first-child,.rst-content table.field-list th:first-child,.wy-table td:first-child,.wy-table th:first-child{border-left-width:0}.rst-content table.docutils thead,.rst-content table.field-list thead,.wy-table thead{color:#000;text-align:left;vertical-align:bottom;white-space:nowrap}.rst-content table.docutils thead th,.rst-content table.field-list thead th,.wy-table thead th{font-weight:700;border-bottom:2px solid #e1e4e5}.rst-content table.docutils td,.rst-content table.field-list td,.wy-table td{background-color:transparent;vertical-align:middle}.rst-content table.docutils td p,.rst-content table.field-list td p,.wy-table td p{line-height:18px}.rst-content table.docutils td p:last-child,.rst-content table.field-list td p:last-child,.wy-table td p:last-child{margin-bottom:0}.rst-content table.docutils .wy-table-cell-min,.rst-content table.field-list .wy-table-cell-min,.wy-table .wy-table-cell-min{width:1%;padding-right:0}.rst-content table.docutils .wy-table-cell-min input[type=checkbox],.rst-content table.field-list .wy-table-cell-min input[type=checkbox],.wy-table .wy-table-cell-min input[type=checkbox]{margin:0}.wy-table-secondary{color:grey;font-size:90%}.wy-table-tertiary{color:grey;font-size:80%}.rst-content table.docutils:not(.field-list) tr:nth-child(2n-1) td,.wy-table-backed,.wy-table-odd td,.wy-table-striped tr:nth-child(2n-1) td{background-color:#f3f6f6}.rst-content table.docutils,.wy-table-bordered-all{border:1px solid #e1e4e5}.rst-content table.docutils td,.wy-table-bordered-all td{border-bottom:1px solid #e1e4e5;border-left:1px solid #e1e4e5}.rst-content table.docutils tbody>tr:last-child td,.wy-table-bordered-all tbody>tr:last-child td{border-bottom-width:0}.wy-table-bordered{border:1px solid #e1e4e5}.wy-table-bordered-rows td{border-bottom:1px solid #e1e4e5}.wy-table-bordered-rows tbody>tr:last-child td{border-bottom-width:0}.wy-table-horizontal td,.wy-table-horizontal th{border-width:0 0 1px;border-bottom:1px solid #e1e4e5}.wy-table-horizontal tbody>tr:last-child td{border-bottom-width:0}.wy-table-responsive{margin-bottom:24px;max-width:100%;overflow:auto}.wy-table-responsive table{margin-bottom:0!important}.wy-table-responsive table td,.wy-table-responsive table th{white-space:nowrap}a{color:#2980b9;text-decoration:none;cursor:pointer}a:hover{color:#3091d1}a:visited{color:#9b59b6}html{height:100%}body,html{overflow-x:hidden}body{font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;font-weight:400;color:#404040;min-height:100%;background:#edf0f2}.wy-text-left{text-align:left}.wy-text-center{text-align:center}.wy-text-right{text-align:right}.wy-text-large{font-size:120%}.wy-text-normal{font-size:100%}.wy-text-small,small{font-size:80%}.wy-text-strike{text-decoration:line-through}.wy-text-warning{color:#e67e22!important}a.wy-text-warning:hover{color:#eb9950!important}.wy-text-info{color:#2980b9!important}a.wy-text-info:hover{color:#409ad5!important}.wy-text-success{color:#27ae60!important}a.wy-text-success:hover{color:#36d278!important}.wy-text-danger{color:#e74c3c!important}a.wy-text-danger:hover{color:#ed7669!important}.wy-text-neutral{color:#404040!important}a.wy-text-neutral:hover{color:#595959!important}.rst-content .toctree-wrapper>p.caption,h1,h2,h3,h4,h5,h6,legend{margin-top:0;font-weight:700;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif}p{line-height:24px;font-size:16px;margin:0 0 24px}h1{font-size:175%}.rst-content .toctree-wrapper>p.caption,h2{font-size:150%}h3{font-size:125%}h4{font-size:115%}h5{font-size:110%}h6{font-size:100%}hr{display:block;height:1px;border:0;border-top:1px solid #e1e4e5;margin:24px 0;padding:0}.rst-content code,.rst-content tt,code{white-space:nowrap;max-width:100%;background:#fff;border:1px solid #e1e4e5;font-size:75%;padding:0 5px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;color:#e74c3c;overflow-x:auto}.rst-content tt.code-large,code.code-large{font-size:90%}.rst-content .section ul,.rst-content .toctree-wrapper ul,.wy-plain-list-disc,article ul{list-style:disc;line-height:24px;margin-bottom:24px}.rst-content .section ul li,.rst-content .toctree-wrapper ul li,.wy-plain-list-disc li,article ul li{list-style:disc;margin-left:24px}.rst-content .section ul li p:last-child,.rst-content .section ul li ul,.rst-content .toctree-wrapper ul li p:last-child,.rst-content .toctree-wrapper ul li ul,.wy-plain-list-disc li p:last-child,.wy-plain-list-disc li ul,article ul li p:last-child,article ul li ul{margin-bottom:0}.rst-content .section ul li li,.rst-content .toctree-wrapper ul li li,.wy-plain-list-disc li li,article ul li li{list-style:circle}.rst-content .section ul li li li,.rst-content .toctree-wrapper ul li li li,.wy-plain-list-disc li li li,article ul li li li{list-style:square}.rst-content .section ul li ol li,.rst-content .toctree-wrapper ul li ol li,.wy-plain-list-disc li ol li,article ul li ol li{list-style:decimal}.rst-content .section ol,.rst-content ol.arabic,.wy-plain-list-decimal,article ol{list-style:decimal;line-height:24px;margin-bottom:24px}.rst-content .section ol li,.rst-content ol.arabic li,.wy-plain-list-decimal li,article ol li{list-style:decimal;margin-left:24px}.rst-content .section ol li p:last-child,.rst-content .section ol li ul,.rst-content ol.arabic li p:last-child,.rst-content ol.arabic li ul,.wy-plain-list-decimal li p:last-child,.wy-plain-list-decimal li ul,article ol li p:last-child,article ol li ul{margin-bottom:0}.rst-content .section ol li ul li,.rst-content ol.arabic li ul li,.wy-plain-list-decimal li ul li,article ol li ul li{list-style:disc}.wy-breadcrumbs{*zoom:1}.wy-breadcrumbs:after,.wy-breadcrumbs:before{display:table;content:""}.wy-breadcrumbs:after{clear:both}.wy-breadcrumbs li{display:inline-block}.wy-breadcrumbs li.wy-breadcrumbs-aside{float:right}.wy-breadcrumbs li a{display:inline-block;padding:5px}.wy-breadcrumbs li a:first-child{padding-left:0}.rst-content .wy-breadcrumbs li tt,.wy-breadcrumbs li .rst-content tt,.wy-breadcrumbs li code{padding:5px;border:none;background:none}.rst-content .wy-breadcrumbs li tt.literal,.wy-breadcrumbs li .rst-content tt.literal,.wy-breadcrumbs li code.literal{color:#404040}.wy-breadcrumbs-extra{margin-bottom:0;color:#b3b3b3;font-size:80%;display:inline-block}@media screen and (max-width:480px){.wy-breadcrumbs-extra,.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}@media print{.wy-breadcrumbs li.wy-breadcrumbs-aside{display:none}}html{font-size:16px}.wy-affix{position:fixed;top:1.618em}.wy-menu a:hover{text-decoration:none}.wy-menu-horiz{*zoom:1}.wy-menu-horiz:after,.wy-menu-horiz:before{display:table;content:""}.wy-menu-horiz:after{clear:both}.wy-menu-horiz li,.wy-menu-horiz ul{display:inline-block}.wy-menu-horiz li:hover{background:hsla(0,0%,100%,.1)}.wy-menu-horiz li.divide-left{border-left:1px solid #404040}.wy-menu-horiz li.divide-right{border-right:1px solid #404040}.wy-menu-horiz a{height:32px;display:inline-block;line-height:32px;padding:0 16px}.wy-menu-vertical{width:300px}.wy-menu-vertical header,.wy-menu-vertical p.caption{color:#55a5d9;height:32px;line-height:32px;padding:0 1.618em;margin:12px 0 0;display:block;font-weight:700;text-transform:uppercase;font-size:85%;white-space:nowrap}.wy-menu-vertical ul{margin-bottom:0}.wy-menu-vertical li.divide-top{border-top:1px solid #404040}.wy-menu-vertical li.divide-bottom{border-bottom:1px solid #404040}.wy-menu-vertical li.current{background:#e3e3e3}.wy-menu-vertical li.current a{color:grey;border-right:1px solid #c9c9c9;padding:.4045em 2.427em}.wy-menu-vertical li.current a:hover{background:#d6d6d6}.rst-content .wy-menu-vertical li tt,.wy-menu-vertical li .rst-content tt,.wy-menu-vertical li code{border:none;background:inherit;color:inherit;padding-left:0;padding-right:0}.wy-menu-vertical li span.toctree-expand{display:block;float:left;margin-left:-1.2em;font-size:.8em;line-height:1.6em;color:#4d4d4d}.wy-menu-vertical li.current>a,.wy-menu-vertical li.on a{color:#404040;font-weight:700;position:relative;background:#fcfcfc;border:none;padding:.4045em 1.618em}.wy-menu-vertical li.current>a:hover,.wy-menu-vertical li.on a:hover{background:#fcfcfc}.wy-menu-vertical li.current>a:hover span.toctree-expand,.wy-menu-vertical li.on a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.current>a span.toctree-expand,.wy-menu-vertical li.on a span.toctree-expand{display:block;font-size:.8em;line-height:1.6em;color:#333}.wy-menu-vertical li.toctree-l1.current>a{border-bottom:1px solid #c9c9c9;border-top:1px solid #c9c9c9}.wy-menu-vertical .toctree-l1.current .toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .toctree-l11>ul{display:none}.wy-menu-vertical .toctree-l1.current .current.toctree-l2>ul,.wy-menu-vertical .toctree-l2.current .current.toctree-l3>ul,.wy-menu-vertical .toctree-l3.current .current.toctree-l4>ul,.wy-menu-vertical .toctree-l4.current .current.toctree-l5>ul,.wy-menu-vertical .toctree-l5.current .current.toctree-l6>ul,.wy-menu-vertical .toctree-l6.current .current.toctree-l7>ul,.wy-menu-vertical .toctree-l7.current .current.toctree-l8>ul,.wy-menu-vertical .toctree-l8.current .current.toctree-l9>ul,.wy-menu-vertical .toctree-l9.current .current.toctree-l10>ul,.wy-menu-vertical .toctree-l10.current .current.toctree-l11>ul{display:block}.wy-menu-vertical li.toctree-l3,.wy-menu-vertical li.toctree-l4{font-size:.9em}.wy-menu-vertical li.toctree-l2 a,.wy-menu-vertical li.toctree-l3 a,.wy-menu-vertical li.toctree-l4 a,.wy-menu-vertical li.toctree-l5 a,.wy-menu-vertical li.toctree-l6 a,.wy-menu-vertical li.toctree-l7 a,.wy-menu-vertical li.toctree-l8 a,.wy-menu-vertical li.toctree-l9 a,.wy-menu-vertical li.toctree-l10 a{color:#404040}.wy-menu-vertical li.toctree-l2 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l3 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l4 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l5 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l6 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l7 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l8 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l9 a:hover span.toctree-expand,.wy-menu-vertical li.toctree-l10 a:hover span.toctree-expand{color:grey}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{display:block}.wy-menu-vertical li.toctree-l2.current>a{padding:.4045em 2.427em}.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a,.wy-menu-vertical li.toctree-l3.current>a{padding:.4045em 4.045em}.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a,.wy-menu-vertical li.toctree-l4.current>a{padding:.4045em 5.663em}.wy-menu-vertical li.toctree-l4.current li.toctree-l5>a,.wy-menu-vertical li.toctree-l5.current>a{padding:.4045em 7.281em}.wy-menu-vertical li.toctree-l5.current li.toctree-l6>a,.wy-menu-vertical li.toctree-l6.current>a{padding:.4045em 8.899em}.wy-menu-vertical li.toctree-l6.current li.toctree-l7>a,.wy-menu-vertical li.toctree-l7.current>a{padding:.4045em 10.517em}.wy-menu-vertical li.toctree-l7.current li.toctree-l8>a,.wy-menu-vertical li.toctree-l8.current>a{padding:.4045em 12.135em}.wy-menu-vertical li.toctree-l8.current li.toctree-l9>a,.wy-menu-vertical li.toctree-l9.current>a{padding:.4045em 13.753em}.wy-menu-vertical li.toctree-l9.current li.toctree-l10>a,.wy-menu-vertical li.toctree-l10.current>a{padding:.4045em 15.371em}.wy-menu-vertical li.toctree-l10.current li.toctree-l11>a{padding:.4045em 16.989em}.wy-menu-vertical li.toctree-l2.current>a,.wy-menu-vertical li.toctree-l2.current li.toctree-l3>a{background:#c9c9c9}.wy-menu-vertical li.toctree-l2 span.toctree-expand{color:#a3a3a3}.wy-menu-vertical li.toctree-l3.current>a,.wy-menu-vertical li.toctree-l3.current li.toctree-l4>a{background:#bdbdbd}.wy-menu-vertical li.toctree-l3 span.toctree-expand{color:#969696}.wy-menu-vertical li.current ul{display:block}.wy-menu-vertical li ul{margin-bottom:0;display:none}.wy-menu-vertical li ul li a{margin-bottom:0;color:#d9d9d9;font-weight:400}.wy-menu-vertical a{line-height:18px;padding:.4045em 1.618em;display:block;position:relative;font-size:90%;color:#d9d9d9}.wy-menu-vertical a:hover{background-color:#4e4a4a;cursor:pointer}.wy-menu-vertical a:hover span.toctree-expand{color:#d9d9d9}.wy-menu-vertical a:active{background-color:#2980b9;cursor:pointer;color:#fff}.wy-menu-vertical a:active span.toctree-expand{color:#fff}.wy-side-nav-search{display:block;width:300px;padding:.809em;margin-bottom:.809em;z-index:200;background-color:#2980b9;text-align:center;color:#fcfcfc}.wy-side-nav-search input[type=text]{width:100%;border-radius:50px;padding:6px 12px;border-color:#2472a4}.wy-side-nav-search img{display:block;margin:auto auto .809em;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-side-nav-search .wy-dropdown>a,.wy-side-nav-search>a{color:#fcfcfc;font-size:100%;font-weight:700;display:inline-block;padding:4px 6px;margin-bottom:.809em}.wy-side-nav-search .wy-dropdown>a:hover,.wy-side-nav-search>a:hover{background:hsla(0,0%,100%,.1)}.wy-side-nav-search .wy-dropdown>a img.logo,.wy-side-nav-search>a img.logo{display:block;margin:0 auto;height:auto;width:auto;border-radius:0;max-width:100%;background:transparent}.wy-side-nav-search .wy-dropdown>a.icon img.logo,.wy-side-nav-search>a.icon img.logo{margin-top:.85em}.wy-side-nav-search>div.version{margin-top:-.4045em;margin-bottom:.809em;font-weight:400;color:hsla(0,0%,100%,.3)}.wy-nav .wy-menu-vertical header{color:#2980b9}.wy-nav .wy-menu-vertical a{color:#b3b3b3}.wy-nav .wy-menu-vertical a:hover{background-color:#2980b9;color:#fff}[data-menu-wrap]{-webkit-transition:all .2s ease-in;-moz-transition:all .2s ease-in;transition:all .2s ease-in;position:absolute;opacity:1;width:100%;opacity:0}[data-menu-wrap].move-center{left:0;right:auto;opacity:1}[data-menu-wrap].move-left{right:auto;left:-100%;opacity:0}[data-menu-wrap].move-right{right:-100%;left:auto;opacity:0}.wy-body-for-nav{background:#fcfcfc}.wy-grid-for-nav{position:absolute;width:100%;height:100%}.wy-nav-side{position:fixed;top:0;bottom:0;left:0;padding-bottom:2em;width:300px;overflow-x:hidden;overflow-y:hidden;min-height:100%;color:#9b9b9b;background:#343131;z-index:200}.wy-side-scroll{width:320px;position:relative;overflow-x:hidden;overflow-y:scroll;height:100%}.wy-nav-top{display:none;background:#2980b9;color:#fff;padding:.4045em .809em;position:relative;line-height:50px;text-align:center;font-size:100%;*zoom:1}.wy-nav-top:after,.wy-nav-top:before{display:table;content:""}.wy-nav-top:after{clear:both}.wy-nav-top a{color:#fff;font-weight:700}.wy-nav-top img{margin-right:12px;height:45px;width:45px;background-color:#2980b9;padding:5px;border-radius:100%}.wy-nav-top i{font-size:30px;float:left;cursor:pointer;padding-top:inherit}.wy-nav-content-wrap{margin-left:300px;background:#fcfcfc;min-height:100%}.wy-nav-content{padding:1.618em 3.236em;height:100%;max-width:800px;margin:auto}.wy-body-mask{position:fixed;width:100%;height:100%;background:rgba(0,0,0,.2);display:none;z-index:499}.wy-body-mask.on{display:block}footer{color:grey}footer p{margin-bottom:12px}.rst-content footer span.commit tt,footer span.commit .rst-content tt,footer span.commit code{padding:0;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:1em;background:none;border:none;color:grey}.rst-footer-buttons{*zoom:1}.rst-footer-buttons:after,.rst-footer-buttons:before{width:100%;display:table;content:""}.rst-footer-buttons:after{clear:both}.rst-breadcrumbs-buttons{margin-top:12px;*zoom:1}.rst-breadcrumbs-buttons:after,.rst-breadcrumbs-buttons:before{display:table;content:""}.rst-breadcrumbs-buttons:after{clear:both}#search-results .search li{margin-bottom:24px;border-bottom:1px solid #e1e4e5;padding-bottom:24px}#search-results .search li:first-child{border-top:1px solid #e1e4e5;padding-top:24px}#search-results .search li a{font-size:120%;margin-bottom:12px;display:inline-block}#search-results .context{color:grey;font-size:90%}.genindextable li>ul{margin-left:24px}@media screen and (max-width:768px){.wy-body-for-nav{background:#fcfcfc}.wy-nav-top{display:block}.wy-nav-side{left:-300px}.wy-nav-side.shift{width:85%;left:0}.wy-menu.wy-menu-vertical,.wy-side-nav-search,.wy-side-scroll{width:auto}.wy-nav-content-wrap{margin-left:0}.wy-nav-content-wrap .wy-nav-content{padding:1.618em}.wy-nav-content-wrap.shift{position:fixed;min-width:100%;left:85%;top:0;height:100%;overflow:hidden}}@media screen and (min-width:1100px){.wy-nav-content-wrap{background:rgba(0,0,0,.05)}.wy-nav-content{margin:0;background:#fcfcfc}}@media print{.rst-versions,.wy-nav-side,footer{display:none}.wy-nav-content-wrap{margin-left:0}}.rst-versions{position:fixed;bottom:0;left:0;width:300px;color:#fcfcfc;background:#1f1d1d;font-family:Lato,proxima-nova,Helvetica Neue,Arial,sans-serif;z-index:400}.rst-versions a{color:#2980b9;text-decoration:none}.rst-versions .rst-badge-small{display:none}.rst-versions .rst-current-version{padding:12px;background-color:#272525;display:block;text-align:right;font-size:90%;cursor:pointer;color:#27ae60;*zoom:1}.rst-versions .rst-current-version:after,.rst-versions .rst-current-version:before{display:table;content:""}.rst-versions .rst-current-version:after{clear:both}.rst-content .code-block-caption .rst-versions .rst-current-version .headerlink,.rst-content .rst-versions .rst-current-version .admonition-title,.rst-content code.download .rst-versions .rst-current-version span:first-child,.rst-content dl dt .rst-versions .rst-current-version .headerlink,.rst-content h1 .rst-versions .rst-current-version .headerlink,.rst-content h2 .rst-versions .rst-current-version .headerlink,.rst-content h3 .rst-versions .rst-current-version .headerlink,.rst-content h4 .rst-versions .rst-current-version .headerlink,.rst-content h5 .rst-versions .rst-current-version .headerlink,.rst-content h6 .rst-versions .rst-current-version .headerlink,.rst-content p.caption .rst-versions .rst-current-version .headerlink,.rst-content table>caption .rst-versions .rst-current-version .headerlink,.rst-content tt.download .rst-versions .rst-current-version span:first-child,.rst-versions .rst-current-version .fa,.rst-versions .rst-current-version .icon,.rst-versions .rst-current-version .rst-content .admonition-title,.rst-versions .rst-current-version .rst-content .code-block-caption .headerlink,.rst-versions .rst-current-version .rst-content code.download span:first-child,.rst-versions .rst-current-version .rst-content dl dt .headerlink,.rst-versions .rst-current-version .rst-content h1 .headerlink,.rst-versions .rst-current-version .rst-content h2 .headerlink,.rst-versions .rst-current-version .rst-content h3 .headerlink,.rst-versions .rst-current-version .rst-content h4 .headerlink,.rst-versions .rst-current-version .rst-content h5 .headerlink,.rst-versions .rst-current-version .rst-content h6 .headerlink,.rst-versions .rst-current-version .rst-content p.caption .headerlink,.rst-versions .rst-current-version .rst-content table>caption .headerlink,.rst-versions .rst-current-version .rst-content tt.download span:first-child,.rst-versions .rst-current-version .wy-menu-vertical li span.toctree-expand,.wy-menu-vertical li .rst-versions .rst-current-version span.toctree-expand{color:#fcfcfc}.rst-versions .rst-current-version .fa-book,.rst-versions .rst-current-version .icon-book{float:left}.rst-versions .rst-current-version.rst-out-of-date{background-color:#e74c3c;color:#fff}.rst-versions .rst-current-version.rst-active-old-version{background-color:#f1c40f;color:#000}.rst-versions.shift-up{height:auto;max-height:100%;overflow-y:scroll}.rst-versions.shift-up .rst-other-versions{display:block}.rst-versions .rst-other-versions{font-size:90%;padding:12px;color:grey;display:none}.rst-versions .rst-other-versions hr{display:block;height:1px;border:0;margin:20px 0;padding:0;border-top:1px solid #413d3d}.rst-versions .rst-other-versions dd{display:inline-block;margin:0}.rst-versions .rst-other-versions dd a{display:inline-block;padding:6px;color:#fcfcfc}.rst-versions.rst-badge{width:auto;bottom:20px;right:20px;left:auto;border:none;max-width:300px;max-height:90%}.rst-versions.rst-badge .fa-book,.rst-versions.rst-badge .icon-book{float:none;line-height:30px}.rst-versions.rst-badge.shift-up .rst-current-version{text-align:right}.rst-versions.rst-badge.shift-up .rst-current-version .fa-book,.rst-versions.rst-badge.shift-up .rst-current-version .icon-book{float:left}.rst-versions.rst-badge>.rst-current-version{width:auto;height:30px;line-height:30px;padding:0 6px;display:block;text-align:center}@media screen and (max-width:768px){.rst-versions{width:85%;display:none}.rst-versions.shift{display:block}}.rst-content img{max-width:100%;height:auto}.rst-content div.figure{margin-bottom:24px}.rst-content div.figure p.caption{font-style:italic}.rst-content div.figure p:last-child.caption{margin-bottom:0}.rst-content div.figure.align-center{text-align:center}.rst-content .section>a>img,.rst-content .section>img{margin-bottom:24px}.rst-content abbr[title]{text-decoration:none}.rst-content.style-external-links a.reference.external:after{font-family:FontAwesome;content:"\f08e";color:#b3b3b3;vertical-align:super;font-size:60%;margin:0 .2em}.rst-content blockquote{margin-left:24px;line-height:24px;margin-bottom:24px}.rst-content pre.literal-block{white-space:pre;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;display:block;overflow:auto}.rst-content div[class^=highlight],.rst-content pre.literal-block{border:1px solid #e1e4e5;overflow-x:auto;margin:1px 0 24px}.rst-content div[class^=highlight] div[class^=highlight],.rst-content pre.literal-block div[class^=highlight]{padding:0;border:none;margin:0}.rst-content div[class^=highlight] td.code{width:100%}.rst-content .linenodiv pre{border-right:1px solid #e6e9ea;margin:0;padding:12px;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;user-select:none;pointer-events:none}.rst-content div[class^=highlight] pre{white-space:pre;margin:0;padding:12px;display:block;overflow:auto}.rst-content div[class^=highlight] pre .hll{display:block;margin:0 -12px;padding:0 12px}.rst-content .linenodiv pre,.rst-content div[class^=highlight] pre,.rst-content pre.literal-block{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;font-size:12px;line-height:1.4}.rst-content div.highlight .gp{user-select:none;pointer-events:none}.rst-content .code-block-caption{font-style:italic;font-size:85%;line-height:1;padding:1em 0;text-align:center}@media print{.rst-content .codeblock,.rst-content div[class^=highlight],.rst-content div[class^=highlight] pre{white-space:pre-wrap}}.rst-content .admonition,.rst-content .admonition-todo,.rst-content .attention,.rst-content .caution,.rst-content .danger,.rst-content .error,.rst-content .hint,.rst-content .important,.rst-content .note,.rst-content .seealso,.rst-content .tip,.rst-content .warning{clear:both}.rst-content .admonition-todo .last,.rst-content .admonition-todo>:last-child,.rst-content .admonition .last,.rst-content .admonition>:last-child,.rst-content .attention .last,.rst-content .attention>:last-child,.rst-content .caution .last,.rst-content .caution>:last-child,.rst-content .danger .last,.rst-content .danger>:last-child,.rst-content .error .last,.rst-content .error>:last-child,.rst-content .hint .last,.rst-content .hint>:last-child,.rst-content .important .last,.rst-content .important>:last-child,.rst-content .note .last,.rst-content .note>:last-child,.rst-content .seealso .last,.rst-content .seealso>:last-child,.rst-content .tip .last,.rst-content .tip>:last-child,.rst-content .warning .last,.rst-content .warning>:last-child{margin-bottom:0}.rst-content .admonition-title:before{margin-right:4px}.rst-content .admonition table{border-color:rgba(0,0,0,.1)}.rst-content .admonition table td,.rst-content .admonition table th{background:transparent!important;border-color:rgba(0,0,0,.1)!important}.rst-content .section ol.loweralpha,.rst-content .section ol.loweralpha>li{list-style:lower-alpha}.rst-content .section ol.upperalpha,.rst-content .section ol.upperalpha>li{list-style:upper-alpha}.rst-content .section ol li>*,.rst-content .section ul li>*{margin-top:12px;margin-bottom:12px}.rst-content .section ol li>:first-child,.rst-content .section ul li>:first-child{margin-top:0}.rst-content .section ol li>p,.rst-content .section ol li>p:last-child,.rst-content .section ul li>p,.rst-content .section ul li>p:last-child{margin-bottom:12px}.rst-content .section ol li>p:only-child,.rst-content .section ol li>p:only-child:last-child,.rst-content .section ul li>p:only-child,.rst-content .section ul li>p:only-child:last-child{margin-bottom:0}.rst-content .section ol li>ol,.rst-content .section ol li>ul,.rst-content .section ul li>ol,.rst-content .section ul li>ul{margin-bottom:12px}.rst-content .section ol.simple li>*,.rst-content .section ol.simple li ol,.rst-content .section ol.simple li ul,.rst-content .section ul.simple li>*,.rst-content .section ul.simple li ol,.rst-content .section ul.simple li ul{margin-top:0;margin-bottom:0}.rst-content .line-block{margin-left:0;margin-bottom:24px;line-height:24px}.rst-content .line-block .line-block{margin-left:24px;margin-bottom:0}.rst-content .topic-title{font-weight:700;margin-bottom:12px}.rst-content .toc-backref{color:#404040}.rst-content .align-right{float:right;margin:0 0 24px 24px}.rst-content .align-left{float:left;margin:0 24px 24px 0}.rst-content .align-center{margin:auto}.rst-content .align-center:not(table){display:block}.rst-content .code-block-caption .headerlink,.rst-content .toctree-wrapper>p.caption .headerlink,.rst-content dl dt .headerlink,.rst-content h1 .headerlink,.rst-content h2 .headerlink,.rst-content h3 .headerlink,.rst-content h4 .headerlink,.rst-content h5 .headerlink,.rst-content h6 .headerlink,.rst-content p.caption .headerlink,.rst-content table>caption .headerlink{visibility:hidden;font-size:14px}.rst-content .code-block-caption .headerlink:after,.rst-content .toctree-wrapper>p.caption .headerlink:after,.rst-content dl dt .headerlink:after,.rst-content h1 .headerlink:after,.rst-content h2 .headerlink:after,.rst-content h3 .headerlink:after,.rst-content h4 .headerlink:after,.rst-content h5 .headerlink:after,.rst-content h6 .headerlink:after,.rst-content p.caption .headerlink:after,.rst-content table>caption .headerlink:after{content:"\f0c1";font-family:FontAwesome}.rst-content .code-block-caption:hover .headerlink:after,.rst-content .toctree-wrapper>p.caption:hover .headerlink:after,.rst-content dl dt:hover .headerlink:after,.rst-content h1:hover .headerlink:after,.rst-content h2:hover .headerlink:after,.rst-content h3:hover .headerlink:after,.rst-content h4:hover .headerlink:after,.rst-content h5:hover .headerlink:after,.rst-content h6:hover .headerlink:after,.rst-content p.caption:hover .headerlink:after,.rst-content table>caption:hover .headerlink:after{visibility:visible}.rst-content table>caption .headerlink:after{font-size:12px}.rst-content .centered{text-align:center}.rst-content .sidebar{float:right;width:40%;display:block;margin:0 0 24px 24px;padding:24px;background:#f3f6f6;border:1px solid #e1e4e5}.rst-content .sidebar dl,.rst-content .sidebar p,.rst-content .sidebar ul{font-size:90%}.rst-content .sidebar .last,.rst-content .sidebar>:last-child{margin-bottom:0}.rst-content .sidebar .sidebar-title{display:block;font-family:Roboto Slab,ff-tisa-web-pro,Georgia,Arial,sans-serif;font-weight:700;background:#e1e4e5;padding:6px 12px;margin:-24px -24px 24px;font-size:100%}.rst-content .highlighted{background:#f1c40f;box-shadow:0 0 0 2px #f1c40f;display:inline;font-weight:700}.rst-content .citation-reference,.rst-content .footnote-reference{vertical-align:baseline;position:relative;top:-.4em;line-height:0;font-size:90%}.rst-content .hlist{width:100%}html.writer-html4 .rst-content table.docutils.citation,html.writer-html4 .rst-content table.docutils.footnote{background:none;border:none}html.writer-html4 .rst-content table.docutils.citation td,html.writer-html4 .rst-content table.docutils.citation tr,html.writer-html4 .rst-content table.docutils.footnote td,html.writer-html4 .rst-content table.docutils.footnote tr{border:none;background-color:transparent!important;white-space:normal}html.writer-html4 .rst-content table.docutils.citation td.label,html.writer-html4 .rst-content table.docutils.footnote td.label{padding-left:0;padding-right:0;vertical-align:top}html.writer-html5 .rst-content dl dt span.classifier:before{content:" : "}html.writer-html5 .rst-content dl.field-list,html.writer-html5 .rst-content dl.footnote{display:grid;grid-template-columns:max-content auto}html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dt{padding-left:1rem}html.writer-html5 .rst-content dl.field-list>dt:after,html.writer-html5 .rst-content dl.footnote>dt:after{content:":"}html.writer-html5 .rst-content dl.field-list>dd,html.writer-html5 .rst-content dl.field-list>dt,html.writer-html5 .rst-content dl.footnote>dd,html.writer-html5 .rst-content dl.footnote>dt{margin-bottom:0}html.writer-html5 .rst-content dl.footnote{font-size:.9rem}html.writer-html5 .rst-content dl.footnote>dt{margin:0 .5rem .5rem 0;line-height:1.2rem;word-break:break-all;font-weight:400}html.writer-html5 .rst-content dl.footnote>dt>span.brackets{margin-right:.5rem}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:before{content:"["}html.writer-html5 .rst-content dl.footnote>dt>span.brackets:after{content:"]"}html.writer-html5 .rst-content dl.footnote>dt>span.fn-backref{font-style:italic}html.writer-html5 .rst-content dl.footnote>dd{margin:0 0 .5rem;line-height:1.2rem}html.writer-html5 .rst-content dl.footnote>dd p,html.writer-html5 .rst-content dl.option-list kbd{font-size:.9rem}.rst-content table.docutils.footnote,html.writer-html4 .rst-content table.docutils.citation,html.writer-html5 .rst-content dl.footnote{color:grey}.rst-content table.docutils.footnote code,.rst-content table.docutils.footnote tt,html.writer-html4 .rst-content table.docutils.citation code,html.writer-html4 .rst-content table.docutils.citation tt,html.writer-html5 .rst-content dl.footnote code,html.writer-html5 .rst-content dl.footnote tt{color:#555}.rst-content .wy-table-responsive.citation,.rst-content .wy-table-responsive.footnote{margin-bottom:0}.rst-content .wy-table-responsive.citation+:not(.citation),.rst-content .wy-table-responsive.footnote+:not(.footnote){margin-top:24px}.rst-content .wy-table-responsive.citation:last-child,.rst-content .wy-table-responsive.footnote:last-child{margin-bottom:24px}.rst-content table.docutils th{border-color:#e1e4e5}html.writer-html5 .rst-content table.docutils th{border:1px solid #e1e4e5}html.writer-html5 .rst-content table.docutils td>p,html.writer-html5 .rst-content table.docutils th>p{line-height:1rem;margin-bottom:0;font-size:.9rem}.rst-content table.docutils td .last,.rst-content table.docutils td .last>:last-child{margin-bottom:0}.rst-content table.field-list,.rst-content table.field-list td{border:none}.rst-content table.field-list td p{font-size:inherit;line-height:inherit}.rst-content table.field-list td>strong{display:inline-block}.rst-content table.field-list .field-name{padding-right:10px;text-align:left;white-space:nowrap}.rst-content table.field-list .field-body{text-align:left}.rst-content code,.rst-content tt{color:#000;font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace;padding:2px 5px}.rst-content code big,.rst-content code em,.rst-content tt big,.rst-content tt em{font-size:100%!important;line-height:normal}.rst-content code.literal,.rst-content tt.literal{color:#e74c3c}.rst-content code.xref,.rst-content tt.xref,a .rst-content code,a .rst-content tt{font-weight:700;color:#404040}.rst-content kbd,.rst-content pre,.rst-content samp{font-family:SFMono-Regular,Menlo,Monaco,Consolas,Liberation Mono,Courier New,Courier,monospace}.rst-content a code,.rst-content a tt{color:#2980b9}.rst-content dl{margin-bottom:24px}.rst-content dl dt{font-weight:700;margin-bottom:12px}.rst-content dl ol,.rst-content dl p,.rst-content dl table,.rst-content dl ul{margin-bottom:12px}.rst-content dl dd{margin:0 0 12px 24px;line-height:24px}html.writer-html4 .rst-content dl:not(.docutils),html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple){margin-bottom:24px}html.writer-html4 .rst-content dl:not(.docutils)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt{display:table;margin:6px 0;font-size:90%;line-height:normal;background:#e7f2fa;color:#2980b9;border-top:3px solid #6ab0de;padding:6px;position:relative}html.writer-html4 .rst-content dl:not(.docutils)>dt:before,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:before{color:#6ab0de}html.writer-html4 .rst-content dl:not(.docutils)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt{margin-bottom:6px;border:none;border-left:3px solid #ccc;background:#f0f0f0;color:#555}html.writer-html4 .rst-content dl:not(.docutils) dl:not(.field-list)>dt .headerlink,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) dl:not(.field-list)>dt .headerlink{color:#404040;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils)>dt:first-child,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple)>dt:first-child{margin-top:0}html.writer-html4 .rst-content dl:not(.docutils) code,html.writer-html4 .rst-content dl:not(.docutils) tt,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) code.descclassname,html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descclassname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descclassname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{background-color:transparent;border:none;padding:0;font-size:100%!important}html.writer-html4 .rst-content dl:not(.docutils) code.descname,html.writer-html4 .rst-content dl:not(.docutils) tt.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) code.descname,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) tt.descname{font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .optional,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .optional{display:inline-block;padding:0 4px;color:#000;font-weight:700}html.writer-html4 .rst-content dl:not(.docutils) .property,html.writer-html5 .rst-content dl[class]:not(.option-list):not(.field-list):not(.footnote):not(.glossary):not(.simple) .property{display:inline-block;padding-right:8px}.rst-content .viewcode-back,.rst-content .viewcode-link{display:inline-block;color:#27ae60;font-size:80%;padding-left:24px}.rst-content .viewcode-back{display:block;float:right}.rst-content p.rubric{margin-bottom:12px;font-weight:700}.rst-content code.download,.rst-content tt.download{background:inherit;padding:inherit;font-weight:400;font-family:inherit;font-size:inherit;color:inherit;border:inherit;white-space:inherit}.rst-content code.download span:first-child,.rst-content tt.download span:first-child{-webkit-font-smoothing:subpixel-antialiased}.rst-content code.download span:first-child:before,.rst-content tt.download span:first-child:before{margin-right:4px}.rst-content .guilabel{border:1px solid #7fbbe3;background:#e7f2fa;font-size:80%;font-weight:700;border-radius:4px;padding:2.4px 6px;margin:auto 2px}.rst-content .versionmodified{font-style:italic}@media screen and (max-width:480px){.rst-content .sidebar{width:100%}}span[id*=MathJax-Span]{color:#404040}.math{text-align:center}@font-face{font-family:Lato;src:url(fonts/lato-normal.woff2?bd03a2cc277bbbc338d464e679fe9942) format("woff2"),url(fonts/lato-normal.woff?27bd77b9162d388cb8d4c4217c7c5e2a) format("woff");font-weight:400;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold.woff2?cccb897485813c7c256901dbca54ecf2) format("woff2"),url(fonts/lato-bold.woff?d878b6c29b10beca227e9eef4246111b) format("woff");font-weight:700;font-style:normal;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-bold-italic.woff2?0b6bb6725576b072c5d0b02ecdd1900d) format("woff2"),url(fonts/lato-bold-italic.woff?9c7e4e9eb485b4a121c760e61bc3707c) format("woff");font-weight:700;font-style:italic;font-display:block}@font-face{font-family:Lato;src:url(fonts/lato-normal-italic.woff2?4eb103b4d12be57cb1d040ed5e162e9d) format("woff2"),url(fonts/lato-normal-italic.woff?f28f2d6482446544ef1ea1ccc6dd5892) format("woff");font-weight:400;font-style:italic;font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:400;src:url(fonts/Roboto-Slab-Regular.woff2?7abf5b8d04d26a2cafea937019bca958) format("woff2"),url(fonts/Roboto-Slab-Regular.woff?c1be9284088d487c5e3ff0a10a92e58c) format("woff");font-display:block}@font-face{font-family:Roboto Slab;font-style:normal;font-weight:700;src:url(fonts/Roboto-Slab-Bold.woff2?9984f4a9bda09be08e83f2506954adbe) format("woff2"),url(fonts/Roboto-Slab-Bold.woff?bed5564a116b05148e3b3bea6fb1162a) format("woff");font-display:block} \ No newline at end of file diff --git a/en/latest/_static/doctools.js b/en/latest/_static/doctools.js new file mode 100644 index 0000000000..527b876ca6 --- /dev/null +++ b/en/latest/_static/doctools.js @@ -0,0 +1,156 @@ +/* + * doctools.js + * ~~~~~~~~~~~ + * + * Base JavaScript utilities for all Sphinx HTML documentation. + * + * :copyright: Copyright 2007-2022 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ +"use strict"; + +const BLACKLISTED_KEY_CONTROL_ELEMENTS = new Set([ + "TEXTAREA", + "INPUT", + "SELECT", + "BUTTON", +]); + +const _ready = (callback) => { + if (document.readyState !== "loading") { + callback(); + } else { + document.addEventListener("DOMContentLoaded", callback); + } +}; + +/** + * Small JavaScript module for the documentation. + */ +const Documentation = { + init: () => { + Documentation.initDomainIndexTable(); + Documentation.initOnKeyListeners(); + }, + + /** + * i18n support + */ + TRANSLATIONS: {}, + PLURAL_EXPR: (n) => (n === 1 ? 0 : 1), + LOCALE: "unknown", + + // gettext and ngettext don't access this so that the functions + // can safely bound to a different name (_ = Documentation.gettext) + gettext: (string) => { + const translated = Documentation.TRANSLATIONS[string]; + switch (typeof translated) { + case "undefined": + return string; // no translation + case "string": + return translated; // translation exists + default: + return translated[0]; // (singular, plural) translation tuple exists + } + }, + + ngettext: (singular, plural, n) => { + const translated = Documentation.TRANSLATIONS[singular]; + if (typeof translated !== "undefined") + return translated[Documentation.PLURAL_EXPR(n)]; + return n === 1 ? singular : plural; + }, + + addTranslations: (catalog) => { + Object.assign(Documentation.TRANSLATIONS, catalog.messages); + Documentation.PLURAL_EXPR = new Function( + "n", + `return (${catalog.plural_expr})` + ); + Documentation.LOCALE = catalog.locale; + }, + + /** + * helper function to focus on search bar + */ + focusSearchBar: () => { + document.querySelectorAll("input[name=q]")[0]?.focus(); + }, + + /** + * Initialise the domain index toggle buttons + */ + initDomainIndexTable: () => { + const toggler = (el) => { + const idNumber = el.id.substr(7); + const toggledRows = document.querySelectorAll(`tr.cg-${idNumber}`); + if (el.src.substr(-9) === "minus.png") { + el.src = `${el.src.substr(0, el.src.length - 9)}plus.png`; + toggledRows.forEach((el) => (el.style.display = "none")); + } else { + el.src = `${el.src.substr(0, el.src.length - 8)}minus.png`; + toggledRows.forEach((el) => (el.style.display = "")); + } + }; + + const togglerElements = document.querySelectorAll("img.toggler"); + togglerElements.forEach((el) => + el.addEventListener("click", (event) => toggler(event.currentTarget)) + ); + togglerElements.forEach((el) => (el.style.display = "")); + if (DOCUMENTATION_OPTIONS.COLLAPSE_INDEX) togglerElements.forEach(toggler); + }, + + initOnKeyListeners: () => { + // only install a listener if it is really needed + if ( + !DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS && + !DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS + ) + return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.altKey || event.ctrlKey || event.metaKey) return; + + if (!event.shiftKey) { + switch (event.key) { + case "ArrowLeft": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const prevLink = document.querySelector('link[rel="prev"]'); + if (prevLink && prevLink.href) { + window.location.href = prevLink.href; + event.preventDefault(); + } + break; + case "ArrowRight": + if (!DOCUMENTATION_OPTIONS.NAVIGATION_WITH_KEYS) break; + + const nextLink = document.querySelector('link[rel="next"]'); + if (nextLink && nextLink.href) { + window.location.href = nextLink.href; + event.preventDefault(); + } + break; + } + } + + // some keyboard layouts may need Shift to get / + switch (event.key) { + case "/": + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) break; + Documentation.focusSearchBar(); + event.preventDefault(); + } + }); + }, +}; + +// quick alias for translations +const _ = Documentation.gettext; + +_ready(Documentation.init); diff --git a/en/latest/_static/documentation_options.js b/en/latest/_static/documentation_options.js new file mode 100644 index 0000000000..f9961581ec --- /dev/null +++ b/en/latest/_static/documentation_options.js @@ -0,0 +1,14 @@ +var DOCUMENTATION_OPTIONS = { + URL_ROOT: document.getElementById("documentation_options").getAttribute('data-url_root'), + VERSION: 'Netris v3.0', + LANGUAGE: 'en', + COLLAPSE_INDEX: false, + BUILDER: 'html', + FILE_SUFFIX: '.html', + LINK_SUFFIX: '.html', + HAS_SOURCE: true, + SOURCELINK_SUFFIX: '.txt', + NAVIGATION_WITH_KEYS: false, + SHOW_SEARCH_SUMMARY: true, + ENABLE_SEARCH_SHORTCUTS: true, +}; \ No newline at end of file diff --git a/en/latest/_static/file.png b/en/latest/_static/file.png new file mode 100644 index 0000000000..a858a410e4 Binary files /dev/null and b/en/latest/_static/file.png differ diff --git a/en/latest/_static/jquery-3.6.0.js b/en/latest/_static/jquery-3.6.0.js new file mode 100644 index 0000000000..fc6c299b73 --- /dev/null +++ b/en/latest/_static/jquery-3.6.0.js @@ -0,0 +1,10881 @@ +/*! + * jQuery JavaScript Library v3.6.0 + * https://jquery.com/ + * + * Includes Sizzle.js + * https://sizzlejs.com/ + * + * Copyright OpenJS Foundation and other contributors + * Released under the MIT license + * https://jquery.org/license + * + * Date: 2021-03-02T17:08Z + */ +( function( global, factory ) { + + "use strict"; + + if ( typeof module === "object" && typeof module.exports === "object" ) { + + // For CommonJS and CommonJS-like environments where a proper `window` + // is present, execute the factory and get jQuery. + // For environments that do not have a `window` with a `document` + // (such as Node.js), expose a factory as module.exports. + // This accentuates the need for the creation of a real `window`. + // e.g. var jQuery = require("jquery")(window); + // See ticket #14549 for more info. + module.exports = global.document ? + factory( global, true ) : + function( w ) { + if ( !w.document ) { + throw new Error( "jQuery requires a window with a document" ); + } + return factory( w ); + }; + } else { + factory( global ); + } + +// Pass this if window is not defined yet +} )( typeof window !== "undefined" ? window : this, function( window, noGlobal ) { + +// Edge <= 12 - 13+, Firefox <=18 - 45+, IE 10 - 11, Safari 5.1 - 9+, iOS 6 - 9.1 +// throw exceptions when non-strict code (e.g., ASP.NET 4.5) accesses strict mode +// arguments.callee.caller (trac-13335). But as of jQuery 3.0 (2016), strict mode should be common +// enough that all such attempts are guarded in a try block. +"use strict"; + +var arr = []; + +var getProto = Object.getPrototypeOf; + +var slice = arr.slice; + +var flat = arr.flat ? function( array ) { + return arr.flat.call( array ); +} : function( array ) { + return arr.concat.apply( [], array ); +}; + + +var push = arr.push; + +var indexOf = arr.indexOf; + +var class2type = {}; + +var toString = class2type.toString; + +var hasOwn = class2type.hasOwnProperty; + +var fnToString = hasOwn.toString; + +var ObjectFunctionString = fnToString.call( Object ); + +var support = {}; + +var isFunction = function isFunction( obj ) { + + // Support: Chrome <=57, Firefox <=52 + // In some browsers, typeof returns "function" for HTML elements + // (i.e., `typeof document.createElement( "object" ) === "function"`). + // We don't want to classify *any* DOM node as a function. + // Support: QtWeb <=3.8.5, WebKit <=534.34, wkhtmltopdf tool <=0.12.5 + // Plus for old WebKit, typeof returns "function" for HTML collections + // (e.g., `typeof document.getElementsByTagName("div") === "function"`). (gh-4756) + return typeof obj === "function" && typeof obj.nodeType !== "number" && + typeof obj.item !== "function"; + }; + + +var isWindow = function isWindow( obj ) { + return obj != null && obj === obj.window; + }; + + +var document = window.document; + + + + var preservedScriptAttributes = { + type: true, + src: true, + nonce: true, + noModule: true + }; + + function DOMEval( code, node, doc ) { + doc = doc || document; + + var i, val, + script = doc.createElement( "script" ); + + script.text = code; + if ( node ) { + for ( i in preservedScriptAttributes ) { + + // Support: Firefox 64+, Edge 18+ + // Some browsers don't support the "nonce" property on scripts. + // On the other hand, just using `getAttribute` is not enough as + // the `nonce` attribute is reset to an empty string whenever it + // becomes browsing-context connected. + // See https://github.com/whatwg/html/issues/2369 + // See https://html.spec.whatwg.org/#nonce-attributes + // The `node.getAttribute` check was added for the sake of + // `jQuery.globalEval` so that it can fake a nonce-containing node + // via an object. + val = node[ i ] || node.getAttribute && node.getAttribute( i ); + if ( val ) { + script.setAttribute( i, val ); + } + } + } + doc.head.appendChild( script ).parentNode.removeChild( script ); + } + + +function toType( obj ) { + if ( obj == null ) { + return obj + ""; + } + + // Support: Android <=2.3 only (functionish RegExp) + return typeof obj === "object" || typeof obj === "function" ? + class2type[ toString.call( obj ) ] || "object" : + typeof obj; +} +/* global Symbol */ +// Defining this global in .eslintrc.json would create a danger of using the global +// unguarded in another place, it seems safer to define global only for this module + + + +var + version = "3.6.0", + + // Define a local copy of jQuery + jQuery = function( selector, context ) { + + // The jQuery object is actually just the init constructor 'enhanced' + // Need init if jQuery is called (just allow error to be thrown if not included) + return new jQuery.fn.init( selector, context ); + }; + +jQuery.fn = jQuery.prototype = { + + // The current version of jQuery being used + jquery: version, + + constructor: jQuery, + + // The default length of a jQuery object is 0 + length: 0, + + toArray: function() { + return slice.call( this ); + }, + + // Get the Nth element in the matched element set OR + // Get the whole matched element set as a clean array + get: function( num ) { + + // Return all the elements in a clean array + if ( num == null ) { + return slice.call( this ); + } + + // Return just the one element from the set + return num < 0 ? this[ num + this.length ] : this[ num ]; + }, + + // Take an array of elements and push it onto the stack + // (returning the new matched element set) + pushStack: function( elems ) { + + // Build a new jQuery matched element set + var ret = jQuery.merge( this.constructor(), elems ); + + // Add the old object onto the stack (as a reference) + ret.prevObject = this; + + // Return the newly-formed element set + return ret; + }, + + // Execute a callback for every element in the matched set. + each: function( callback ) { + return jQuery.each( this, callback ); + }, + + map: function( callback ) { + return this.pushStack( jQuery.map( this, function( elem, i ) { + return callback.call( elem, i, elem ); + } ) ); + }, + + slice: function() { + return this.pushStack( slice.apply( this, arguments ) ); + }, + + first: function() { + return this.eq( 0 ); + }, + + last: function() { + return this.eq( -1 ); + }, + + even: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return ( i + 1 ) % 2; + } ) ); + }, + + odd: function() { + return this.pushStack( jQuery.grep( this, function( _elem, i ) { + return i % 2; + } ) ); + }, + + eq: function( i ) { + var len = this.length, + j = +i + ( i < 0 ? len : 0 ); + return this.pushStack( j >= 0 && j < len ? [ this[ j ] ] : [] ); + }, + + end: function() { + return this.prevObject || this.constructor(); + }, + + // For internal use only. + // Behaves like an Array's method, not like a jQuery method. + push: push, + sort: arr.sort, + splice: arr.splice +}; + +jQuery.extend = jQuery.fn.extend = function() { + var options, name, src, copy, copyIsArray, clone, + target = arguments[ 0 ] || {}, + i = 1, + length = arguments.length, + deep = false; + + // Handle a deep copy situation + if ( typeof target === "boolean" ) { + deep = target; + + // Skip the boolean and the target + target = arguments[ i ] || {}; + i++; + } + + // Handle case when target is a string or something (possible in deep copy) + if ( typeof target !== "object" && !isFunction( target ) ) { + target = {}; + } + + // Extend jQuery itself if only one argument is passed + if ( i === length ) { + target = this; + i--; + } + + for ( ; i < length; i++ ) { + + // Only deal with non-null/undefined values + if ( ( options = arguments[ i ] ) != null ) { + + // Extend the base object + for ( name in options ) { + copy = options[ name ]; + + // Prevent Object.prototype pollution + // Prevent never-ending loop + if ( name === "__proto__" || target === copy ) { + continue; + } + + // Recurse if we're merging plain objects or arrays + if ( deep && copy && ( jQuery.isPlainObject( copy ) || + ( copyIsArray = Array.isArray( copy ) ) ) ) { + src = target[ name ]; + + // Ensure proper type for the source value + if ( copyIsArray && !Array.isArray( src ) ) { + clone = []; + } else if ( !copyIsArray && !jQuery.isPlainObject( src ) ) { + clone = {}; + } else { + clone = src; + } + copyIsArray = false; + + // Never move original objects, clone them + target[ name ] = jQuery.extend( deep, clone, copy ); + + // Don't bring in undefined values + } else if ( copy !== undefined ) { + target[ name ] = copy; + } + } + } + } + + // Return the modified object + return target; +}; + +jQuery.extend( { + + // Unique for each copy of jQuery on the page + expando: "jQuery" + ( version + Math.random() ).replace( /\D/g, "" ), + + // Assume jQuery is ready without the ready module + isReady: true, + + error: function( msg ) { + throw new Error( msg ); + }, + + noop: function() {}, + + isPlainObject: function( obj ) { + var proto, Ctor; + + // Detect obvious negatives + // Use toString instead of jQuery.type to catch host objects + if ( !obj || toString.call( obj ) !== "[object Object]" ) { + return false; + } + + proto = getProto( obj ); + + // Objects with no prototype (e.g., `Object.create( null )`) are plain + if ( !proto ) { + return true; + } + + // Objects with prototype are plain iff they were constructed by a global Object function + Ctor = hasOwn.call( proto, "constructor" ) && proto.constructor; + return typeof Ctor === "function" && fnToString.call( Ctor ) === ObjectFunctionString; + }, + + isEmptyObject: function( obj ) { + var name; + + for ( name in obj ) { + return false; + } + return true; + }, + + // Evaluates a script in a provided context; falls back to the global one + // if not specified. + globalEval: function( code, options, doc ) { + DOMEval( code, { nonce: options && options.nonce }, doc ); + }, + + each: function( obj, callback ) { + var length, i = 0; + + if ( isArrayLike( obj ) ) { + length = obj.length; + for ( ; i < length; i++ ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } else { + for ( i in obj ) { + if ( callback.call( obj[ i ], i, obj[ i ] ) === false ) { + break; + } + } + } + + return obj; + }, + + // results is for internal usage only + makeArray: function( arr, results ) { + var ret = results || []; + + if ( arr != null ) { + if ( isArrayLike( Object( arr ) ) ) { + jQuery.merge( ret, + typeof arr === "string" ? + [ arr ] : arr + ); + } else { + push.call( ret, arr ); + } + } + + return ret; + }, + + inArray: function( elem, arr, i ) { + return arr == null ? -1 : indexOf.call( arr, elem, i ); + }, + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + merge: function( first, second ) { + var len = +second.length, + j = 0, + i = first.length; + + for ( ; j < len; j++ ) { + first[ i++ ] = second[ j ]; + } + + first.length = i; + + return first; + }, + + grep: function( elems, callback, invert ) { + var callbackInverse, + matches = [], + i = 0, + length = elems.length, + callbackExpect = !invert; + + // Go through the array, only saving the items + // that pass the validator function + for ( ; i < length; i++ ) { + callbackInverse = !callback( elems[ i ], i ); + if ( callbackInverse !== callbackExpect ) { + matches.push( elems[ i ] ); + } + } + + return matches; + }, + + // arg is for internal usage only + map: function( elems, callback, arg ) { + var length, value, + i = 0, + ret = []; + + // Go through the array, translating each of the items to their new values + if ( isArrayLike( elems ) ) { + length = elems.length; + for ( ; i < length; i++ ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + + // Go through every key on the object, + } else { + for ( i in elems ) { + value = callback( elems[ i ], i, arg ); + + if ( value != null ) { + ret.push( value ); + } + } + } + + // Flatten any nested arrays + return flat( ret ); + }, + + // A global GUID counter for objects + guid: 1, + + // jQuery.support is not used in Core but other projects attach their + // properties to it so it needs to exist. + support: support +} ); + +if ( typeof Symbol === "function" ) { + jQuery.fn[ Symbol.iterator ] = arr[ Symbol.iterator ]; +} + +// Populate the class2type map +jQuery.each( "Boolean Number String Function Array Date RegExp Object Error Symbol".split( " " ), + function( _i, name ) { + class2type[ "[object " + name + "]" ] = name.toLowerCase(); + } ); + +function isArrayLike( obj ) { + + // Support: real iOS 8.2 only (not reproducible in simulator) + // `in` check used to prevent JIT error (gh-2145) + // hasOwn isn't used here due to false negatives + // regarding Nodelist length in IE + var length = !!obj && "length" in obj && obj.length, + type = toType( obj ); + + if ( isFunction( obj ) || isWindow( obj ) ) { + return false; + } + + return type === "array" || length === 0 || + typeof length === "number" && length > 0 && ( length - 1 ) in obj; +} +var Sizzle = +/*! + * Sizzle CSS Selector Engine v2.3.6 + * https://sizzlejs.com/ + * + * Copyright JS Foundation and other contributors + * Released under the MIT license + * https://js.foundation/ + * + * Date: 2021-02-16 + */ +( function( window ) { +var i, + support, + Expr, + getText, + isXML, + tokenize, + compile, + select, + outermostContext, + sortInput, + hasDuplicate, + + // Local document vars + setDocument, + document, + docElem, + documentIsHTML, + rbuggyQSA, + rbuggyMatches, + matches, + contains, + + // Instance-specific data + expando = "sizzle" + 1 * new Date(), + preferredDoc = window.document, + dirruns = 0, + done = 0, + classCache = createCache(), + tokenCache = createCache(), + compilerCache = createCache(), + nonnativeSelectorCache = createCache(), + sortOrder = function( a, b ) { + if ( a === b ) { + hasDuplicate = true; + } + return 0; + }, + + // Instance methods + hasOwn = ( {} ).hasOwnProperty, + arr = [], + pop = arr.pop, + pushNative = arr.push, + push = arr.push, + slice = arr.slice, + + // Use a stripped-down indexOf as it's faster than native + // https://jsperf.com/thor-indexof-vs-for/5 + indexOf = function( list, elem ) { + var i = 0, + len = list.length; + for ( ; i < len; i++ ) { + if ( list[ i ] === elem ) { + return i; + } + } + return -1; + }, + + booleans = "checked|selected|async|autofocus|autoplay|controls|defer|disabled|hidden|" + + "ismap|loop|multiple|open|readonly|required|scoped", + + // Regular expressions + + // http://www.w3.org/TR/css3-selectors/#whitespace + whitespace = "[\\x20\\t\\r\\n\\f]", + + // https://www.w3.org/TR/css-syntax-3/#ident-token-diagram + identifier = "(?:\\\\[\\da-fA-F]{1,6}" + whitespace + + "?|\\\\[^\\r\\n\\f]|[\\w-]|[^\0-\\x7f])+", + + // Attribute selectors: http://www.w3.org/TR/selectors/#attribute-selectors + attributes = "\\[" + whitespace + "*(" + identifier + ")(?:" + whitespace + + + // Operator (capture 2) + "*([*^$|!~]?=)" + whitespace + + + // "Attribute values must be CSS identifiers [capture 5] + // or strings [capture 3 or capture 4]" + "*(?:'((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\"|(" + identifier + "))|)" + + whitespace + "*\\]", + + pseudos = ":(" + identifier + ")(?:\\((" + + + // To reduce the number of selectors needing tokenize in the preFilter, prefer arguments: + // 1. quoted (capture 3; capture 4 or capture 5) + "('((?:\\\\.|[^\\\\'])*)'|\"((?:\\\\.|[^\\\\\"])*)\")|" + + + // 2. simple (capture 6) + "((?:\\\\.|[^\\\\()[\\]]|" + attributes + ")*)|" + + + // 3. anything else (capture 2) + ".*" + + ")\\)|)", + + // Leading and non-escaped trailing whitespace, capturing some non-whitespace characters preceding the latter + rwhitespace = new RegExp( whitespace + "+", "g" ), + rtrim = new RegExp( "^" + whitespace + "+|((?:^|[^\\\\])(?:\\\\.)*)" + + whitespace + "+$", "g" ), + + rcomma = new RegExp( "^" + whitespace + "*," + whitespace + "*" ), + rcombinators = new RegExp( "^" + whitespace + "*([>+~]|" + whitespace + ")" + whitespace + + "*" ), + rdescend = new RegExp( whitespace + "|>" ), + + rpseudo = new RegExp( pseudos ), + ridentifier = new RegExp( "^" + identifier + "$" ), + + matchExpr = { + "ID": new RegExp( "^#(" + identifier + ")" ), + "CLASS": new RegExp( "^\\.(" + identifier + ")" ), + "TAG": new RegExp( "^(" + identifier + "|[*])" ), + "ATTR": new RegExp( "^" + attributes ), + "PSEUDO": new RegExp( "^" + pseudos ), + "CHILD": new RegExp( "^:(only|first|last|nth|nth-last)-(child|of-type)(?:\\(" + + whitespace + "*(even|odd|(([+-]|)(\\d*)n|)" + whitespace + "*(?:([+-]|)" + + whitespace + "*(\\d+)|))" + whitespace + "*\\)|)", "i" ), + "bool": new RegExp( "^(?:" + booleans + ")$", "i" ), + + // For use in libraries implementing .is() + // We use this for POS matching in `select` + "needsContext": new RegExp( "^" + whitespace + + "*[>+~]|:(even|odd|eq|gt|lt|nth|first|last)(?:\\(" + whitespace + + "*((?:-\\d)?\\d*)" + whitespace + "*\\)|)(?=[^-]|$)", "i" ) + }, + + rhtml = /HTML$/i, + rinputs = /^(?:input|select|textarea|button)$/i, + rheader = /^h\d$/i, + + rnative = /^[^{]+\{\s*\[native \w/, + + // Easily-parseable/retrievable ID or TAG or CLASS selectors + rquickExpr = /^(?:#([\w-]+)|(\w+)|\.([\w-]+))$/, + + rsibling = /[+~]/, + + // CSS escapes + // http://www.w3.org/TR/CSS21/syndata.html#escaped-characters + runescape = new RegExp( "\\\\[\\da-fA-F]{1,6}" + whitespace + "?|\\\\([^\\r\\n\\f])", "g" ), + funescape = function( escape, nonHex ) { + var high = "0x" + escape.slice( 1 ) - 0x10000; + + return nonHex ? + + // Strip the backslash prefix from a non-hex escape sequence + nonHex : + + // Replace a hexadecimal escape sequence with the encoded Unicode code point + // Support: IE <=11+ + // For values outside the Basic Multilingual Plane (BMP), manually construct a + // surrogate pair + high < 0 ? + String.fromCharCode( high + 0x10000 ) : + String.fromCharCode( high >> 10 | 0xD800, high & 0x3FF | 0xDC00 ); + }, + + // CSS string/identifier serialization + // https://drafts.csswg.org/cssom/#common-serializing-idioms + rcssescape = /([\0-\x1f\x7f]|^-?\d)|^-$|[^\0-\x1f\x7f-\uFFFF\w-]/g, + fcssescape = function( ch, asCodePoint ) { + if ( asCodePoint ) { + + // U+0000 NULL becomes U+FFFD REPLACEMENT CHARACTER + if ( ch === "\0" ) { + return "\uFFFD"; + } + + // Control characters and (dependent upon position) numbers get escaped as code points + return ch.slice( 0, -1 ) + "\\" + + ch.charCodeAt( ch.length - 1 ).toString( 16 ) + " "; + } + + // Other potentially-special ASCII characters get backslash-escaped + return "\\" + ch; + }, + + // Used for iframes + // See setDocument() + // Removing the function wrapper causes a "Permission Denied" + // error in IE + unloadHandler = function() { + setDocument(); + }, + + inDisabledFieldset = addCombinator( + function( elem ) { + return elem.disabled === true && elem.nodeName.toLowerCase() === "fieldset"; + }, + { dir: "parentNode", next: "legend" } + ); + +// Optimize for push.apply( _, NodeList ) +try { + push.apply( + ( arr = slice.call( preferredDoc.childNodes ) ), + preferredDoc.childNodes + ); + + // Support: Android<4.0 + // Detect silently failing push.apply + // eslint-disable-next-line no-unused-expressions + arr[ preferredDoc.childNodes.length ].nodeType; +} catch ( e ) { + push = { apply: arr.length ? + + // Leverage slice if possible + function( target, els ) { + pushNative.apply( target, slice.call( els ) ); + } : + + // Support: IE<9 + // Otherwise append directly + function( target, els ) { + var j = target.length, + i = 0; + + // Can't trust NodeList.length + while ( ( target[ j++ ] = els[ i++ ] ) ) {} + target.length = j - 1; + } + }; +} + +function Sizzle( selector, context, results, seed ) { + var m, i, elem, nid, match, groups, newSelector, + newContext = context && context.ownerDocument, + + // nodeType defaults to 9, since context defaults to document + nodeType = context ? context.nodeType : 9; + + results = results || []; + + // Return early from calls with invalid selector or context + if ( typeof selector !== "string" || !selector || + nodeType !== 1 && nodeType !== 9 && nodeType !== 11 ) { + + return results; + } + + // Try to shortcut find operations (as opposed to filters) in HTML documents + if ( !seed ) { + setDocument( context ); + context = context || document; + + if ( documentIsHTML ) { + + // If the selector is sufficiently simple, try using a "get*By*" DOM method + // (excepting DocumentFragment context, where the methods don't exist) + if ( nodeType !== 11 && ( match = rquickExpr.exec( selector ) ) ) { + + // ID selector + if ( ( m = match[ 1 ] ) ) { + + // Document context + if ( nodeType === 9 ) { + if ( ( elem = context.getElementById( m ) ) ) { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( elem.id === m ) { + results.push( elem ); + return results; + } + } else { + return results; + } + + // Element context + } else { + + // Support: IE, Opera, Webkit + // TODO: identify versions + // getElementById can match elements by name instead of ID + if ( newContext && ( elem = newContext.getElementById( m ) ) && + contains( context, elem ) && + elem.id === m ) { + + results.push( elem ); + return results; + } + } + + // Type selector + } else if ( match[ 2 ] ) { + push.apply( results, context.getElementsByTagName( selector ) ); + return results; + + // Class selector + } else if ( ( m = match[ 3 ] ) && support.getElementsByClassName && + context.getElementsByClassName ) { + + push.apply( results, context.getElementsByClassName( m ) ); + return results; + } + } + + // Take advantage of querySelectorAll + if ( support.qsa && + !nonnativeSelectorCache[ selector + " " ] && + ( !rbuggyQSA || !rbuggyQSA.test( selector ) ) && + + // Support: IE 8 only + // Exclude object elements + ( nodeType !== 1 || context.nodeName.toLowerCase() !== "object" ) ) { + + newSelector = selector; + newContext = context; + + // qSA considers elements outside a scoping root when evaluating child or + // descendant combinators, which is not what we want. + // In such cases, we work around the behavior by prefixing every selector in the + // list with an ID selector referencing the scope context. + // The technique has to be used as well when a leading combinator is used + // as such selectors are not recognized by querySelectorAll. + // Thanks to Andrew Dupont for this technique. + if ( nodeType === 1 && + ( rdescend.test( selector ) || rcombinators.test( selector ) ) ) { + + // Expand context for sibling selectors + newContext = rsibling.test( selector ) && testContext( context.parentNode ) || + context; + + // We can use :scope instead of the ID hack if the browser + // supports it & if we're not changing the context. + if ( newContext !== context || !support.scope ) { + + // Capture the context ID, setting it first if necessary + if ( ( nid = context.getAttribute( "id" ) ) ) { + nid = nid.replace( rcssescape, fcssescape ); + } else { + context.setAttribute( "id", ( nid = expando ) ); + } + } + + // Prefix every selector in the list + groups = tokenize( selector ); + i = groups.length; + while ( i-- ) { + groups[ i ] = ( nid ? "#" + nid : ":scope" ) + " " + + toSelector( groups[ i ] ); + } + newSelector = groups.join( "," ); + } + + try { + push.apply( results, + newContext.querySelectorAll( newSelector ) + ); + return results; + } catch ( qsaError ) { + nonnativeSelectorCache( selector, true ); + } finally { + if ( nid === expando ) { + context.removeAttribute( "id" ); + } + } + } + } + } + + // All others + return select( selector.replace( rtrim, "$1" ), context, results, seed ); +} + +/** + * Create key-value caches of limited size + * @returns {function(string, object)} Returns the Object data after storing it on itself with + * property name the (space-suffixed) string and (if the cache is larger than Expr.cacheLength) + * deleting the oldest entry + */ +function createCache() { + var keys = []; + + function cache( key, value ) { + + // Use (key + " ") to avoid collision with native prototype properties (see Issue #157) + if ( keys.push( key + " " ) > Expr.cacheLength ) { + + // Only keep the most recent entries + delete cache[ keys.shift() ]; + } + return ( cache[ key + " " ] = value ); + } + return cache; +} + +/** + * Mark a function for special use by Sizzle + * @param {Function} fn The function to mark + */ +function markFunction( fn ) { + fn[ expando ] = true; + return fn; +} + +/** + * Support testing using an element + * @param {Function} fn Passed the created element and returns a boolean result + */ +function assert( fn ) { + var el = document.createElement( "fieldset" ); + + try { + return !!fn( el ); + } catch ( e ) { + return false; + } finally { + + // Remove from its parent by default + if ( el.parentNode ) { + el.parentNode.removeChild( el ); + } + + // release memory in IE + el = null; + } +} + +/** + * Adds the same handler for all of the specified attrs + * @param {String} attrs Pipe-separated list of attributes + * @param {Function} handler The method that will be applied + */ +function addHandle( attrs, handler ) { + var arr = attrs.split( "|" ), + i = arr.length; + + while ( i-- ) { + Expr.attrHandle[ arr[ i ] ] = handler; + } +} + +/** + * Checks document order of two siblings + * @param {Element} a + * @param {Element} b + * @returns {Number} Returns less than 0 if a precedes b, greater than 0 if a follows b + */ +function siblingCheck( a, b ) { + var cur = b && a, + diff = cur && a.nodeType === 1 && b.nodeType === 1 && + a.sourceIndex - b.sourceIndex; + + // Use IE sourceIndex if available on both nodes + if ( diff ) { + return diff; + } + + // Check if b follows a + if ( cur ) { + while ( ( cur = cur.nextSibling ) ) { + if ( cur === b ) { + return -1; + } + } + } + + return a ? 1 : -1; +} + +/** + * Returns a function to use in pseudos for input types + * @param {String} type + */ +function createInputPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for buttons + * @param {String} type + */ +function createButtonPseudo( type ) { + return function( elem ) { + var name = elem.nodeName.toLowerCase(); + return ( name === "input" || name === "button" ) && elem.type === type; + }; +} + +/** + * Returns a function to use in pseudos for :enabled/:disabled + * @param {Boolean} disabled true for :disabled; false for :enabled + */ +function createDisabledPseudo( disabled ) { + + // Known :disabled false positives: fieldset[disabled] > legend:nth-of-type(n+2) :can-disable + return function( elem ) { + + // Only certain elements can match :enabled or :disabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-enabled + // https://html.spec.whatwg.org/multipage/scripting.html#selector-disabled + if ( "form" in elem ) { + + // Check for inherited disabledness on relevant non-disabled elements: + // * listed form-associated elements in a disabled fieldset + // https://html.spec.whatwg.org/multipage/forms.html#category-listed + // https://html.spec.whatwg.org/multipage/forms.html#concept-fe-disabled + // * option elements in a disabled optgroup + // https://html.spec.whatwg.org/multipage/forms.html#concept-option-disabled + // All such elements have a "form" property. + if ( elem.parentNode && elem.disabled === false ) { + + // Option elements defer to a parent optgroup if present + if ( "label" in elem ) { + if ( "label" in elem.parentNode ) { + return elem.parentNode.disabled === disabled; + } else { + return elem.disabled === disabled; + } + } + + // Support: IE 6 - 11 + // Use the isDisabled shortcut property to check for disabled fieldset ancestors + return elem.isDisabled === disabled || + + // Where there is no isDisabled, check manually + /* jshint -W018 */ + elem.isDisabled !== !disabled && + inDisabledFieldset( elem ) === disabled; + } + + return elem.disabled === disabled; + + // Try to winnow out elements that can't be disabled before trusting the disabled property. + // Some victims get caught in our net (label, legend, menu, track), but it shouldn't + // even exist on them, let alone have a boolean value. + } else if ( "label" in elem ) { + return elem.disabled === disabled; + } + + // Remaining elements are neither :enabled nor :disabled + return false; + }; +} + +/** + * Returns a function to use in pseudos for positionals + * @param {Function} fn + */ +function createPositionalPseudo( fn ) { + return markFunction( function( argument ) { + argument = +argument; + return markFunction( function( seed, matches ) { + var j, + matchIndexes = fn( [], seed.length, argument ), + i = matchIndexes.length; + + // Match elements found at the specified indexes + while ( i-- ) { + if ( seed[ ( j = matchIndexes[ i ] ) ] ) { + seed[ j ] = !( matches[ j ] = seed[ j ] ); + } + } + } ); + } ); +} + +/** + * Checks a node for validity as a Sizzle context + * @param {Element|Object=} context + * @returns {Element|Object|Boolean} The input node if acceptable, otherwise a falsy value + */ +function testContext( context ) { + return context && typeof context.getElementsByTagName !== "undefined" && context; +} + +// Expose support vars for convenience +support = Sizzle.support = {}; + +/** + * Detects XML nodes + * @param {Element|Object} elem An element or a document + * @returns {Boolean} True iff elem is a non-HTML XML node + */ +isXML = Sizzle.isXML = function( elem ) { + var namespace = elem && elem.namespaceURI, + docElem = elem && ( elem.ownerDocument || elem ).documentElement; + + // Support: IE <=8 + // Assume HTML when documentElement doesn't yet exist, such as inside loading iframes + // https://bugs.jquery.com/ticket/4833 + return !rhtml.test( namespace || docElem && docElem.nodeName || "HTML" ); +}; + +/** + * Sets document-related variables once based on the current document + * @param {Element|Object} [doc] An element or document object to use to set the document + * @returns {Object} Returns the current document + */ +setDocument = Sizzle.setDocument = function( node ) { + var hasCompare, subWindow, + doc = node ? node.ownerDocument || node : preferredDoc; + + // Return early if doc is invalid or already selected + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( doc == document || doc.nodeType !== 9 || !doc.documentElement ) { + return document; + } + + // Update global variables + document = doc; + docElem = document.documentElement; + documentIsHTML = !isXML( document ); + + // Support: IE 9 - 11+, Edge 12 - 18+ + // Accessing iframe documents after unload throws "permission denied" errors (jQuery #13936) + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( preferredDoc != document && + ( subWindow = document.defaultView ) && subWindow.top !== subWindow ) { + + // Support: IE 11, Edge + if ( subWindow.addEventListener ) { + subWindow.addEventListener( "unload", unloadHandler, false ); + + // Support: IE 9 - 10 only + } else if ( subWindow.attachEvent ) { + subWindow.attachEvent( "onunload", unloadHandler ); + } + } + + // Support: IE 8 - 11+, Edge 12 - 18+, Chrome <=16 - 25 only, Firefox <=3.6 - 31 only, + // Safari 4 - 5 only, Opera <=11.6 - 12.x only + // IE/Edge & older browsers don't support the :scope pseudo-class. + // Support: Safari 6.0 only + // Safari 6.0 supports :scope but it's an alias of :root there. + support.scope = assert( function( el ) { + docElem.appendChild( el ).appendChild( document.createElement( "div" ) ); + return typeof el.querySelectorAll !== "undefined" && + !el.querySelectorAll( ":scope fieldset div" ).length; + } ); + + /* Attributes + ---------------------------------------------------------------------- */ + + // Support: IE<8 + // Verify that getAttribute really returns attributes and not properties + // (excepting IE8 booleans) + support.attributes = assert( function( el ) { + el.className = "i"; + return !el.getAttribute( "className" ); + } ); + + /* getElement(s)By* + ---------------------------------------------------------------------- */ + + // Check if getElementsByTagName("*") returns only elements + support.getElementsByTagName = assert( function( el ) { + el.appendChild( document.createComment( "" ) ); + return !el.getElementsByTagName( "*" ).length; + } ); + + // Support: IE<9 + support.getElementsByClassName = rnative.test( document.getElementsByClassName ); + + // Support: IE<10 + // Check if getElementById returns elements by name + // The broken getElementById methods don't pick up programmatically-set names, + // so use a roundabout getElementsByName test + support.getById = assert( function( el ) { + docElem.appendChild( el ).id = expando; + return !document.getElementsByName || !document.getElementsByName( expando ).length; + } ); + + // ID filter and find + if ( support.getById ) { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + return elem.getAttribute( "id" ) === attrId; + }; + }; + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var elem = context.getElementById( id ); + return elem ? [ elem ] : []; + } + }; + } else { + Expr.filter[ "ID" ] = function( id ) { + var attrId = id.replace( runescape, funescape ); + return function( elem ) { + var node = typeof elem.getAttributeNode !== "undefined" && + elem.getAttributeNode( "id" ); + return node && node.value === attrId; + }; + }; + + // Support: IE 6 - 7 only + // getElementById is not reliable as a find shortcut + Expr.find[ "ID" ] = function( id, context ) { + if ( typeof context.getElementById !== "undefined" && documentIsHTML ) { + var node, i, elems, + elem = context.getElementById( id ); + + if ( elem ) { + + // Verify the id attribute + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + + // Fall back on getElementsByName + elems = context.getElementsByName( id ); + i = 0; + while ( ( elem = elems[ i++ ] ) ) { + node = elem.getAttributeNode( "id" ); + if ( node && node.value === id ) { + return [ elem ]; + } + } + } + + return []; + } + }; + } + + // Tag + Expr.find[ "TAG" ] = support.getElementsByTagName ? + function( tag, context ) { + if ( typeof context.getElementsByTagName !== "undefined" ) { + return context.getElementsByTagName( tag ); + + // DocumentFragment nodes don't have gEBTN + } else if ( support.qsa ) { + return context.querySelectorAll( tag ); + } + } : + + function( tag, context ) { + var elem, + tmp = [], + i = 0, + + // By happy coincidence, a (broken) gEBTN appears on DocumentFragment nodes too + results = context.getElementsByTagName( tag ); + + // Filter out possible comments + if ( tag === "*" ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem.nodeType === 1 ) { + tmp.push( elem ); + } + } + + return tmp; + } + return results; + }; + + // Class + Expr.find[ "CLASS" ] = support.getElementsByClassName && function( className, context ) { + if ( typeof context.getElementsByClassName !== "undefined" && documentIsHTML ) { + return context.getElementsByClassName( className ); + } + }; + + /* QSA/matchesSelector + ---------------------------------------------------------------------- */ + + // QSA and matchesSelector support + + // matchesSelector(:active) reports false when true (IE9/Opera 11.5) + rbuggyMatches = []; + + // qSa(:focus) reports false when true (Chrome 21) + // We allow this because of a bug in IE8/9 that throws an error + // whenever `document.activeElement` is accessed on an iframe + // So, we allow :focus to pass through QSA all the time to avoid the IE error + // See https://bugs.jquery.com/ticket/13378 + rbuggyQSA = []; + + if ( ( support.qsa = rnative.test( document.querySelectorAll ) ) ) { + + // Build QSA regex + // Regex strategy adopted from Diego Perini + assert( function( el ) { + + var input; + + // Select is set to empty string on purpose + // This is to test IE's treatment of not explicitly + // setting a boolean content attribute, + // since its presence should be enough + // https://bugs.jquery.com/ticket/12359 + docElem.appendChild( el ).innerHTML = "" + + ""; + + // Support: IE8, Opera 11-12.16 + // Nothing should be selected when empty strings follow ^= or $= or *= + // The test attribute must be unknown in Opera but "safe" for WinRT + // https://msdn.microsoft.com/en-us/library/ie/hh465388.aspx#attribute_section + if ( el.querySelectorAll( "[msallowcapture^='']" ).length ) { + rbuggyQSA.push( "[*^$]=" + whitespace + "*(?:''|\"\")" ); + } + + // Support: IE8 + // Boolean attributes and "value" are not treated correctly + if ( !el.querySelectorAll( "[selected]" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*(?:value|" + booleans + ")" ); + } + + // Support: Chrome<29, Android<4.4, Safari<7.0+, iOS<7.0+, PhantomJS<1.9.8+ + if ( !el.querySelectorAll( "[id~=" + expando + "-]" ).length ) { + rbuggyQSA.push( "~=" ); + } + + // Support: IE 11+, Edge 15 - 18+ + // IE 11/Edge don't find elements on a `[name='']` query in some cases. + // Adding a temporary attribute to the document before the selection works + // around the issue. + // Interestingly, IE 10 & older don't seem to have the issue. + input = document.createElement( "input" ); + input.setAttribute( "name", "" ); + el.appendChild( input ); + if ( !el.querySelectorAll( "[name='']" ).length ) { + rbuggyQSA.push( "\\[" + whitespace + "*name" + whitespace + "*=" + + whitespace + "*(?:''|\"\")" ); + } + + // Webkit/Opera - :checked should return selected option elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + // IE8 throws error here and will not see later tests + if ( !el.querySelectorAll( ":checked" ).length ) { + rbuggyQSA.push( ":checked" ); + } + + // Support: Safari 8+, iOS 8+ + // https://bugs.webkit.org/show_bug.cgi?id=136851 + // In-page `selector#id sibling-combinator selector` fails + if ( !el.querySelectorAll( "a#" + expando + "+*" ).length ) { + rbuggyQSA.push( ".#.+[+~]" ); + } + + // Support: Firefox <=3.6 - 5 only + // Old Firefox doesn't throw on a badly-escaped identifier. + el.querySelectorAll( "\\\f" ); + rbuggyQSA.push( "[\\r\\n\\f]" ); + } ); + + assert( function( el ) { + el.innerHTML = "" + + ""; + + // Support: Windows 8 Native Apps + // The type and name attributes are restricted during .innerHTML assignment + var input = document.createElement( "input" ); + input.setAttribute( "type", "hidden" ); + el.appendChild( input ).setAttribute( "name", "D" ); + + // Support: IE8 + // Enforce case-sensitivity of name attribute + if ( el.querySelectorAll( "[name=d]" ).length ) { + rbuggyQSA.push( "name" + whitespace + "*[*^$|!~]?=" ); + } + + // FF 3.5 - :enabled/:disabled and hidden elements (hidden elements are still enabled) + // IE8 throws error here and will not see later tests + if ( el.querySelectorAll( ":enabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: IE9-11+ + // IE's :disabled selector does not pick up the children of disabled fieldsets + docElem.appendChild( el ).disabled = true; + if ( el.querySelectorAll( ":disabled" ).length !== 2 ) { + rbuggyQSA.push( ":enabled", ":disabled" ); + } + + // Support: Opera 10 - 11 only + // Opera 10-11 does not throw on post-comma invalid pseudos + el.querySelectorAll( "*,:x" ); + rbuggyQSA.push( ",.*:" ); + } ); + } + + if ( ( support.matchesSelector = rnative.test( ( matches = docElem.matches || + docElem.webkitMatchesSelector || + docElem.mozMatchesSelector || + docElem.oMatchesSelector || + docElem.msMatchesSelector ) ) ) ) { + + assert( function( el ) { + + // Check to see if it's possible to do matchesSelector + // on a disconnected node (IE 9) + support.disconnectedMatch = matches.call( el, "*" ); + + // This should fail with an exception + // Gecko does not error, returns false instead + matches.call( el, "[s!='']:x" ); + rbuggyMatches.push( "!=", pseudos ); + } ); + } + + rbuggyQSA = rbuggyQSA.length && new RegExp( rbuggyQSA.join( "|" ) ); + rbuggyMatches = rbuggyMatches.length && new RegExp( rbuggyMatches.join( "|" ) ); + + /* Contains + ---------------------------------------------------------------------- */ + hasCompare = rnative.test( docElem.compareDocumentPosition ); + + // Element contains another + // Purposefully self-exclusive + // As in, an element does not contain itself + contains = hasCompare || rnative.test( docElem.contains ) ? + function( a, b ) { + var adown = a.nodeType === 9 ? a.documentElement : a, + bup = b && b.parentNode; + return a === bup || !!( bup && bup.nodeType === 1 && ( + adown.contains ? + adown.contains( bup ) : + a.compareDocumentPosition && a.compareDocumentPosition( bup ) & 16 + ) ); + } : + function( a, b ) { + if ( b ) { + while ( ( b = b.parentNode ) ) { + if ( b === a ) { + return true; + } + } + } + return false; + }; + + /* Sorting + ---------------------------------------------------------------------- */ + + // Document order sorting + sortOrder = hasCompare ? + function( a, b ) { + + // Flag for duplicate removal + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + // Sort on method existence if only one input has compareDocumentPosition + var compare = !a.compareDocumentPosition - !b.compareDocumentPosition; + if ( compare ) { + return compare; + } + + // Calculate position if both inputs belong to the same document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + compare = ( a.ownerDocument || a ) == ( b.ownerDocument || b ) ? + a.compareDocumentPosition( b ) : + + // Otherwise we know they are disconnected + 1; + + // Disconnected nodes + if ( compare & 1 || + ( !support.sortDetached && b.compareDocumentPosition( a ) === compare ) ) { + + // Choose the first element that is related to our preferred document + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( a == document || a.ownerDocument == preferredDoc && + contains( preferredDoc, a ) ) { + return -1; + } + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( b == document || b.ownerDocument == preferredDoc && + contains( preferredDoc, b ) ) { + return 1; + } + + // Maintain original order + return sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + } + + return compare & 4 ? -1 : 1; + } : + function( a, b ) { + + // Exit early if the nodes are identical + if ( a === b ) { + hasDuplicate = true; + return 0; + } + + var cur, + i = 0, + aup = a.parentNode, + bup = b.parentNode, + ap = [ a ], + bp = [ b ]; + + // Parentless nodes are either documents or disconnected + if ( !aup || !bup ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + return a == document ? -1 : + b == document ? 1 : + /* eslint-enable eqeqeq */ + aup ? -1 : + bup ? 1 : + sortInput ? + ( indexOf( sortInput, a ) - indexOf( sortInput, b ) ) : + 0; + + // If the nodes are siblings, we can do a quick check + } else if ( aup === bup ) { + return siblingCheck( a, b ); + } + + // Otherwise we need full lists of their ancestors for comparison + cur = a; + while ( ( cur = cur.parentNode ) ) { + ap.unshift( cur ); + } + cur = b; + while ( ( cur = cur.parentNode ) ) { + bp.unshift( cur ); + } + + // Walk down the tree looking for a discrepancy + while ( ap[ i ] === bp[ i ] ) { + i++; + } + + return i ? + + // Do a sibling check if the nodes have a common ancestor + siblingCheck( ap[ i ], bp[ i ] ) : + + // Otherwise nodes in our document sort first + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + /* eslint-disable eqeqeq */ + ap[ i ] == preferredDoc ? -1 : + bp[ i ] == preferredDoc ? 1 : + /* eslint-enable eqeqeq */ + 0; + }; + + return document; +}; + +Sizzle.matches = function( expr, elements ) { + return Sizzle( expr, null, null, elements ); +}; + +Sizzle.matchesSelector = function( elem, expr ) { + setDocument( elem ); + + if ( support.matchesSelector && documentIsHTML && + !nonnativeSelectorCache[ expr + " " ] && + ( !rbuggyMatches || !rbuggyMatches.test( expr ) ) && + ( !rbuggyQSA || !rbuggyQSA.test( expr ) ) ) { + + try { + var ret = matches.call( elem, expr ); + + // IE 9's matchesSelector returns false on disconnected nodes + if ( ret || support.disconnectedMatch || + + // As well, disconnected nodes are said to be in a document + // fragment in IE 9 + elem.document && elem.document.nodeType !== 11 ) { + return ret; + } + } catch ( e ) { + nonnativeSelectorCache( expr, true ); + } + } + + return Sizzle( expr, document, null, [ elem ] ).length > 0; +}; + +Sizzle.contains = function( context, elem ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( context.ownerDocument || context ) != document ) { + setDocument( context ); + } + return contains( context, elem ); +}; + +Sizzle.attr = function( elem, name ) { + + // Set document vars if needed + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( ( elem.ownerDocument || elem ) != document ) { + setDocument( elem ); + } + + var fn = Expr.attrHandle[ name.toLowerCase() ], + + // Don't get fooled by Object.prototype properties (jQuery #13807) + val = fn && hasOwn.call( Expr.attrHandle, name.toLowerCase() ) ? + fn( elem, name, !documentIsHTML ) : + undefined; + + return val !== undefined ? + val : + support.attributes || !documentIsHTML ? + elem.getAttribute( name ) : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; +}; + +Sizzle.escape = function( sel ) { + return ( sel + "" ).replace( rcssescape, fcssescape ); +}; + +Sizzle.error = function( msg ) { + throw new Error( "Syntax error, unrecognized expression: " + msg ); +}; + +/** + * Document sorting and removing duplicates + * @param {ArrayLike} results + */ +Sizzle.uniqueSort = function( results ) { + var elem, + duplicates = [], + j = 0, + i = 0; + + // Unless we *know* we can detect duplicates, assume their presence + hasDuplicate = !support.detectDuplicates; + sortInput = !support.sortStable && results.slice( 0 ); + results.sort( sortOrder ); + + if ( hasDuplicate ) { + while ( ( elem = results[ i++ ] ) ) { + if ( elem === results[ i ] ) { + j = duplicates.push( i ); + } + } + while ( j-- ) { + results.splice( duplicates[ j ], 1 ); + } + } + + // Clear input after sorting to release objects + // See https://github.com/jquery/sizzle/pull/225 + sortInput = null; + + return results; +}; + +/** + * Utility function for retrieving the text value of an array of DOM nodes + * @param {Array|Element} elem + */ +getText = Sizzle.getText = function( elem ) { + var node, + ret = "", + i = 0, + nodeType = elem.nodeType; + + if ( !nodeType ) { + + // If no nodeType, this is expected to be an array + while ( ( node = elem[ i++ ] ) ) { + + // Do not traverse comment nodes + ret += getText( node ); + } + } else if ( nodeType === 1 || nodeType === 9 || nodeType === 11 ) { + + // Use textContent for elements + // innerText usage removed for consistency of new lines (jQuery #11153) + if ( typeof elem.textContent === "string" ) { + return elem.textContent; + } else { + + // Traverse its children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + ret += getText( elem ); + } + } + } else if ( nodeType === 3 || nodeType === 4 ) { + return elem.nodeValue; + } + + // Do not include comment or processing instruction nodes + + return ret; +}; + +Expr = Sizzle.selectors = { + + // Can be adjusted by the user + cacheLength: 50, + + createPseudo: markFunction, + + match: matchExpr, + + attrHandle: {}, + + find: {}, + + relative: { + ">": { dir: "parentNode", first: true }, + " ": { dir: "parentNode" }, + "+": { dir: "previousSibling", first: true }, + "~": { dir: "previousSibling" } + }, + + preFilter: { + "ATTR": function( match ) { + match[ 1 ] = match[ 1 ].replace( runescape, funescape ); + + // Move the given value to match[3] whether quoted or unquoted + match[ 3 ] = ( match[ 3 ] || match[ 4 ] || + match[ 5 ] || "" ).replace( runescape, funescape ); + + if ( match[ 2 ] === "~=" ) { + match[ 3 ] = " " + match[ 3 ] + " "; + } + + return match.slice( 0, 4 ); + }, + + "CHILD": function( match ) { + + /* matches from matchExpr["CHILD"] + 1 type (only|nth|...) + 2 what (child|of-type) + 3 argument (even|odd|\d*|\d*n([+-]\d+)?|...) + 4 xn-component of xn+y argument ([+-]?\d*n|) + 5 sign of xn-component + 6 x of xn-component + 7 sign of y-component + 8 y of y-component + */ + match[ 1 ] = match[ 1 ].toLowerCase(); + + if ( match[ 1 ].slice( 0, 3 ) === "nth" ) { + + // nth-* requires argument + if ( !match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + // numeric x and y parameters for Expr.filter.CHILD + // remember that false/true cast respectively to 0/1 + match[ 4 ] = +( match[ 4 ] ? + match[ 5 ] + ( match[ 6 ] || 1 ) : + 2 * ( match[ 3 ] === "even" || match[ 3 ] === "odd" ) ); + match[ 5 ] = +( ( match[ 7 ] + match[ 8 ] ) || match[ 3 ] === "odd" ); + + // other types prohibit arguments + } else if ( match[ 3 ] ) { + Sizzle.error( match[ 0 ] ); + } + + return match; + }, + + "PSEUDO": function( match ) { + var excess, + unquoted = !match[ 6 ] && match[ 2 ]; + + if ( matchExpr[ "CHILD" ].test( match[ 0 ] ) ) { + return null; + } + + // Accept quoted arguments as-is + if ( match[ 3 ] ) { + match[ 2 ] = match[ 4 ] || match[ 5 ] || ""; + + // Strip excess characters from unquoted arguments + } else if ( unquoted && rpseudo.test( unquoted ) && + + // Get excess from tokenize (recursively) + ( excess = tokenize( unquoted, true ) ) && + + // advance to the next closing parenthesis + ( excess = unquoted.indexOf( ")", unquoted.length - excess ) - unquoted.length ) ) { + + // excess is a negative index + match[ 0 ] = match[ 0 ].slice( 0, excess ); + match[ 2 ] = unquoted.slice( 0, excess ); + } + + // Return only captures needed by the pseudo filter method (type and argument) + return match.slice( 0, 3 ); + } + }, + + filter: { + + "TAG": function( nodeNameSelector ) { + var nodeName = nodeNameSelector.replace( runescape, funescape ).toLowerCase(); + return nodeNameSelector === "*" ? + function() { + return true; + } : + function( elem ) { + return elem.nodeName && elem.nodeName.toLowerCase() === nodeName; + }; + }, + + "CLASS": function( className ) { + var pattern = classCache[ className + " " ]; + + return pattern || + ( pattern = new RegExp( "(^|" + whitespace + + ")" + className + "(" + whitespace + "|$)" ) ) && classCache( + className, function( elem ) { + return pattern.test( + typeof elem.className === "string" && elem.className || + typeof elem.getAttribute !== "undefined" && + elem.getAttribute( "class" ) || + "" + ); + } ); + }, + + "ATTR": function( name, operator, check ) { + return function( elem ) { + var result = Sizzle.attr( elem, name ); + + if ( result == null ) { + return operator === "!="; + } + if ( !operator ) { + return true; + } + + result += ""; + + /* eslint-disable max-len */ + + return operator === "=" ? result === check : + operator === "!=" ? result !== check : + operator === "^=" ? check && result.indexOf( check ) === 0 : + operator === "*=" ? check && result.indexOf( check ) > -1 : + operator === "$=" ? check && result.slice( -check.length ) === check : + operator === "~=" ? ( " " + result.replace( rwhitespace, " " ) + " " ).indexOf( check ) > -1 : + operator === "|=" ? result === check || result.slice( 0, check.length + 1 ) === check + "-" : + false; + /* eslint-enable max-len */ + + }; + }, + + "CHILD": function( type, what, _argument, first, last ) { + var simple = type.slice( 0, 3 ) !== "nth", + forward = type.slice( -4 ) !== "last", + ofType = what === "of-type"; + + return first === 1 && last === 0 ? + + // Shortcut for :nth-*(n) + function( elem ) { + return !!elem.parentNode; + } : + + function( elem, _context, xml ) { + var cache, uniqueCache, outerCache, node, nodeIndex, start, + dir = simple !== forward ? "nextSibling" : "previousSibling", + parent = elem.parentNode, + name = ofType && elem.nodeName.toLowerCase(), + useCache = !xml && !ofType, + diff = false; + + if ( parent ) { + + // :(first|last|only)-(child|of-type) + if ( simple ) { + while ( dir ) { + node = elem; + while ( ( node = node[ dir ] ) ) { + if ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) { + + return false; + } + } + + // Reverse direction for :only-* (if we haven't yet done so) + start = dir = type === "only" && !start && "nextSibling"; + } + return true; + } + + start = [ forward ? parent.firstChild : parent.lastChild ]; + + // non-xml :nth-child(...) stores cache data on `parent` + if ( forward && useCache ) { + + // Seek `elem` from a previously-cached index + + // ...in a gzip-friendly way + node = parent; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex && cache[ 2 ]; + node = nodeIndex && parent.childNodes[ nodeIndex ]; + + while ( ( node = ++nodeIndex && node && node[ dir ] || + + // Fallback to seeking `elem` from the start + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + // When found, cache indexes on `parent` and break + if ( node.nodeType === 1 && ++diff && node === elem ) { + uniqueCache[ type ] = [ dirruns, nodeIndex, diff ]; + break; + } + } + + } else { + + // Use previously-cached element index if available + if ( useCache ) { + + // ...in a gzip-friendly way + node = elem; + outerCache = node[ expando ] || ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + cache = uniqueCache[ type ] || []; + nodeIndex = cache[ 0 ] === dirruns && cache[ 1 ]; + diff = nodeIndex; + } + + // xml :nth-child(...) + // or :nth-last-child(...) or :nth(-last)?-of-type(...) + if ( diff === false ) { + + // Use the same loop as above to seek `elem` from the start + while ( ( node = ++nodeIndex && node && node[ dir ] || + ( diff = nodeIndex = 0 ) || start.pop() ) ) { + + if ( ( ofType ? + node.nodeName.toLowerCase() === name : + node.nodeType === 1 ) && + ++diff ) { + + // Cache the index of each encountered element + if ( useCache ) { + outerCache = node[ expando ] || + ( node[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ node.uniqueID ] || + ( outerCache[ node.uniqueID ] = {} ); + + uniqueCache[ type ] = [ dirruns, diff ]; + } + + if ( node === elem ) { + break; + } + } + } + } + } + + // Incorporate the offset, then check against cycle size + diff -= last; + return diff === first || ( diff % first === 0 && diff / first >= 0 ); + } + }; + }, + + "PSEUDO": function( pseudo, argument ) { + + // pseudo-class names are case-insensitive + // http://www.w3.org/TR/selectors/#pseudo-classes + // Prioritize by case sensitivity in case custom pseudos are added with uppercase letters + // Remember that setFilters inherits from pseudos + var args, + fn = Expr.pseudos[ pseudo ] || Expr.setFilters[ pseudo.toLowerCase() ] || + Sizzle.error( "unsupported pseudo: " + pseudo ); + + // The user may use createPseudo to indicate that + // arguments are needed to create the filter function + // just as Sizzle does + if ( fn[ expando ] ) { + return fn( argument ); + } + + // But maintain support for old signatures + if ( fn.length > 1 ) { + args = [ pseudo, pseudo, "", argument ]; + return Expr.setFilters.hasOwnProperty( pseudo.toLowerCase() ) ? + markFunction( function( seed, matches ) { + var idx, + matched = fn( seed, argument ), + i = matched.length; + while ( i-- ) { + idx = indexOf( seed, matched[ i ] ); + seed[ idx ] = !( matches[ idx ] = matched[ i ] ); + } + } ) : + function( elem ) { + return fn( elem, 0, args ); + }; + } + + return fn; + } + }, + + pseudos: { + + // Potentially complex pseudos + "not": markFunction( function( selector ) { + + // Trim the selector passed to compile + // to avoid treating leading and trailing + // spaces as combinators + var input = [], + results = [], + matcher = compile( selector.replace( rtrim, "$1" ) ); + + return matcher[ expando ] ? + markFunction( function( seed, matches, _context, xml ) { + var elem, + unmatched = matcher( seed, null, xml, [] ), + i = seed.length; + + // Match elements unmatched by `matcher` + while ( i-- ) { + if ( ( elem = unmatched[ i ] ) ) { + seed[ i ] = !( matches[ i ] = elem ); + } + } + } ) : + function( elem, _context, xml ) { + input[ 0 ] = elem; + matcher( input, null, xml, results ); + + // Don't keep the element (issue #299) + input[ 0 ] = null; + return !results.pop(); + }; + } ), + + "has": markFunction( function( selector ) { + return function( elem ) { + return Sizzle( selector, elem ).length > 0; + }; + } ), + + "contains": markFunction( function( text ) { + text = text.replace( runescape, funescape ); + return function( elem ) { + return ( elem.textContent || getText( elem ) ).indexOf( text ) > -1; + }; + } ), + + // "Whether an element is represented by a :lang() selector + // is based solely on the element's language value + // being equal to the identifier C, + // or beginning with the identifier C immediately followed by "-". + // The matching of C against the element's language value is performed case-insensitively. + // The identifier C does not have to be a valid language name." + // http://www.w3.org/TR/selectors/#lang-pseudo + "lang": markFunction( function( lang ) { + + // lang value must be a valid identifier + if ( !ridentifier.test( lang || "" ) ) { + Sizzle.error( "unsupported lang: " + lang ); + } + lang = lang.replace( runescape, funescape ).toLowerCase(); + return function( elem ) { + var elemLang; + do { + if ( ( elemLang = documentIsHTML ? + elem.lang : + elem.getAttribute( "xml:lang" ) || elem.getAttribute( "lang" ) ) ) { + + elemLang = elemLang.toLowerCase(); + return elemLang === lang || elemLang.indexOf( lang + "-" ) === 0; + } + } while ( ( elem = elem.parentNode ) && elem.nodeType === 1 ); + return false; + }; + } ), + + // Miscellaneous + "target": function( elem ) { + var hash = window.location && window.location.hash; + return hash && hash.slice( 1 ) === elem.id; + }, + + "root": function( elem ) { + return elem === docElem; + }, + + "focus": function( elem ) { + return elem === document.activeElement && + ( !document.hasFocus || document.hasFocus() ) && + !!( elem.type || elem.href || ~elem.tabIndex ); + }, + + // Boolean properties + "enabled": createDisabledPseudo( false ), + "disabled": createDisabledPseudo( true ), + + "checked": function( elem ) { + + // In CSS3, :checked should return both checked and selected elements + // http://www.w3.org/TR/2011/REC-css3-selectors-20110929/#checked + var nodeName = elem.nodeName.toLowerCase(); + return ( nodeName === "input" && !!elem.checked ) || + ( nodeName === "option" && !!elem.selected ); + }, + + "selected": function( elem ) { + + // Accessing this property makes selected-by-default + // options in Safari work properly + if ( elem.parentNode ) { + // eslint-disable-next-line no-unused-expressions + elem.parentNode.selectedIndex; + } + + return elem.selected === true; + }, + + // Contents + "empty": function( elem ) { + + // http://www.w3.org/TR/selectors/#empty-pseudo + // :empty is negated by element (1) or content nodes (text: 3; cdata: 4; entity ref: 5), + // but not by others (comment: 8; processing instruction: 7; etc.) + // nodeType < 6 works because attributes (2) do not appear as children + for ( elem = elem.firstChild; elem; elem = elem.nextSibling ) { + if ( elem.nodeType < 6 ) { + return false; + } + } + return true; + }, + + "parent": function( elem ) { + return !Expr.pseudos[ "empty" ]( elem ); + }, + + // Element/input types + "header": function( elem ) { + return rheader.test( elem.nodeName ); + }, + + "input": function( elem ) { + return rinputs.test( elem.nodeName ); + }, + + "button": function( elem ) { + var name = elem.nodeName.toLowerCase(); + return name === "input" && elem.type === "button" || name === "button"; + }, + + "text": function( elem ) { + var attr; + return elem.nodeName.toLowerCase() === "input" && + elem.type === "text" && + + // Support: IE<8 + // New HTML5 attribute values (e.g., "search") appear with elem.type === "text" + ( ( attr = elem.getAttribute( "type" ) ) == null || + attr.toLowerCase() === "text" ); + }, + + // Position-in-collection + "first": createPositionalPseudo( function() { + return [ 0 ]; + } ), + + "last": createPositionalPseudo( function( _matchIndexes, length ) { + return [ length - 1 ]; + } ), + + "eq": createPositionalPseudo( function( _matchIndexes, length, argument ) { + return [ argument < 0 ? argument + length : argument ]; + } ), + + "even": createPositionalPseudo( function( matchIndexes, length ) { + var i = 0; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "odd": createPositionalPseudo( function( matchIndexes, length ) { + var i = 1; + for ( ; i < length; i += 2 ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "lt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? + argument + length : + argument > length ? + length : + argument; + for ( ; --i >= 0; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ), + + "gt": createPositionalPseudo( function( matchIndexes, length, argument ) { + var i = argument < 0 ? argument + length : argument; + for ( ; ++i < length; ) { + matchIndexes.push( i ); + } + return matchIndexes; + } ) + } +}; + +Expr.pseudos[ "nth" ] = Expr.pseudos[ "eq" ]; + +// Add button/input type pseudos +for ( i in { radio: true, checkbox: true, file: true, password: true, image: true } ) { + Expr.pseudos[ i ] = createInputPseudo( i ); +} +for ( i in { submit: true, reset: true } ) { + Expr.pseudos[ i ] = createButtonPseudo( i ); +} + +// Easy API for creating new setFilters +function setFilters() {} +setFilters.prototype = Expr.filters = Expr.pseudos; +Expr.setFilters = new setFilters(); + +tokenize = Sizzle.tokenize = function( selector, parseOnly ) { + var matched, match, tokens, type, + soFar, groups, preFilters, + cached = tokenCache[ selector + " " ]; + + if ( cached ) { + return parseOnly ? 0 : cached.slice( 0 ); + } + + soFar = selector; + groups = []; + preFilters = Expr.preFilter; + + while ( soFar ) { + + // Comma and first run + if ( !matched || ( match = rcomma.exec( soFar ) ) ) { + if ( match ) { + + // Don't consume trailing commas as valid + soFar = soFar.slice( match[ 0 ].length ) || soFar; + } + groups.push( ( tokens = [] ) ); + } + + matched = false; + + // Combinators + if ( ( match = rcombinators.exec( soFar ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + + // Cast descendant combinators to space + type: match[ 0 ].replace( rtrim, " " ) + } ); + soFar = soFar.slice( matched.length ); + } + + // Filters + for ( type in Expr.filter ) { + if ( ( match = matchExpr[ type ].exec( soFar ) ) && ( !preFilters[ type ] || + ( match = preFilters[ type ]( match ) ) ) ) { + matched = match.shift(); + tokens.push( { + value: matched, + type: type, + matches: match + } ); + soFar = soFar.slice( matched.length ); + } + } + + if ( !matched ) { + break; + } + } + + // Return the length of the invalid excess + // if we're just parsing + // Otherwise, throw an error or return tokens + return parseOnly ? + soFar.length : + soFar ? + Sizzle.error( selector ) : + + // Cache the tokens + tokenCache( selector, groups ).slice( 0 ); +}; + +function toSelector( tokens ) { + var i = 0, + len = tokens.length, + selector = ""; + for ( ; i < len; i++ ) { + selector += tokens[ i ].value; + } + return selector; +} + +function addCombinator( matcher, combinator, base ) { + var dir = combinator.dir, + skip = combinator.next, + key = skip || dir, + checkNonElements = base && key === "parentNode", + doneName = done++; + + return combinator.first ? + + // Check against closest ancestor/preceding element + function( elem, context, xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + return matcher( elem, context, xml ); + } + } + return false; + } : + + // Check against all ancestor/preceding elements + function( elem, context, xml ) { + var oldCache, uniqueCache, outerCache, + newCache = [ dirruns, doneName ]; + + // We can't set arbitrary data on XML nodes, so they don't benefit from combinator caching + if ( xml ) { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + if ( matcher( elem, context, xml ) ) { + return true; + } + } + } + } else { + while ( ( elem = elem[ dir ] ) ) { + if ( elem.nodeType === 1 || checkNonElements ) { + outerCache = elem[ expando ] || ( elem[ expando ] = {} ); + + // Support: IE <9 only + // Defend against cloned attroperties (jQuery gh-1709) + uniqueCache = outerCache[ elem.uniqueID ] || + ( outerCache[ elem.uniqueID ] = {} ); + + if ( skip && skip === elem.nodeName.toLowerCase() ) { + elem = elem[ dir ] || elem; + } else if ( ( oldCache = uniqueCache[ key ] ) && + oldCache[ 0 ] === dirruns && oldCache[ 1 ] === doneName ) { + + // Assign to newCache so results back-propagate to previous elements + return ( newCache[ 2 ] = oldCache[ 2 ] ); + } else { + + // Reuse newcache so results back-propagate to previous elements + uniqueCache[ key ] = newCache; + + // A match means we're done; a fail means we have to keep checking + if ( ( newCache[ 2 ] = matcher( elem, context, xml ) ) ) { + return true; + } + } + } + } + } + return false; + }; +} + +function elementMatcher( matchers ) { + return matchers.length > 1 ? + function( elem, context, xml ) { + var i = matchers.length; + while ( i-- ) { + if ( !matchers[ i ]( elem, context, xml ) ) { + return false; + } + } + return true; + } : + matchers[ 0 ]; +} + +function multipleContexts( selector, contexts, results ) { + var i = 0, + len = contexts.length; + for ( ; i < len; i++ ) { + Sizzle( selector, contexts[ i ], results ); + } + return results; +} + +function condense( unmatched, map, filter, context, xml ) { + var elem, + newUnmatched = [], + i = 0, + len = unmatched.length, + mapped = map != null; + + for ( ; i < len; i++ ) { + if ( ( elem = unmatched[ i ] ) ) { + if ( !filter || filter( elem, context, xml ) ) { + newUnmatched.push( elem ); + if ( mapped ) { + map.push( i ); + } + } + } + } + + return newUnmatched; +} + +function setMatcher( preFilter, selector, matcher, postFilter, postFinder, postSelector ) { + if ( postFilter && !postFilter[ expando ] ) { + postFilter = setMatcher( postFilter ); + } + if ( postFinder && !postFinder[ expando ] ) { + postFinder = setMatcher( postFinder, postSelector ); + } + return markFunction( function( seed, results, context, xml ) { + var temp, i, elem, + preMap = [], + postMap = [], + preexisting = results.length, + + // Get initial elements from seed or context + elems = seed || multipleContexts( + selector || "*", + context.nodeType ? [ context ] : context, + [] + ), + + // Prefilter to get matcher input, preserving a map for seed-results synchronization + matcherIn = preFilter && ( seed || !selector ) ? + condense( elems, preMap, preFilter, context, xml ) : + elems, + + matcherOut = matcher ? + + // If we have a postFinder, or filtered seed, or non-seed postFilter or preexisting results, + postFinder || ( seed ? preFilter : preexisting || postFilter ) ? + + // ...intermediate processing is necessary + [] : + + // ...otherwise use results directly + results : + matcherIn; + + // Find primary matches + if ( matcher ) { + matcher( matcherIn, matcherOut, context, xml ); + } + + // Apply postFilter + if ( postFilter ) { + temp = condense( matcherOut, postMap ); + postFilter( temp, [], context, xml ); + + // Un-match failing elements by moving them back to matcherIn + i = temp.length; + while ( i-- ) { + if ( ( elem = temp[ i ] ) ) { + matcherOut[ postMap[ i ] ] = !( matcherIn[ postMap[ i ] ] = elem ); + } + } + } + + if ( seed ) { + if ( postFinder || preFilter ) { + if ( postFinder ) { + + // Get the final matcherOut by condensing this intermediate into postFinder contexts + temp = []; + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) ) { + + // Restore matcherIn since elem is not yet a final match + temp.push( ( matcherIn[ i ] = elem ) ); + } + } + postFinder( null, ( matcherOut = [] ), temp, xml ); + } + + // Move matched elements from seed to results to keep them synchronized + i = matcherOut.length; + while ( i-- ) { + if ( ( elem = matcherOut[ i ] ) && + ( temp = postFinder ? indexOf( seed, elem ) : preMap[ i ] ) > -1 ) { + + seed[ temp ] = !( results[ temp ] = elem ); + } + } + } + + // Add elements to results, through postFinder if defined + } else { + matcherOut = condense( + matcherOut === results ? + matcherOut.splice( preexisting, matcherOut.length ) : + matcherOut + ); + if ( postFinder ) { + postFinder( null, results, matcherOut, xml ); + } else { + push.apply( results, matcherOut ); + } + } + } ); +} + +function matcherFromTokens( tokens ) { + var checkContext, matcher, j, + len = tokens.length, + leadingRelative = Expr.relative[ tokens[ 0 ].type ], + implicitRelative = leadingRelative || Expr.relative[ " " ], + i = leadingRelative ? 1 : 0, + + // The foundational matcher ensures that elements are reachable from top-level context(s) + matchContext = addCombinator( function( elem ) { + return elem === checkContext; + }, implicitRelative, true ), + matchAnyContext = addCombinator( function( elem ) { + return indexOf( checkContext, elem ) > -1; + }, implicitRelative, true ), + matchers = [ function( elem, context, xml ) { + var ret = ( !leadingRelative && ( xml || context !== outermostContext ) ) || ( + ( checkContext = context ).nodeType ? + matchContext( elem, context, xml ) : + matchAnyContext( elem, context, xml ) ); + + // Avoid hanging onto element (issue #299) + checkContext = null; + return ret; + } ]; + + for ( ; i < len; i++ ) { + if ( ( matcher = Expr.relative[ tokens[ i ].type ] ) ) { + matchers = [ addCombinator( elementMatcher( matchers ), matcher ) ]; + } else { + matcher = Expr.filter[ tokens[ i ].type ].apply( null, tokens[ i ].matches ); + + // Return special upon seeing a positional matcher + if ( matcher[ expando ] ) { + + // Find the next relative operator (if any) for proper handling + j = ++i; + for ( ; j < len; j++ ) { + if ( Expr.relative[ tokens[ j ].type ] ) { + break; + } + } + return setMatcher( + i > 1 && elementMatcher( matchers ), + i > 1 && toSelector( + + // If the preceding token was a descendant combinator, insert an implicit any-element `*` + tokens + .slice( 0, i - 1 ) + .concat( { value: tokens[ i - 2 ].type === " " ? "*" : "" } ) + ).replace( rtrim, "$1" ), + matcher, + i < j && matcherFromTokens( tokens.slice( i, j ) ), + j < len && matcherFromTokens( ( tokens = tokens.slice( j ) ) ), + j < len && toSelector( tokens ) + ); + } + matchers.push( matcher ); + } + } + + return elementMatcher( matchers ); +} + +function matcherFromGroupMatchers( elementMatchers, setMatchers ) { + var bySet = setMatchers.length > 0, + byElement = elementMatchers.length > 0, + superMatcher = function( seed, context, xml, results, outermost ) { + var elem, j, matcher, + matchedCount = 0, + i = "0", + unmatched = seed && [], + setMatched = [], + contextBackup = outermostContext, + + // We must always have either seed elements or outermost context + elems = seed || byElement && Expr.find[ "TAG" ]( "*", outermost ), + + // Use integer dirruns iff this is the outermost matcher + dirrunsUnique = ( dirruns += contextBackup == null ? 1 : Math.random() || 0.1 ), + len = elems.length; + + if ( outermost ) { + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + outermostContext = context == document || context || outermost; + } + + // Add elements passing elementMatchers directly to results + // Support: IE<9, Safari + // Tolerate NodeList properties (IE: "length"; Safari: ) matching elements by id + for ( ; i !== len && ( elem = elems[ i ] ) != null; i++ ) { + if ( byElement && elem ) { + j = 0; + + // Support: IE 11+, Edge 17 - 18+ + // IE/Edge sometimes throw a "Permission denied" error when strict-comparing + // two documents; shallow comparisons work. + // eslint-disable-next-line eqeqeq + if ( !context && elem.ownerDocument != document ) { + setDocument( elem ); + xml = !documentIsHTML; + } + while ( ( matcher = elementMatchers[ j++ ] ) ) { + if ( matcher( elem, context || document, xml ) ) { + results.push( elem ); + break; + } + } + if ( outermost ) { + dirruns = dirrunsUnique; + } + } + + // Track unmatched elements for set filters + if ( bySet ) { + + // They will have gone through all possible matchers + if ( ( elem = !matcher && elem ) ) { + matchedCount--; + } + + // Lengthen the array for every element, matched or not + if ( seed ) { + unmatched.push( elem ); + } + } + } + + // `i` is now the count of elements visited above, and adding it to `matchedCount` + // makes the latter nonnegative. + matchedCount += i; + + // Apply set filters to unmatched elements + // NOTE: This can be skipped if there are no unmatched elements (i.e., `matchedCount` + // equals `i`), unless we didn't visit _any_ elements in the above loop because we have + // no element matchers and no seed. + // Incrementing an initially-string "0" `i` allows `i` to remain a string only in that + // case, which will result in a "00" `matchedCount` that differs from `i` but is also + // numerically zero. + if ( bySet && i !== matchedCount ) { + j = 0; + while ( ( matcher = setMatchers[ j++ ] ) ) { + matcher( unmatched, setMatched, context, xml ); + } + + if ( seed ) { + + // Reintegrate element matches to eliminate the need for sorting + if ( matchedCount > 0 ) { + while ( i-- ) { + if ( !( unmatched[ i ] || setMatched[ i ] ) ) { + setMatched[ i ] = pop.call( results ); + } + } + } + + // Discard index placeholder values to get only actual matches + setMatched = condense( setMatched ); + } + + // Add matches to results + push.apply( results, setMatched ); + + // Seedless set matches succeeding multiple successful matchers stipulate sorting + if ( outermost && !seed && setMatched.length > 0 && + ( matchedCount + setMatchers.length ) > 1 ) { + + Sizzle.uniqueSort( results ); + } + } + + // Override manipulation of globals by nested matchers + if ( outermost ) { + dirruns = dirrunsUnique; + outermostContext = contextBackup; + } + + return unmatched; + }; + + return bySet ? + markFunction( superMatcher ) : + superMatcher; +} + +compile = Sizzle.compile = function( selector, match /* Internal Use Only */ ) { + var i, + setMatchers = [], + elementMatchers = [], + cached = compilerCache[ selector + " " ]; + + if ( !cached ) { + + // Generate a function of recursive functions that can be used to check each element + if ( !match ) { + match = tokenize( selector ); + } + i = match.length; + while ( i-- ) { + cached = matcherFromTokens( match[ i ] ); + if ( cached[ expando ] ) { + setMatchers.push( cached ); + } else { + elementMatchers.push( cached ); + } + } + + // Cache the compiled function + cached = compilerCache( + selector, + matcherFromGroupMatchers( elementMatchers, setMatchers ) + ); + + // Save selector and tokenization + cached.selector = selector; + } + return cached; +}; + +/** + * A low-level selection function that works with Sizzle's compiled + * selector functions + * @param {String|Function} selector A selector or a pre-compiled + * selector function built with Sizzle.compile + * @param {Element} context + * @param {Array} [results] + * @param {Array} [seed] A set of elements to match against + */ +select = Sizzle.select = function( selector, context, results, seed ) { + var i, tokens, token, type, find, + compiled = typeof selector === "function" && selector, + match = !seed && tokenize( ( selector = compiled.selector || selector ) ); + + results = results || []; + + // Try to minimize operations if there is only one selector in the list and no seed + // (the latter of which guarantees us context) + if ( match.length === 1 ) { + + // Reduce context if the leading compound selector is an ID + tokens = match[ 0 ] = match[ 0 ].slice( 0 ); + if ( tokens.length > 2 && ( token = tokens[ 0 ] ).type === "ID" && + context.nodeType === 9 && documentIsHTML && Expr.relative[ tokens[ 1 ].type ] ) { + + context = ( Expr.find[ "ID" ]( token.matches[ 0 ] + .replace( runescape, funescape ), context ) || [] )[ 0 ]; + if ( !context ) { + return results; + + // Precompiled matchers will still verify ancestry, so step up a level + } else if ( compiled ) { + context = context.parentNode; + } + + selector = selector.slice( tokens.shift().value.length ); + } + + // Fetch a seed set for right-to-left matching + i = matchExpr[ "needsContext" ].test( selector ) ? 0 : tokens.length; + while ( i-- ) { + token = tokens[ i ]; + + // Abort if we hit a combinator + if ( Expr.relative[ ( type = token.type ) ] ) { + break; + } + if ( ( find = Expr.find[ type ] ) ) { + + // Search, expanding context for leading sibling combinators + if ( ( seed = find( + token.matches[ 0 ].replace( runescape, funescape ), + rsibling.test( tokens[ 0 ].type ) && testContext( context.parentNode ) || + context + ) ) ) { + + // If seed is empty or no tokens remain, we can return early + tokens.splice( i, 1 ); + selector = seed.length && toSelector( tokens ); + if ( !selector ) { + push.apply( results, seed ); + return results; + } + + break; + } + } + } + } + + // Compile and execute a filtering function if one is not provided + // Provide `match` to avoid retokenization if we modified the selector above + ( compiled || compile( selector, match ) )( + seed, + context, + !documentIsHTML, + results, + !context || rsibling.test( selector ) && testContext( context.parentNode ) || context + ); + return results; +}; + +// One-time assignments + +// Sort stability +support.sortStable = expando.split( "" ).sort( sortOrder ).join( "" ) === expando; + +// Support: Chrome 14-35+ +// Always assume duplicates if they aren't passed to the comparison function +support.detectDuplicates = !!hasDuplicate; + +// Initialize against the default document +setDocument(); + +// Support: Webkit<537.32 - Safari 6.0.3/Chrome 25 (fixed in Chrome 27) +// Detached nodes confoundingly follow *each other* +support.sortDetached = assert( function( el ) { + + // Should return 1, but returns 4 (following) + return el.compareDocumentPosition( document.createElement( "fieldset" ) ) & 1; +} ); + +// Support: IE<8 +// Prevent attribute/property "interpolation" +// https://msdn.microsoft.com/en-us/library/ms536429%28VS.85%29.aspx +if ( !assert( function( el ) { + el.innerHTML = ""; + return el.firstChild.getAttribute( "href" ) === "#"; +} ) ) { + addHandle( "type|href|height|width", function( elem, name, isXML ) { + if ( !isXML ) { + return elem.getAttribute( name, name.toLowerCase() === "type" ? 1 : 2 ); + } + } ); +} + +// Support: IE<9 +// Use defaultValue in place of getAttribute("value") +if ( !support.attributes || !assert( function( el ) { + el.innerHTML = ""; + el.firstChild.setAttribute( "value", "" ); + return el.firstChild.getAttribute( "value" ) === ""; +} ) ) { + addHandle( "value", function( elem, _name, isXML ) { + if ( !isXML && elem.nodeName.toLowerCase() === "input" ) { + return elem.defaultValue; + } + } ); +} + +// Support: IE<9 +// Use getAttributeNode to fetch booleans when getAttribute lies +if ( !assert( function( el ) { + return el.getAttribute( "disabled" ) == null; +} ) ) { + addHandle( booleans, function( elem, name, isXML ) { + var val; + if ( !isXML ) { + return elem[ name ] === true ? name.toLowerCase() : + ( val = elem.getAttributeNode( name ) ) && val.specified ? + val.value : + null; + } + } ); +} + +return Sizzle; + +} )( window ); + + + +jQuery.find = Sizzle; +jQuery.expr = Sizzle.selectors; + +// Deprecated +jQuery.expr[ ":" ] = jQuery.expr.pseudos; +jQuery.uniqueSort = jQuery.unique = Sizzle.uniqueSort; +jQuery.text = Sizzle.getText; +jQuery.isXMLDoc = Sizzle.isXML; +jQuery.contains = Sizzle.contains; +jQuery.escapeSelector = Sizzle.escape; + + + + +var dir = function( elem, dir, until ) { + var matched = [], + truncate = until !== undefined; + + while ( ( elem = elem[ dir ] ) && elem.nodeType !== 9 ) { + if ( elem.nodeType === 1 ) { + if ( truncate && jQuery( elem ).is( until ) ) { + break; + } + matched.push( elem ); + } + } + return matched; +}; + + +var siblings = function( n, elem ) { + var matched = []; + + for ( ; n; n = n.nextSibling ) { + if ( n.nodeType === 1 && n !== elem ) { + matched.push( n ); + } + } + + return matched; +}; + + +var rneedsContext = jQuery.expr.match.needsContext; + + + +function nodeName( elem, name ) { + + return elem.nodeName && elem.nodeName.toLowerCase() === name.toLowerCase(); + +} +var rsingleTag = ( /^<([a-z][^\/\0>:\x20\t\r\n\f]*)[\x20\t\r\n\f]*\/?>(?:<\/\1>|)$/i ); + + + +// Implement the identical functionality for filter and not +function winnow( elements, qualifier, not ) { + if ( isFunction( qualifier ) ) { + return jQuery.grep( elements, function( elem, i ) { + return !!qualifier.call( elem, i, elem ) !== not; + } ); + } + + // Single element + if ( qualifier.nodeType ) { + return jQuery.grep( elements, function( elem ) { + return ( elem === qualifier ) !== not; + } ); + } + + // Arraylike of elements (jQuery, arguments, Array) + if ( typeof qualifier !== "string" ) { + return jQuery.grep( elements, function( elem ) { + return ( indexOf.call( qualifier, elem ) > -1 ) !== not; + } ); + } + + // Filtered directly for both simple and complex selectors + return jQuery.filter( qualifier, elements, not ); +} + +jQuery.filter = function( expr, elems, not ) { + var elem = elems[ 0 ]; + + if ( not ) { + expr = ":not(" + expr + ")"; + } + + if ( elems.length === 1 && elem.nodeType === 1 ) { + return jQuery.find.matchesSelector( elem, expr ) ? [ elem ] : []; + } + + return jQuery.find.matches( expr, jQuery.grep( elems, function( elem ) { + return elem.nodeType === 1; + } ) ); +}; + +jQuery.fn.extend( { + find: function( selector ) { + var i, ret, + len = this.length, + self = this; + + if ( typeof selector !== "string" ) { + return this.pushStack( jQuery( selector ).filter( function() { + for ( i = 0; i < len; i++ ) { + if ( jQuery.contains( self[ i ], this ) ) { + return true; + } + } + } ) ); + } + + ret = this.pushStack( [] ); + + for ( i = 0; i < len; i++ ) { + jQuery.find( selector, self[ i ], ret ); + } + + return len > 1 ? jQuery.uniqueSort( ret ) : ret; + }, + filter: function( selector ) { + return this.pushStack( winnow( this, selector || [], false ) ); + }, + not: function( selector ) { + return this.pushStack( winnow( this, selector || [], true ) ); + }, + is: function( selector ) { + return !!winnow( + this, + + // If this is a positional/relative selector, check membership in the returned set + // so $("p:first").is("p:last") won't return true for a doc with two "p". + typeof selector === "string" && rneedsContext.test( selector ) ? + jQuery( selector ) : + selector || [], + false + ).length; + } +} ); + + +// Initialize a jQuery object + + +// A central reference to the root jQuery(document) +var rootjQuery, + + // A simple way to check for HTML strings + // Prioritize #id over to avoid XSS via location.hash (#9521) + // Strict HTML recognition (#11290: must start with <) + // Shortcut simple #id case for speed + rquickExpr = /^(?:\s*(<[\w\W]+>)[^>]*|#([\w-]+))$/, + + init = jQuery.fn.init = function( selector, context, root ) { + var match, elem; + + // HANDLE: $(""), $(null), $(undefined), $(false) + if ( !selector ) { + return this; + } + + // Method init() accepts an alternate rootjQuery + // so migrate can support jQuery.sub (gh-2101) + root = root || rootjQuery; + + // Handle HTML strings + if ( typeof selector === "string" ) { + if ( selector[ 0 ] === "<" && + selector[ selector.length - 1 ] === ">" && + selector.length >= 3 ) { + + // Assume that strings that start and end with <> are HTML and skip the regex check + match = [ null, selector, null ]; + + } else { + match = rquickExpr.exec( selector ); + } + + // Match html or make sure no context is specified for #id + if ( match && ( match[ 1 ] || !context ) ) { + + // HANDLE: $(html) -> $(array) + if ( match[ 1 ] ) { + context = context instanceof jQuery ? context[ 0 ] : context; + + // Option to run scripts is true for back-compat + // Intentionally let the error be thrown if parseHTML is not present + jQuery.merge( this, jQuery.parseHTML( + match[ 1 ], + context && context.nodeType ? context.ownerDocument || context : document, + true + ) ); + + // HANDLE: $(html, props) + if ( rsingleTag.test( match[ 1 ] ) && jQuery.isPlainObject( context ) ) { + for ( match in context ) { + + // Properties of context are called as methods if possible + if ( isFunction( this[ match ] ) ) { + this[ match ]( context[ match ] ); + + // ...and otherwise set as attributes + } else { + this.attr( match, context[ match ] ); + } + } + } + + return this; + + // HANDLE: $(#id) + } else { + elem = document.getElementById( match[ 2 ] ); + + if ( elem ) { + + // Inject the element directly into the jQuery object + this[ 0 ] = elem; + this.length = 1; + } + return this; + } + + // HANDLE: $(expr, $(...)) + } else if ( !context || context.jquery ) { + return ( context || root ).find( selector ); + + // HANDLE: $(expr, context) + // (which is just equivalent to: $(context).find(expr) + } else { + return this.constructor( context ).find( selector ); + } + + // HANDLE: $(DOMElement) + } else if ( selector.nodeType ) { + this[ 0 ] = selector; + this.length = 1; + return this; + + // HANDLE: $(function) + // Shortcut for document ready + } else if ( isFunction( selector ) ) { + return root.ready !== undefined ? + root.ready( selector ) : + + // Execute immediately if ready is not present + selector( jQuery ); + } + + return jQuery.makeArray( selector, this ); + }; + +// Give the init function the jQuery prototype for later instantiation +init.prototype = jQuery.fn; + +// Initialize central reference +rootjQuery = jQuery( document ); + + +var rparentsprev = /^(?:parents|prev(?:Until|All))/, + + // Methods guaranteed to produce a unique set when starting from a unique set + guaranteedUnique = { + children: true, + contents: true, + next: true, + prev: true + }; + +jQuery.fn.extend( { + has: function( target ) { + var targets = jQuery( target, this ), + l = targets.length; + + return this.filter( function() { + var i = 0; + for ( ; i < l; i++ ) { + if ( jQuery.contains( this, targets[ i ] ) ) { + return true; + } + } + } ); + }, + + closest: function( selectors, context ) { + var cur, + i = 0, + l = this.length, + matched = [], + targets = typeof selectors !== "string" && jQuery( selectors ); + + // Positional selectors never match, since there's no _selection_ context + if ( !rneedsContext.test( selectors ) ) { + for ( ; i < l; i++ ) { + for ( cur = this[ i ]; cur && cur !== context; cur = cur.parentNode ) { + + // Always skip document fragments + if ( cur.nodeType < 11 && ( targets ? + targets.index( cur ) > -1 : + + // Don't pass non-elements to Sizzle + cur.nodeType === 1 && + jQuery.find.matchesSelector( cur, selectors ) ) ) { + + matched.push( cur ); + break; + } + } + } + } + + return this.pushStack( matched.length > 1 ? jQuery.uniqueSort( matched ) : matched ); + }, + + // Determine the position of an element within the set + index: function( elem ) { + + // No argument, return index in parent + if ( !elem ) { + return ( this[ 0 ] && this[ 0 ].parentNode ) ? this.first().prevAll().length : -1; + } + + // Index in selector + if ( typeof elem === "string" ) { + return indexOf.call( jQuery( elem ), this[ 0 ] ); + } + + // Locate the position of the desired element + return indexOf.call( this, + + // If it receives a jQuery object, the first element is used + elem.jquery ? elem[ 0 ] : elem + ); + }, + + add: function( selector, context ) { + return this.pushStack( + jQuery.uniqueSort( + jQuery.merge( this.get(), jQuery( selector, context ) ) + ) + ); + }, + + addBack: function( selector ) { + return this.add( selector == null ? + this.prevObject : this.prevObject.filter( selector ) + ); + } +} ); + +function sibling( cur, dir ) { + while ( ( cur = cur[ dir ] ) && cur.nodeType !== 1 ) {} + return cur; +} + +jQuery.each( { + parent: function( elem ) { + var parent = elem.parentNode; + return parent && parent.nodeType !== 11 ? parent : null; + }, + parents: function( elem ) { + return dir( elem, "parentNode" ); + }, + parentsUntil: function( elem, _i, until ) { + return dir( elem, "parentNode", until ); + }, + next: function( elem ) { + return sibling( elem, "nextSibling" ); + }, + prev: function( elem ) { + return sibling( elem, "previousSibling" ); + }, + nextAll: function( elem ) { + return dir( elem, "nextSibling" ); + }, + prevAll: function( elem ) { + return dir( elem, "previousSibling" ); + }, + nextUntil: function( elem, _i, until ) { + return dir( elem, "nextSibling", until ); + }, + prevUntil: function( elem, _i, until ) { + return dir( elem, "previousSibling", until ); + }, + siblings: function( elem ) { + return siblings( ( elem.parentNode || {} ).firstChild, elem ); + }, + children: function( elem ) { + return siblings( elem.firstChild ); + }, + contents: function( elem ) { + if ( elem.contentDocument != null && + + // Support: IE 11+ + // elements with no `data` attribute has an object + // `contentDocument` with a `null` prototype. + getProto( elem.contentDocument ) ) { + + return elem.contentDocument; + } + + // Support: IE 9 - 11 only, iOS 7 only, Android Browser <=4.3 only + // Treat the template element as a regular one in browsers that + // don't support it. + if ( nodeName( elem, "template" ) ) { + elem = elem.content || elem; + } + + return jQuery.merge( [], elem.childNodes ); + } +}, function( name, fn ) { + jQuery.fn[ name ] = function( until, selector ) { + var matched = jQuery.map( this, fn, until ); + + if ( name.slice( -5 ) !== "Until" ) { + selector = until; + } + + if ( selector && typeof selector === "string" ) { + matched = jQuery.filter( selector, matched ); + } + + if ( this.length > 1 ) { + + // Remove duplicates + if ( !guaranteedUnique[ name ] ) { + jQuery.uniqueSort( matched ); + } + + // Reverse order for parents* and prev-derivatives + if ( rparentsprev.test( name ) ) { + matched.reverse(); + } + } + + return this.pushStack( matched ); + }; +} ); +var rnothtmlwhite = ( /[^\x20\t\r\n\f]+/g ); + + + +// Convert String-formatted options into Object-formatted ones +function createOptions( options ) { + var object = {}; + jQuery.each( options.match( rnothtmlwhite ) || [], function( _, flag ) { + object[ flag ] = true; + } ); + return object; +} + +/* + * Create a callback list using the following parameters: + * + * options: an optional list of space-separated options that will change how + * the callback list behaves or a more traditional option object + * + * By default a callback list will act like an event callback list and can be + * "fired" multiple times. + * + * Possible options: + * + * once: will ensure the callback list can only be fired once (like a Deferred) + * + * memory: will keep track of previous values and will call any callback added + * after the list has been fired right away with the latest "memorized" + * values (like a Deferred) + * + * unique: will ensure a callback can only be added once (no duplicate in the list) + * + * stopOnFalse: interrupt callings when a callback returns false + * + */ +jQuery.Callbacks = function( options ) { + + // Convert options from String-formatted to Object-formatted if needed + // (we check in cache first) + options = typeof options === "string" ? + createOptions( options ) : + jQuery.extend( {}, options ); + + var // Flag to know if list is currently firing + firing, + + // Last fire value for non-forgettable lists + memory, + + // Flag to know if list was already fired + fired, + + // Flag to prevent firing + locked, + + // Actual callback list + list = [], + + // Queue of execution data for repeatable lists + queue = [], + + // Index of currently firing callback (modified by add/remove as needed) + firingIndex = -1, + + // Fire callbacks + fire = function() { + + // Enforce single-firing + locked = locked || options.once; + + // Execute callbacks for all pending executions, + // respecting firingIndex overrides and runtime changes + fired = firing = true; + for ( ; queue.length; firingIndex = -1 ) { + memory = queue.shift(); + while ( ++firingIndex < list.length ) { + + // Run callback and check for early termination + if ( list[ firingIndex ].apply( memory[ 0 ], memory[ 1 ] ) === false && + options.stopOnFalse ) { + + // Jump to end and forget the data so .add doesn't re-fire + firingIndex = list.length; + memory = false; + } + } + } + + // Forget the data if we're done with it + if ( !options.memory ) { + memory = false; + } + + firing = false; + + // Clean up if we're done firing for good + if ( locked ) { + + // Keep an empty list if we have data for future add calls + if ( memory ) { + list = []; + + // Otherwise, this object is spent + } else { + list = ""; + } + } + }, + + // Actual Callbacks object + self = { + + // Add a callback or a collection of callbacks to the list + add: function() { + if ( list ) { + + // If we have memory from a past run, we should fire after adding + if ( memory && !firing ) { + firingIndex = list.length - 1; + queue.push( memory ); + } + + ( function add( args ) { + jQuery.each( args, function( _, arg ) { + if ( isFunction( arg ) ) { + if ( !options.unique || !self.has( arg ) ) { + list.push( arg ); + } + } else if ( arg && arg.length && toType( arg ) !== "string" ) { + + // Inspect recursively + add( arg ); + } + } ); + } )( arguments ); + + if ( memory && !firing ) { + fire(); + } + } + return this; + }, + + // Remove a callback from the list + remove: function() { + jQuery.each( arguments, function( _, arg ) { + var index; + while ( ( index = jQuery.inArray( arg, list, index ) ) > -1 ) { + list.splice( index, 1 ); + + // Handle firing indexes + if ( index <= firingIndex ) { + firingIndex--; + } + } + } ); + return this; + }, + + // Check if a given callback is in the list. + // If no argument is given, return whether or not list has callbacks attached. + has: function( fn ) { + return fn ? + jQuery.inArray( fn, list ) > -1 : + list.length > 0; + }, + + // Remove all callbacks from the list + empty: function() { + if ( list ) { + list = []; + } + return this; + }, + + // Disable .fire and .add + // Abort any current/pending executions + // Clear all callbacks and values + disable: function() { + locked = queue = []; + list = memory = ""; + return this; + }, + disabled: function() { + return !list; + }, + + // Disable .fire + // Also disable .add unless we have memory (since it would have no effect) + // Abort any pending executions + lock: function() { + locked = queue = []; + if ( !memory && !firing ) { + list = memory = ""; + } + return this; + }, + locked: function() { + return !!locked; + }, + + // Call all callbacks with the given context and arguments + fireWith: function( context, args ) { + if ( !locked ) { + args = args || []; + args = [ context, args.slice ? args.slice() : args ]; + queue.push( args ); + if ( !firing ) { + fire(); + } + } + return this; + }, + + // Call all the callbacks with the given arguments + fire: function() { + self.fireWith( this, arguments ); + return this; + }, + + // To know if the callbacks have already been called at least once + fired: function() { + return !!fired; + } + }; + + return self; +}; + + +function Identity( v ) { + return v; +} +function Thrower( ex ) { + throw ex; +} + +function adoptValue( value, resolve, reject, noValue ) { + var method; + + try { + + // Check for promise aspect first to privilege synchronous behavior + if ( value && isFunction( ( method = value.promise ) ) ) { + method.call( value ).done( resolve ).fail( reject ); + + // Other thenables + } else if ( value && isFunction( ( method = value.then ) ) ) { + method.call( value, resolve, reject ); + + // Other non-thenables + } else { + + // Control `resolve` arguments by letting Array#slice cast boolean `noValue` to integer: + // * false: [ value ].slice( 0 ) => resolve( value ) + // * true: [ value ].slice( 1 ) => resolve() + resolve.apply( undefined, [ value ].slice( noValue ) ); + } + + // For Promises/A+, convert exceptions into rejections + // Since jQuery.when doesn't unwrap thenables, we can skip the extra checks appearing in + // Deferred#then to conditionally suppress rejection. + } catch ( value ) { + + // Support: Android 4.0 only + // Strict mode functions invoked without .call/.apply get global-object context + reject.apply( undefined, [ value ] ); + } +} + +jQuery.extend( { + + Deferred: function( func ) { + var tuples = [ + + // action, add listener, callbacks, + // ... .then handlers, argument index, [final state] + [ "notify", "progress", jQuery.Callbacks( "memory" ), + jQuery.Callbacks( "memory" ), 2 ], + [ "resolve", "done", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 0, "resolved" ], + [ "reject", "fail", jQuery.Callbacks( "once memory" ), + jQuery.Callbacks( "once memory" ), 1, "rejected" ] + ], + state = "pending", + promise = { + state: function() { + return state; + }, + always: function() { + deferred.done( arguments ).fail( arguments ); + return this; + }, + "catch": function( fn ) { + return promise.then( null, fn ); + }, + + // Keep pipe for back-compat + pipe: function( /* fnDone, fnFail, fnProgress */ ) { + var fns = arguments; + + return jQuery.Deferred( function( newDefer ) { + jQuery.each( tuples, function( _i, tuple ) { + + // Map tuples (progress, done, fail) to arguments (done, fail, progress) + var fn = isFunction( fns[ tuple[ 4 ] ] ) && fns[ tuple[ 4 ] ]; + + // deferred.progress(function() { bind to newDefer or newDefer.notify }) + // deferred.done(function() { bind to newDefer or newDefer.resolve }) + // deferred.fail(function() { bind to newDefer or newDefer.reject }) + deferred[ tuple[ 1 ] ]( function() { + var returned = fn && fn.apply( this, arguments ); + if ( returned && isFunction( returned.promise ) ) { + returned.promise() + .progress( newDefer.notify ) + .done( newDefer.resolve ) + .fail( newDefer.reject ); + } else { + newDefer[ tuple[ 0 ] + "With" ]( + this, + fn ? [ returned ] : arguments + ); + } + } ); + } ); + fns = null; + } ).promise(); + }, + then: function( onFulfilled, onRejected, onProgress ) { + var maxDepth = 0; + function resolve( depth, deferred, handler, special ) { + return function() { + var that = this, + args = arguments, + mightThrow = function() { + var returned, then; + + // Support: Promises/A+ section 2.3.3.3.3 + // https://promisesaplus.com/#point-59 + // Ignore double-resolution attempts + if ( depth < maxDepth ) { + return; + } + + returned = handler.apply( that, args ); + + // Support: Promises/A+ section 2.3.1 + // https://promisesaplus.com/#point-48 + if ( returned === deferred.promise() ) { + throw new TypeError( "Thenable self-resolution" ); + } + + // Support: Promises/A+ sections 2.3.3.1, 3.5 + // https://promisesaplus.com/#point-54 + // https://promisesaplus.com/#point-75 + // Retrieve `then` only once + then = returned && + + // Support: Promises/A+ section 2.3.4 + // https://promisesaplus.com/#point-64 + // Only check objects and functions for thenability + ( typeof returned === "object" || + typeof returned === "function" ) && + returned.then; + + // Handle a returned thenable + if ( isFunction( then ) ) { + + // Special processors (notify) just wait for resolution + if ( special ) { + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ) + ); + + // Normal processors (resolve) also hook into progress + } else { + + // ...and disregard older resolution values + maxDepth++; + + then.call( + returned, + resolve( maxDepth, deferred, Identity, special ), + resolve( maxDepth, deferred, Thrower, special ), + resolve( maxDepth, deferred, Identity, + deferred.notifyWith ) + ); + } + + // Handle all other returned values + } else { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Identity ) { + that = undefined; + args = [ returned ]; + } + + // Process the value(s) + // Default process is resolve + ( special || deferred.resolveWith )( that, args ); + } + }, + + // Only normal processors (resolve) catch and reject exceptions + process = special ? + mightThrow : + function() { + try { + mightThrow(); + } catch ( e ) { + + if ( jQuery.Deferred.exceptionHook ) { + jQuery.Deferred.exceptionHook( e, + process.stackTrace ); + } + + // Support: Promises/A+ section 2.3.3.3.4.1 + // https://promisesaplus.com/#point-61 + // Ignore post-resolution exceptions + if ( depth + 1 >= maxDepth ) { + + // Only substitute handlers pass on context + // and multiple values (non-spec behavior) + if ( handler !== Thrower ) { + that = undefined; + args = [ e ]; + } + + deferred.rejectWith( that, args ); + } + } + }; + + // Support: Promises/A+ section 2.3.3.3.1 + // https://promisesaplus.com/#point-57 + // Re-resolve promises immediately to dodge false rejection from + // subsequent errors + if ( depth ) { + process(); + } else { + + // Call an optional hook to record the stack, in case of exception + // since it's otherwise lost when execution goes async + if ( jQuery.Deferred.getStackHook ) { + process.stackTrace = jQuery.Deferred.getStackHook(); + } + window.setTimeout( process ); + } + }; + } + + return jQuery.Deferred( function( newDefer ) { + + // progress_handlers.add( ... ) + tuples[ 0 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onProgress ) ? + onProgress : + Identity, + newDefer.notifyWith + ) + ); + + // fulfilled_handlers.add( ... ) + tuples[ 1 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onFulfilled ) ? + onFulfilled : + Identity + ) + ); + + // rejected_handlers.add( ... ) + tuples[ 2 ][ 3 ].add( + resolve( + 0, + newDefer, + isFunction( onRejected ) ? + onRejected : + Thrower + ) + ); + } ).promise(); + }, + + // Get a promise for this deferred + // If obj is provided, the promise aspect is added to the object + promise: function( obj ) { + return obj != null ? jQuery.extend( obj, promise ) : promise; + } + }, + deferred = {}; + + // Add list-specific methods + jQuery.each( tuples, function( i, tuple ) { + var list = tuple[ 2 ], + stateString = tuple[ 5 ]; + + // promise.progress = list.add + // promise.done = list.add + // promise.fail = list.add + promise[ tuple[ 1 ] ] = list.add; + + // Handle state + if ( stateString ) { + list.add( + function() { + + // state = "resolved" (i.e., fulfilled) + // state = "rejected" + state = stateString; + }, + + // rejected_callbacks.disable + // fulfilled_callbacks.disable + tuples[ 3 - i ][ 2 ].disable, + + // rejected_handlers.disable + // fulfilled_handlers.disable + tuples[ 3 - i ][ 3 ].disable, + + // progress_callbacks.lock + tuples[ 0 ][ 2 ].lock, + + // progress_handlers.lock + tuples[ 0 ][ 3 ].lock + ); + } + + // progress_handlers.fire + // fulfilled_handlers.fire + // rejected_handlers.fire + list.add( tuple[ 3 ].fire ); + + // deferred.notify = function() { deferred.notifyWith(...) } + // deferred.resolve = function() { deferred.resolveWith(...) } + // deferred.reject = function() { deferred.rejectWith(...) } + deferred[ tuple[ 0 ] ] = function() { + deferred[ tuple[ 0 ] + "With" ]( this === deferred ? undefined : this, arguments ); + return this; + }; + + // deferred.notifyWith = list.fireWith + // deferred.resolveWith = list.fireWith + // deferred.rejectWith = list.fireWith + deferred[ tuple[ 0 ] + "With" ] = list.fireWith; + } ); + + // Make the deferred a promise + promise.promise( deferred ); + + // Call given func if any + if ( func ) { + func.call( deferred, deferred ); + } + + // All done! + return deferred; + }, + + // Deferred helper + when: function( singleValue ) { + var + + // count of uncompleted subordinates + remaining = arguments.length, + + // count of unprocessed arguments + i = remaining, + + // subordinate fulfillment data + resolveContexts = Array( i ), + resolveValues = slice.call( arguments ), + + // the primary Deferred + primary = jQuery.Deferred(), + + // subordinate callback factory + updateFunc = function( i ) { + return function( value ) { + resolveContexts[ i ] = this; + resolveValues[ i ] = arguments.length > 1 ? slice.call( arguments ) : value; + if ( !( --remaining ) ) { + primary.resolveWith( resolveContexts, resolveValues ); + } + }; + }; + + // Single- and empty arguments are adopted like Promise.resolve + if ( remaining <= 1 ) { + adoptValue( singleValue, primary.done( updateFunc( i ) ).resolve, primary.reject, + !remaining ); + + // Use .then() to unwrap secondary thenables (cf. gh-3000) + if ( primary.state() === "pending" || + isFunction( resolveValues[ i ] && resolveValues[ i ].then ) ) { + + return primary.then(); + } + } + + // Multiple arguments are aggregated like Promise.all array elements + while ( i-- ) { + adoptValue( resolveValues[ i ], updateFunc( i ), primary.reject ); + } + + return primary.promise(); + } +} ); + + +// These usually indicate a programmer mistake during development, +// warn about them ASAP rather than swallowing them by default. +var rerrorNames = /^(Eval|Internal|Range|Reference|Syntax|Type|URI)Error$/; + +jQuery.Deferred.exceptionHook = function( error, stack ) { + + // Support: IE 8 - 9 only + // Console exists when dev tools are open, which can happen at any time + if ( window.console && window.console.warn && error && rerrorNames.test( error.name ) ) { + window.console.warn( "jQuery.Deferred exception: " + error.message, error.stack, stack ); + } +}; + + + + +jQuery.readyException = function( error ) { + window.setTimeout( function() { + throw error; + } ); +}; + + + + +// The deferred used on DOM ready +var readyList = jQuery.Deferred(); + +jQuery.fn.ready = function( fn ) { + + readyList + .then( fn ) + + // Wrap jQuery.readyException in a function so that the lookup + // happens at the time of error handling instead of callback + // registration. + .catch( function( error ) { + jQuery.readyException( error ); + } ); + + return this; +}; + +jQuery.extend( { + + // Is the DOM ready to be used? Set to true once it occurs. + isReady: false, + + // A counter to track how many items to wait for before + // the ready event fires. See #6781 + readyWait: 1, + + // Handle when the DOM is ready + ready: function( wait ) { + + // Abort if there are pending holds or we're already ready + if ( wait === true ? --jQuery.readyWait : jQuery.isReady ) { + return; + } + + // Remember that the DOM is ready + jQuery.isReady = true; + + // If a normal DOM Ready event fired, decrement, and wait if need be + if ( wait !== true && --jQuery.readyWait > 0 ) { + return; + } + + // If there are functions bound, to execute + readyList.resolveWith( document, [ jQuery ] ); + } +} ); + +jQuery.ready.then = readyList.then; + +// The ready event handler and self cleanup method +function completed() { + document.removeEventListener( "DOMContentLoaded", completed ); + window.removeEventListener( "load", completed ); + jQuery.ready(); +} + +// Catch cases where $(document).ready() is called +// after the browser event has already occurred. +// Support: IE <=9 - 10 only +// Older IE sometimes signals "interactive" too soon +if ( document.readyState === "complete" || + ( document.readyState !== "loading" && !document.documentElement.doScroll ) ) { + + // Handle it asynchronously to allow scripts the opportunity to delay ready + window.setTimeout( jQuery.ready ); + +} else { + + // Use the handy event callback + document.addEventListener( "DOMContentLoaded", completed ); + + // A fallback to window.onload, that will always work + window.addEventListener( "load", completed ); +} + + + + +// Multifunctional method to get and set values of a collection +// The value/s can optionally be executed if it's a function +var access = function( elems, fn, key, value, chainable, emptyGet, raw ) { + var i = 0, + len = elems.length, + bulk = key == null; + + // Sets many values + if ( toType( key ) === "object" ) { + chainable = true; + for ( i in key ) { + access( elems, fn, i, key[ i ], true, emptyGet, raw ); + } + + // Sets one value + } else if ( value !== undefined ) { + chainable = true; + + if ( !isFunction( value ) ) { + raw = true; + } + + if ( bulk ) { + + // Bulk operations run against the entire set + if ( raw ) { + fn.call( elems, value ); + fn = null; + + // ...except when executing function values + } else { + bulk = fn; + fn = function( elem, _key, value ) { + return bulk.call( jQuery( elem ), value ); + }; + } + } + + if ( fn ) { + for ( ; i < len; i++ ) { + fn( + elems[ i ], key, raw ? + value : + value.call( elems[ i ], i, fn( elems[ i ], key ) ) + ); + } + } + } + + if ( chainable ) { + return elems; + } + + // Gets + if ( bulk ) { + return fn.call( elems ); + } + + return len ? fn( elems[ 0 ], key ) : emptyGet; +}; + + +// Matches dashed string for camelizing +var rmsPrefix = /^-ms-/, + rdashAlpha = /-([a-z])/g; + +// Used by camelCase as callback to replace() +function fcamelCase( _all, letter ) { + return letter.toUpperCase(); +} + +// Convert dashed to camelCase; used by the css and data modules +// Support: IE <=9 - 11, Edge 12 - 15 +// Microsoft forgot to hump their vendor prefix (#9572) +function camelCase( string ) { + return string.replace( rmsPrefix, "ms-" ).replace( rdashAlpha, fcamelCase ); +} +var acceptData = function( owner ) { + + // Accepts only: + // - Node + // - Node.ELEMENT_NODE + // - Node.DOCUMENT_NODE + // - Object + // - Any + return owner.nodeType === 1 || owner.nodeType === 9 || !( +owner.nodeType ); +}; + + + + +function Data() { + this.expando = jQuery.expando + Data.uid++; +} + +Data.uid = 1; + +Data.prototype = { + + cache: function( owner ) { + + // Check if the owner object already has a cache + var value = owner[ this.expando ]; + + // If not, create one + if ( !value ) { + value = {}; + + // We can accept data for non-element nodes in modern browsers, + // but we should not, see #8335. + // Always return an empty object. + if ( acceptData( owner ) ) { + + // If it is a node unlikely to be stringify-ed or looped over + // use plain assignment + if ( owner.nodeType ) { + owner[ this.expando ] = value; + + // Otherwise secure it in a non-enumerable property + // configurable must be true to allow the property to be + // deleted when data is removed + } else { + Object.defineProperty( owner, this.expando, { + value: value, + configurable: true + } ); + } + } + } + + return value; + }, + set: function( owner, data, value ) { + var prop, + cache = this.cache( owner ); + + // Handle: [ owner, key, value ] args + // Always use camelCase key (gh-2257) + if ( typeof data === "string" ) { + cache[ camelCase( data ) ] = value; + + // Handle: [ owner, { properties } ] args + } else { + + // Copy the properties one-by-one to the cache object + for ( prop in data ) { + cache[ camelCase( prop ) ] = data[ prop ]; + } + } + return cache; + }, + get: function( owner, key ) { + return key === undefined ? + this.cache( owner ) : + + // Always use camelCase key (gh-2257) + owner[ this.expando ] && owner[ this.expando ][ camelCase( key ) ]; + }, + access: function( owner, key, value ) { + + // In cases where either: + // + // 1. No key was specified + // 2. A string key was specified, but no value provided + // + // Take the "read" path and allow the get method to determine + // which value to return, respectively either: + // + // 1. The entire cache object + // 2. The data stored at the key + // + if ( key === undefined || + ( ( key && typeof key === "string" ) && value === undefined ) ) { + + return this.get( owner, key ); + } + + // When the key is not a string, or both a key and value + // are specified, set or extend (existing objects) with either: + // + // 1. An object of properties + // 2. A key and value + // + this.set( owner, key, value ); + + // Since the "set" path can have two possible entry points + // return the expected data based on which path was taken[*] + return value !== undefined ? value : key; + }, + remove: function( owner, key ) { + var i, + cache = owner[ this.expando ]; + + if ( cache === undefined ) { + return; + } + + if ( key !== undefined ) { + + // Support array or space separated string of keys + if ( Array.isArray( key ) ) { + + // If key is an array of keys... + // We always set camelCase keys, so remove that. + key = key.map( camelCase ); + } else { + key = camelCase( key ); + + // If a key with the spaces exists, use it. + // Otherwise, create an array by matching non-whitespace + key = key in cache ? + [ key ] : + ( key.match( rnothtmlwhite ) || [] ); + } + + i = key.length; + + while ( i-- ) { + delete cache[ key[ i ] ]; + } + } + + // Remove the expando if there's no more data + if ( key === undefined || jQuery.isEmptyObject( cache ) ) { + + // Support: Chrome <=35 - 45 + // Webkit & Blink performance suffers when deleting properties + // from DOM nodes, so set to undefined instead + // https://bugs.chromium.org/p/chromium/issues/detail?id=378607 (bug restricted) + if ( owner.nodeType ) { + owner[ this.expando ] = undefined; + } else { + delete owner[ this.expando ]; + } + } + }, + hasData: function( owner ) { + var cache = owner[ this.expando ]; + return cache !== undefined && !jQuery.isEmptyObject( cache ); + } +}; +var dataPriv = new Data(); + +var dataUser = new Data(); + + + +// Implementation Summary +// +// 1. Enforce API surface and semantic compatibility with 1.9.x branch +// 2. Improve the module's maintainability by reducing the storage +// paths to a single mechanism. +// 3. Use the same single mechanism to support "private" and "user" data. +// 4. _Never_ expose "private" data to user code (TODO: Drop _data, _removeData) +// 5. Avoid exposing implementation details on user objects (eg. expando properties) +// 6. Provide a clear path for implementation upgrade to WeakMap in 2014 + +var rbrace = /^(?:\{[\w\W]*\}|\[[\w\W]*\])$/, + rmultiDash = /[A-Z]/g; + +function getData( data ) { + if ( data === "true" ) { + return true; + } + + if ( data === "false" ) { + return false; + } + + if ( data === "null" ) { + return null; + } + + // Only convert to a number if it doesn't change the string + if ( data === +data + "" ) { + return +data; + } + + if ( rbrace.test( data ) ) { + return JSON.parse( data ); + } + + return data; +} + +function dataAttr( elem, key, data ) { + var name; + + // If nothing was found internally, try to fetch any + // data from the HTML5 data-* attribute + if ( data === undefined && elem.nodeType === 1 ) { + name = "data-" + key.replace( rmultiDash, "-$&" ).toLowerCase(); + data = elem.getAttribute( name ); + + if ( typeof data === "string" ) { + try { + data = getData( data ); + } catch ( e ) {} + + // Make sure we set the data so it isn't changed later + dataUser.set( elem, key, data ); + } else { + data = undefined; + } + } + return data; +} + +jQuery.extend( { + hasData: function( elem ) { + return dataUser.hasData( elem ) || dataPriv.hasData( elem ); + }, + + data: function( elem, name, data ) { + return dataUser.access( elem, name, data ); + }, + + removeData: function( elem, name ) { + dataUser.remove( elem, name ); + }, + + // TODO: Now that all calls to _data and _removeData have been replaced + // with direct calls to dataPriv methods, these can be deprecated. + _data: function( elem, name, data ) { + return dataPriv.access( elem, name, data ); + }, + + _removeData: function( elem, name ) { + dataPriv.remove( elem, name ); + } +} ); + +jQuery.fn.extend( { + data: function( key, value ) { + var i, name, data, + elem = this[ 0 ], + attrs = elem && elem.attributes; + + // Gets all values + if ( key === undefined ) { + if ( this.length ) { + data = dataUser.get( elem ); + + if ( elem.nodeType === 1 && !dataPriv.get( elem, "hasDataAttrs" ) ) { + i = attrs.length; + while ( i-- ) { + + // Support: IE 11 only + // The attrs elements can be null (#14894) + if ( attrs[ i ] ) { + name = attrs[ i ].name; + if ( name.indexOf( "data-" ) === 0 ) { + name = camelCase( name.slice( 5 ) ); + dataAttr( elem, name, data[ name ] ); + } + } + } + dataPriv.set( elem, "hasDataAttrs", true ); + } + } + + return data; + } + + // Sets multiple values + if ( typeof key === "object" ) { + return this.each( function() { + dataUser.set( this, key ); + } ); + } + + return access( this, function( value ) { + var data; + + // The calling jQuery object (element matches) is not empty + // (and therefore has an element appears at this[ 0 ]) and the + // `value` parameter was not undefined. An empty jQuery object + // will result in `undefined` for elem = this[ 0 ] which will + // throw an exception if an attempt to read a data cache is made. + if ( elem && value === undefined ) { + + // Attempt to get data from the cache + // The key will always be camelCased in Data + data = dataUser.get( elem, key ); + if ( data !== undefined ) { + return data; + } + + // Attempt to "discover" the data in + // HTML5 custom data-* attrs + data = dataAttr( elem, key ); + if ( data !== undefined ) { + return data; + } + + // We tried really hard, but the data doesn't exist. + return; + } + + // Set the data... + this.each( function() { + + // We always store the camelCased key + dataUser.set( this, key, value ); + } ); + }, null, value, arguments.length > 1, null, true ); + }, + + removeData: function( key ) { + return this.each( function() { + dataUser.remove( this, key ); + } ); + } +} ); + + +jQuery.extend( { + queue: function( elem, type, data ) { + var queue; + + if ( elem ) { + type = ( type || "fx" ) + "queue"; + queue = dataPriv.get( elem, type ); + + // Speed up dequeue by getting out quickly if this is just a lookup + if ( data ) { + if ( !queue || Array.isArray( data ) ) { + queue = dataPriv.access( elem, type, jQuery.makeArray( data ) ); + } else { + queue.push( data ); + } + } + return queue || []; + } + }, + + dequeue: function( elem, type ) { + type = type || "fx"; + + var queue = jQuery.queue( elem, type ), + startLength = queue.length, + fn = queue.shift(), + hooks = jQuery._queueHooks( elem, type ), + next = function() { + jQuery.dequeue( elem, type ); + }; + + // If the fx queue is dequeued, always remove the progress sentinel + if ( fn === "inprogress" ) { + fn = queue.shift(); + startLength--; + } + + if ( fn ) { + + // Add a progress sentinel to prevent the fx queue from being + // automatically dequeued + if ( type === "fx" ) { + queue.unshift( "inprogress" ); + } + + // Clear up the last queue stop function + delete hooks.stop; + fn.call( elem, next, hooks ); + } + + if ( !startLength && hooks ) { + hooks.empty.fire(); + } + }, + + // Not public - generate a queueHooks object, or return the current one + _queueHooks: function( elem, type ) { + var key = type + "queueHooks"; + return dataPriv.get( elem, key ) || dataPriv.access( elem, key, { + empty: jQuery.Callbacks( "once memory" ).add( function() { + dataPriv.remove( elem, [ type + "queue", key ] ); + } ) + } ); + } +} ); + +jQuery.fn.extend( { + queue: function( type, data ) { + var setter = 2; + + if ( typeof type !== "string" ) { + data = type; + type = "fx"; + setter--; + } + + if ( arguments.length < setter ) { + return jQuery.queue( this[ 0 ], type ); + } + + return data === undefined ? + this : + this.each( function() { + var queue = jQuery.queue( this, type, data ); + + // Ensure a hooks for this queue + jQuery._queueHooks( this, type ); + + if ( type === "fx" && queue[ 0 ] !== "inprogress" ) { + jQuery.dequeue( this, type ); + } + } ); + }, + dequeue: function( type ) { + return this.each( function() { + jQuery.dequeue( this, type ); + } ); + }, + clearQueue: function( type ) { + return this.queue( type || "fx", [] ); + }, + + // Get a promise resolved when queues of a certain type + // are emptied (fx is the type by default) + promise: function( type, obj ) { + var tmp, + count = 1, + defer = jQuery.Deferred(), + elements = this, + i = this.length, + resolve = function() { + if ( !( --count ) ) { + defer.resolveWith( elements, [ elements ] ); + } + }; + + if ( typeof type !== "string" ) { + obj = type; + type = undefined; + } + type = type || "fx"; + + while ( i-- ) { + tmp = dataPriv.get( elements[ i ], type + "queueHooks" ); + if ( tmp && tmp.empty ) { + count++; + tmp.empty.add( resolve ); + } + } + resolve(); + return defer.promise( obj ); + } +} ); +var pnum = ( /[+-]?(?:\d*\.|)\d+(?:[eE][+-]?\d+|)/ ).source; + +var rcssNum = new RegExp( "^(?:([+-])=|)(" + pnum + ")([a-z%]*)$", "i" ); + + +var cssExpand = [ "Top", "Right", "Bottom", "Left" ]; + +var documentElement = document.documentElement; + + + + var isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ); + }, + composed = { composed: true }; + + // Support: IE 9 - 11+, Edge 12 - 18+, iOS 10.0 - 10.2 only + // Check attachment across shadow DOM boundaries when possible (gh-3504) + // Support: iOS 10.0-10.2 only + // Early iOS 10 versions support `attachShadow` but not `getRootNode`, + // leading to errors. We need to check for `getRootNode`. + if ( documentElement.getRootNode ) { + isAttached = function( elem ) { + return jQuery.contains( elem.ownerDocument, elem ) || + elem.getRootNode( composed ) === elem.ownerDocument; + }; + } +var isHiddenWithinTree = function( elem, el ) { + + // isHiddenWithinTree might be called from jQuery#filter function; + // in that case, element will be second argument + elem = el || elem; + + // Inline style trumps all + return elem.style.display === "none" || + elem.style.display === "" && + + // Otherwise, check computed style + // Support: Firefox <=43 - 45 + // Disconnected elements can have computed display: none, so first confirm that elem is + // in the document. + isAttached( elem ) && + + jQuery.css( elem, "display" ) === "none"; + }; + + + +function adjustCSS( elem, prop, valueParts, tween ) { + var adjusted, scale, + maxIterations = 20, + currentValue = tween ? + function() { + return tween.cur(); + } : + function() { + return jQuery.css( elem, prop, "" ); + }, + initial = currentValue(), + unit = valueParts && valueParts[ 3 ] || ( jQuery.cssNumber[ prop ] ? "" : "px" ), + + // Starting value computation is required for potential unit mismatches + initialInUnit = elem.nodeType && + ( jQuery.cssNumber[ prop ] || unit !== "px" && +initial ) && + rcssNum.exec( jQuery.css( elem, prop ) ); + + if ( initialInUnit && initialInUnit[ 3 ] !== unit ) { + + // Support: Firefox <=54 + // Halve the iteration target value to prevent interference from CSS upper bounds (gh-2144) + initial = initial / 2; + + // Trust units reported by jQuery.css + unit = unit || initialInUnit[ 3 ]; + + // Iteratively approximate from a nonzero starting point + initialInUnit = +initial || 1; + + while ( maxIterations-- ) { + + // Evaluate and update our best guess (doubling guesses that zero out). + // Finish if the scale equals or crosses 1 (making the old*new product non-positive). + jQuery.style( elem, prop, initialInUnit + unit ); + if ( ( 1 - scale ) * ( 1 - ( scale = currentValue() / initial || 0.5 ) ) <= 0 ) { + maxIterations = 0; + } + initialInUnit = initialInUnit / scale; + + } + + initialInUnit = initialInUnit * 2; + jQuery.style( elem, prop, initialInUnit + unit ); + + // Make sure we update the tween properties later on + valueParts = valueParts || []; + } + + if ( valueParts ) { + initialInUnit = +initialInUnit || +initial || 0; + + // Apply relative offset (+=/-=) if specified + adjusted = valueParts[ 1 ] ? + initialInUnit + ( valueParts[ 1 ] + 1 ) * valueParts[ 2 ] : + +valueParts[ 2 ]; + if ( tween ) { + tween.unit = unit; + tween.start = initialInUnit; + tween.end = adjusted; + } + } + return adjusted; +} + + +var defaultDisplayMap = {}; + +function getDefaultDisplay( elem ) { + var temp, + doc = elem.ownerDocument, + nodeName = elem.nodeName, + display = defaultDisplayMap[ nodeName ]; + + if ( display ) { + return display; + } + + temp = doc.body.appendChild( doc.createElement( nodeName ) ); + display = jQuery.css( temp, "display" ); + + temp.parentNode.removeChild( temp ); + + if ( display === "none" ) { + display = "block"; + } + defaultDisplayMap[ nodeName ] = display; + + return display; +} + +function showHide( elements, show ) { + var display, elem, + values = [], + index = 0, + length = elements.length; + + // Determine new display value for elements that need to change + for ( ; index < length; index++ ) { + elem = elements[ index ]; + if ( !elem.style ) { + continue; + } + + display = elem.style.display; + if ( show ) { + + // Since we force visibility upon cascade-hidden elements, an immediate (and slow) + // check is required in this first loop unless we have a nonempty display value (either + // inline or about-to-be-restored) + if ( display === "none" ) { + values[ index ] = dataPriv.get( elem, "display" ) || null; + if ( !values[ index ] ) { + elem.style.display = ""; + } + } + if ( elem.style.display === "" && isHiddenWithinTree( elem ) ) { + values[ index ] = getDefaultDisplay( elem ); + } + } else { + if ( display !== "none" ) { + values[ index ] = "none"; + + // Remember what we're overwriting + dataPriv.set( elem, "display", display ); + } + } + } + + // Set the display of the elements in a second loop to avoid constant reflow + for ( index = 0; index < length; index++ ) { + if ( values[ index ] != null ) { + elements[ index ].style.display = values[ index ]; + } + } + + return elements; +} + +jQuery.fn.extend( { + show: function() { + return showHide( this, true ); + }, + hide: function() { + return showHide( this ); + }, + toggle: function( state ) { + if ( typeof state === "boolean" ) { + return state ? this.show() : this.hide(); + } + + return this.each( function() { + if ( isHiddenWithinTree( this ) ) { + jQuery( this ).show(); + } else { + jQuery( this ).hide(); + } + } ); + } +} ); +var rcheckableType = ( /^(?:checkbox|radio)$/i ); + +var rtagName = ( /<([a-z][^\/\0>\x20\t\r\n\f]*)/i ); + +var rscriptType = ( /^$|^module$|\/(?:java|ecma)script/i ); + + + +( function() { + var fragment = document.createDocumentFragment(), + div = fragment.appendChild( document.createElement( "div" ) ), + input = document.createElement( "input" ); + + // Support: Android 4.0 - 4.3 only + // Check state lost if the name is set (#11217) + // Support: Windows Web Apps (WWA) + // `name` and `type` must use .setAttribute for WWA (#14901) + input.setAttribute( "type", "radio" ); + input.setAttribute( "checked", "checked" ); + input.setAttribute( "name", "t" ); + + div.appendChild( input ); + + // Support: Android <=4.1 only + // Older WebKit doesn't clone checked state correctly in fragments + support.checkClone = div.cloneNode( true ).cloneNode( true ).lastChild.checked; + + // Support: IE <=11 only + // Make sure textarea (and checkbox) defaultValue is properly cloned + div.innerHTML = ""; + support.noCloneChecked = !!div.cloneNode( true ).lastChild.defaultValue; + + // Support: IE <=9 only + // IE <=9 replaces "; + support.option = !!div.lastChild; +} )(); + + +// We have to close these tags to support XHTML (#13200) +var wrapMap = { + + // XHTML parsers do not magically insert elements in the + // same way that tag soup parsers do. So we cannot shorten + // this by omitting or other required elements. + thead: [ 1, "", "
" ], + col: [ 2, "", "
" ], + tr: [ 2, "", "
" ], + td: [ 3, "", "
" ], + + _default: [ 0, "", "" ] +}; + +wrapMap.tbody = wrapMap.tfoot = wrapMap.colgroup = wrapMap.caption = wrapMap.thead; +wrapMap.th = wrapMap.td; + +// Support: IE <=9 only +if ( !support.option ) { + wrapMap.optgroup = wrapMap.option = [ 1, "" ]; +} + + +function getAll( context, tag ) { + + // Support: IE <=9 - 11 only + // Use typeof to avoid zero-argument method invocation on host objects (#15151) + var ret; + + if ( typeof context.getElementsByTagName !== "undefined" ) { + ret = context.getElementsByTagName( tag || "*" ); + + } else if ( typeof context.querySelectorAll !== "undefined" ) { + ret = context.querySelectorAll( tag || "*" ); + + } else { + ret = []; + } + + if ( tag === undefined || tag && nodeName( context, tag ) ) { + return jQuery.merge( [ context ], ret ); + } + + return ret; +} + + +// Mark scripts as having already been evaluated +function setGlobalEval( elems, refElements ) { + var i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + dataPriv.set( + elems[ i ], + "globalEval", + !refElements || dataPriv.get( refElements[ i ], "globalEval" ) + ); + } +} + + +var rhtml = /<|&#?\w+;/; + +function buildFragment( elems, context, scripts, selection, ignored ) { + var elem, tmp, tag, wrap, attached, j, + fragment = context.createDocumentFragment(), + nodes = [], + i = 0, + l = elems.length; + + for ( ; i < l; i++ ) { + elem = elems[ i ]; + + if ( elem || elem === 0 ) { + + // Add nodes directly + if ( toType( elem ) === "object" ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, elem.nodeType ? [ elem ] : elem ); + + // Convert non-html into a text node + } else if ( !rhtml.test( elem ) ) { + nodes.push( context.createTextNode( elem ) ); + + // Convert html into DOM nodes + } else { + tmp = tmp || fragment.appendChild( context.createElement( "div" ) ); + + // Deserialize a standard representation + tag = ( rtagName.exec( elem ) || [ "", "" ] )[ 1 ].toLowerCase(); + wrap = wrapMap[ tag ] || wrapMap._default; + tmp.innerHTML = wrap[ 1 ] + jQuery.htmlPrefilter( elem ) + wrap[ 2 ]; + + // Descend through wrappers to the right content + j = wrap[ 0 ]; + while ( j-- ) { + tmp = tmp.lastChild; + } + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( nodes, tmp.childNodes ); + + // Remember the top-level container + tmp = fragment.firstChild; + + // Ensure the created nodes are orphaned (#12392) + tmp.textContent = ""; + } + } + } + + // Remove wrapper from fragment + fragment.textContent = ""; + + i = 0; + while ( ( elem = nodes[ i++ ] ) ) { + + // Skip elements already in the context collection (trac-4087) + if ( selection && jQuery.inArray( elem, selection ) > -1 ) { + if ( ignored ) { + ignored.push( elem ); + } + continue; + } + + attached = isAttached( elem ); + + // Append to fragment + tmp = getAll( fragment.appendChild( elem ), "script" ); + + // Preserve script evaluation history + if ( attached ) { + setGlobalEval( tmp ); + } + + // Capture executables + if ( scripts ) { + j = 0; + while ( ( elem = tmp[ j++ ] ) ) { + if ( rscriptType.test( elem.type || "" ) ) { + scripts.push( elem ); + } + } + } + } + + return fragment; +} + + +var rtypenamespace = /^([^.]*)(?:\.(.+)|)/; + +function returnTrue() { + return true; +} + +function returnFalse() { + return false; +} + +// Support: IE <=9 - 11+ +// focus() and blur() are asynchronous, except when they are no-op. +// So expect focus to be synchronous when the element is already active, +// and blur to be synchronous when the element is not already active. +// (focus and blur are always synchronous in other supported browsers, +// this just defines when we can count on it). +function expectSync( elem, type ) { + return ( elem === safeActiveElement() ) === ( type === "focus" ); +} + +// Support: IE <=9 only +// Accessing document.activeElement can throw unexpectedly +// https://bugs.jquery.com/ticket/13393 +function safeActiveElement() { + try { + return document.activeElement; + } catch ( err ) { } +} + +function on( elem, types, selector, data, fn, one ) { + var origFn, type; + + // Types can be a map of types/handlers + if ( typeof types === "object" ) { + + // ( types-Object, selector, data ) + if ( typeof selector !== "string" ) { + + // ( types-Object, data ) + data = data || selector; + selector = undefined; + } + for ( type in types ) { + on( elem, type, selector, data, types[ type ], one ); + } + return elem; + } + + if ( data == null && fn == null ) { + + // ( types, fn ) + fn = selector; + data = selector = undefined; + } else if ( fn == null ) { + if ( typeof selector === "string" ) { + + // ( types, selector, fn ) + fn = data; + data = undefined; + } else { + + // ( types, data, fn ) + fn = data; + data = selector; + selector = undefined; + } + } + if ( fn === false ) { + fn = returnFalse; + } else if ( !fn ) { + return elem; + } + + if ( one === 1 ) { + origFn = fn; + fn = function( event ) { + + // Can use an empty set, since event contains the info + jQuery().off( event ); + return origFn.apply( this, arguments ); + }; + + // Use same guid so caller can remove using origFn + fn.guid = origFn.guid || ( origFn.guid = jQuery.guid++ ); + } + return elem.each( function() { + jQuery.event.add( this, types, fn, data, selector ); + } ); +} + +/* + * Helper functions for managing events -- not part of the public interface. + * Props to Dean Edwards' addEvent library for many of the ideas. + */ +jQuery.event = { + + global: {}, + + add: function( elem, types, handler, data, selector ) { + + var handleObjIn, eventHandle, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.get( elem ); + + // Only attach events to objects that accept data + if ( !acceptData( elem ) ) { + return; + } + + // Caller can pass in an object of custom data in lieu of the handler + if ( handler.handler ) { + handleObjIn = handler; + handler = handleObjIn.handler; + selector = handleObjIn.selector; + } + + // Ensure that invalid selectors throw exceptions at attach time + // Evaluate against documentElement in case elem is a non-element node (e.g., document) + if ( selector ) { + jQuery.find.matchesSelector( documentElement, selector ); + } + + // Make sure that the handler has a unique ID, used to find/remove it later + if ( !handler.guid ) { + handler.guid = jQuery.guid++; + } + + // Init the element's event structure and main handler, if this is the first + if ( !( events = elemData.events ) ) { + events = elemData.events = Object.create( null ); + } + if ( !( eventHandle = elemData.handle ) ) { + eventHandle = elemData.handle = function( e ) { + + // Discard the second event of a jQuery.event.trigger() and + // when an event is called after a page has unloaded + return typeof jQuery !== "undefined" && jQuery.event.triggered !== e.type ? + jQuery.event.dispatch.apply( elem, arguments ) : undefined; + }; + } + + // Handle multiple events separated by a space + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // There *must* be a type, no attaching namespace-only handlers + if ( !type ) { + continue; + } + + // If event changes its type, use the special event handlers for the changed type + special = jQuery.event.special[ type ] || {}; + + // If selector defined, determine special event api type, otherwise given type + type = ( selector ? special.delegateType : special.bindType ) || type; + + // Update special based on newly reset type + special = jQuery.event.special[ type ] || {}; + + // handleObj is passed to all event handlers + handleObj = jQuery.extend( { + type: type, + origType: origType, + data: data, + handler: handler, + guid: handler.guid, + selector: selector, + needsContext: selector && jQuery.expr.match.needsContext.test( selector ), + namespace: namespaces.join( "." ) + }, handleObjIn ); + + // Init the event handler queue if we're the first + if ( !( handlers = events[ type ] ) ) { + handlers = events[ type ] = []; + handlers.delegateCount = 0; + + // Only use addEventListener if the special events handler returns false + if ( !special.setup || + special.setup.call( elem, data, namespaces, eventHandle ) === false ) { + + if ( elem.addEventListener ) { + elem.addEventListener( type, eventHandle ); + } + } + } + + if ( special.add ) { + special.add.call( elem, handleObj ); + + if ( !handleObj.handler.guid ) { + handleObj.handler.guid = handler.guid; + } + } + + // Add to the element's handler list, delegates in front + if ( selector ) { + handlers.splice( handlers.delegateCount++, 0, handleObj ); + } else { + handlers.push( handleObj ); + } + + // Keep track of which events have ever been used, for event optimization + jQuery.event.global[ type ] = true; + } + + }, + + // Detach an event or set of events from an element + remove: function( elem, types, handler, selector, mappedTypes ) { + + var j, origCount, tmp, + events, t, handleObj, + special, handlers, type, namespaces, origType, + elemData = dataPriv.hasData( elem ) && dataPriv.get( elem ); + + if ( !elemData || !( events = elemData.events ) ) { + return; + } + + // Once for each type.namespace in types; type may be omitted + types = ( types || "" ).match( rnothtmlwhite ) || [ "" ]; + t = types.length; + while ( t-- ) { + tmp = rtypenamespace.exec( types[ t ] ) || []; + type = origType = tmp[ 1 ]; + namespaces = ( tmp[ 2 ] || "" ).split( "." ).sort(); + + // Unbind all events (on this namespace, if provided) for the element + if ( !type ) { + for ( type in events ) { + jQuery.event.remove( elem, type + types[ t ], handler, selector, true ); + } + continue; + } + + special = jQuery.event.special[ type ] || {}; + type = ( selector ? special.delegateType : special.bindType ) || type; + handlers = events[ type ] || []; + tmp = tmp[ 2 ] && + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ); + + // Remove matching events + origCount = j = handlers.length; + while ( j-- ) { + handleObj = handlers[ j ]; + + if ( ( mappedTypes || origType === handleObj.origType ) && + ( !handler || handler.guid === handleObj.guid ) && + ( !tmp || tmp.test( handleObj.namespace ) ) && + ( !selector || selector === handleObj.selector || + selector === "**" && handleObj.selector ) ) { + handlers.splice( j, 1 ); + + if ( handleObj.selector ) { + handlers.delegateCount--; + } + if ( special.remove ) { + special.remove.call( elem, handleObj ); + } + } + } + + // Remove generic event handler if we removed something and no more handlers exist + // (avoids potential for endless recursion during removal of special event handlers) + if ( origCount && !handlers.length ) { + if ( !special.teardown || + special.teardown.call( elem, namespaces, elemData.handle ) === false ) { + + jQuery.removeEvent( elem, type, elemData.handle ); + } + + delete events[ type ]; + } + } + + // Remove data and the expando if it's no longer used + if ( jQuery.isEmptyObject( events ) ) { + dataPriv.remove( elem, "handle events" ); + } + }, + + dispatch: function( nativeEvent ) { + + var i, j, ret, matched, handleObj, handlerQueue, + args = new Array( arguments.length ), + + // Make a writable jQuery.Event from the native event object + event = jQuery.event.fix( nativeEvent ), + + handlers = ( + dataPriv.get( this, "events" ) || Object.create( null ) + )[ event.type ] || [], + special = jQuery.event.special[ event.type ] || {}; + + // Use the fix-ed jQuery.Event rather than the (read-only) native event + args[ 0 ] = event; + + for ( i = 1; i < arguments.length; i++ ) { + args[ i ] = arguments[ i ]; + } + + event.delegateTarget = this; + + // Call the preDispatch hook for the mapped type, and let it bail if desired + if ( special.preDispatch && special.preDispatch.call( this, event ) === false ) { + return; + } + + // Determine handlers + handlerQueue = jQuery.event.handlers.call( this, event, handlers ); + + // Run delegates first; they may want to stop propagation beneath us + i = 0; + while ( ( matched = handlerQueue[ i++ ] ) && !event.isPropagationStopped() ) { + event.currentTarget = matched.elem; + + j = 0; + while ( ( handleObj = matched.handlers[ j++ ] ) && + !event.isImmediatePropagationStopped() ) { + + // If the event is namespaced, then each handler is only invoked if it is + // specially universal or its namespaces are a superset of the event's. + if ( !event.rnamespace || handleObj.namespace === false || + event.rnamespace.test( handleObj.namespace ) ) { + + event.handleObj = handleObj; + event.data = handleObj.data; + + ret = ( ( jQuery.event.special[ handleObj.origType ] || {} ).handle || + handleObj.handler ).apply( matched.elem, args ); + + if ( ret !== undefined ) { + if ( ( event.result = ret ) === false ) { + event.preventDefault(); + event.stopPropagation(); + } + } + } + } + } + + // Call the postDispatch hook for the mapped type + if ( special.postDispatch ) { + special.postDispatch.call( this, event ); + } + + return event.result; + }, + + handlers: function( event, handlers ) { + var i, handleObj, sel, matchedHandlers, matchedSelectors, + handlerQueue = [], + delegateCount = handlers.delegateCount, + cur = event.target; + + // Find delegate handlers + if ( delegateCount && + + // Support: IE <=9 + // Black-hole SVG instance trees (trac-13180) + cur.nodeType && + + // Support: Firefox <=42 + // Suppress spec-violating clicks indicating a non-primary pointer button (trac-3861) + // https://www.w3.org/TR/DOM-Level-3-Events/#event-type-click + // Support: IE 11 only + // ...but not arrow key "clicks" of radio inputs, which can have `button` -1 (gh-2343) + !( event.type === "click" && event.button >= 1 ) ) { + + for ( ; cur !== this; cur = cur.parentNode || this ) { + + // Don't check non-elements (#13208) + // Don't process clicks on disabled elements (#6911, #8165, #11382, #11764) + if ( cur.nodeType === 1 && !( event.type === "click" && cur.disabled === true ) ) { + matchedHandlers = []; + matchedSelectors = {}; + for ( i = 0; i < delegateCount; i++ ) { + handleObj = handlers[ i ]; + + // Don't conflict with Object.prototype properties (#13203) + sel = handleObj.selector + " "; + + if ( matchedSelectors[ sel ] === undefined ) { + matchedSelectors[ sel ] = handleObj.needsContext ? + jQuery( sel, this ).index( cur ) > -1 : + jQuery.find( sel, this, null, [ cur ] ).length; + } + if ( matchedSelectors[ sel ] ) { + matchedHandlers.push( handleObj ); + } + } + if ( matchedHandlers.length ) { + handlerQueue.push( { elem: cur, handlers: matchedHandlers } ); + } + } + } + } + + // Add the remaining (directly-bound) handlers + cur = this; + if ( delegateCount < handlers.length ) { + handlerQueue.push( { elem: cur, handlers: handlers.slice( delegateCount ) } ); + } + + return handlerQueue; + }, + + addProp: function( name, hook ) { + Object.defineProperty( jQuery.Event.prototype, name, { + enumerable: true, + configurable: true, + + get: isFunction( hook ) ? + function() { + if ( this.originalEvent ) { + return hook( this.originalEvent ); + } + } : + function() { + if ( this.originalEvent ) { + return this.originalEvent[ name ]; + } + }, + + set: function( value ) { + Object.defineProperty( this, name, { + enumerable: true, + configurable: true, + writable: true, + value: value + } ); + } + } ); + }, + + fix: function( originalEvent ) { + return originalEvent[ jQuery.expando ] ? + originalEvent : + new jQuery.Event( originalEvent ); + }, + + special: { + load: { + + // Prevent triggered image.load events from bubbling to window.load + noBubble: true + }, + click: { + + // Utilize native event to ensure correct state for checkable inputs + setup: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Claim the first handler + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + // dataPriv.set( el, "click", ... ) + leverageNative( el, "click", returnTrue ); + } + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function( data ) { + + // For mutual compressibility with _default, replace `this` access with a local var. + // `|| data` is dead code meant only to preserve the variable through minification. + var el = this || data; + + // Force setup before triggering a click + if ( rcheckableType.test( el.type ) && + el.click && nodeName( el, "input" ) ) { + + leverageNative( el, "click" ); + } + + // Return non-false to allow normal event-path propagation + return true; + }, + + // For cross-browser consistency, suppress native .click() on links + // Also prevent it if we're currently inside a leveraged native-event stack + _default: function( event ) { + var target = event.target; + return rcheckableType.test( target.type ) && + target.click && nodeName( target, "input" ) && + dataPriv.get( target, "click" ) || + nodeName( target, "a" ); + } + }, + + beforeunload: { + postDispatch: function( event ) { + + // Support: Firefox 20+ + // Firefox doesn't alert if the returnValue field is not set. + if ( event.result !== undefined && event.originalEvent ) { + event.originalEvent.returnValue = event.result; + } + } + } + } +}; + +// Ensure the presence of an event listener that handles manually-triggered +// synthetic events by interrupting progress until reinvoked in response to +// *native* events that it fires directly, ensuring that state changes have +// already occurred before other listeners are invoked. +function leverageNative( el, type, expectSync ) { + + // Missing expectSync indicates a trigger call, which must force setup through jQuery.event.add + if ( !expectSync ) { + if ( dataPriv.get( el, type ) === undefined ) { + jQuery.event.add( el, type, returnTrue ); + } + return; + } + + // Register the controller as a special universal handler for all event namespaces + dataPriv.set( el, type, false ); + jQuery.event.add( el, type, { + namespace: false, + handler: function( event ) { + var notAsync, result, + saved = dataPriv.get( this, type ); + + if ( ( event.isTrigger & 1 ) && this[ type ] ) { + + // Interrupt processing of the outer synthetic .trigger()ed event + // Saved data should be false in such cases, but might be a leftover capture object + // from an async native handler (gh-4350) + if ( !saved.length ) { + + // Store arguments for use when handling the inner native event + // There will always be at least one argument (an event object), so this array + // will not be confused with a leftover capture object. + saved = slice.call( arguments ); + dataPriv.set( this, type, saved ); + + // Trigger the native event and capture its result + // Support: IE <=9 - 11+ + // focus() and blur() are asynchronous + notAsync = expectSync( this, type ); + this[ type ](); + result = dataPriv.get( this, type ); + if ( saved !== result || notAsync ) { + dataPriv.set( this, type, false ); + } else { + result = {}; + } + if ( saved !== result ) { + + // Cancel the outer synthetic event + event.stopImmediatePropagation(); + event.preventDefault(); + + // Support: Chrome 86+ + // In Chrome, if an element having a focusout handler is blurred by + // clicking outside of it, it invokes the handler synchronously. If + // that handler calls `.remove()` on the element, the data is cleared, + // leaving `result` undefined. We need to guard against this. + return result && result.value; + } + + // If this is an inner synthetic event for an event with a bubbling surrogate + // (focus or blur), assume that the surrogate already propagated from triggering the + // native event and prevent that from happening again here. + // This technically gets the ordering wrong w.r.t. to `.trigger()` (in which the + // bubbling surrogate propagates *after* the non-bubbling base), but that seems + // less bad than duplication. + } else if ( ( jQuery.event.special[ type ] || {} ).delegateType ) { + event.stopPropagation(); + } + + // If this is a native event triggered above, everything is now in order + // Fire an inner synthetic event with the original arguments + } else if ( saved.length ) { + + // ...and capture the result + dataPriv.set( this, type, { + value: jQuery.event.trigger( + + // Support: IE <=9 - 11+ + // Extend with the prototype to reset the above stopImmediatePropagation() + jQuery.extend( saved[ 0 ], jQuery.Event.prototype ), + saved.slice( 1 ), + this + ) + } ); + + // Abort handling of the native event + event.stopImmediatePropagation(); + } + } + } ); +} + +jQuery.removeEvent = function( elem, type, handle ) { + + // This "if" is needed for plain objects + if ( elem.removeEventListener ) { + elem.removeEventListener( type, handle ); + } +}; + +jQuery.Event = function( src, props ) { + + // Allow instantiation without the 'new' keyword + if ( !( this instanceof jQuery.Event ) ) { + return new jQuery.Event( src, props ); + } + + // Event object + if ( src && src.type ) { + this.originalEvent = src; + this.type = src.type; + + // Events bubbling up the document may have been marked as prevented + // by a handler lower down the tree; reflect the correct value. + this.isDefaultPrevented = src.defaultPrevented || + src.defaultPrevented === undefined && + + // Support: Android <=2.3 only + src.returnValue === false ? + returnTrue : + returnFalse; + + // Create target properties + // Support: Safari <=6 - 7 only + // Target should not be a text node (#504, #13143) + this.target = ( src.target && src.target.nodeType === 3 ) ? + src.target.parentNode : + src.target; + + this.currentTarget = src.currentTarget; + this.relatedTarget = src.relatedTarget; + + // Event type + } else { + this.type = src; + } + + // Put explicitly provided properties onto the event object + if ( props ) { + jQuery.extend( this, props ); + } + + // Create a timestamp if incoming event doesn't have one + this.timeStamp = src && src.timeStamp || Date.now(); + + // Mark it as fixed + this[ jQuery.expando ] = true; +}; + +// jQuery.Event is based on DOM3 Events as specified by the ECMAScript Language Binding +// https://www.w3.org/TR/2003/WD-DOM-Level-3-Events-20030331/ecma-script-binding.html +jQuery.Event.prototype = { + constructor: jQuery.Event, + isDefaultPrevented: returnFalse, + isPropagationStopped: returnFalse, + isImmediatePropagationStopped: returnFalse, + isSimulated: false, + + preventDefault: function() { + var e = this.originalEvent; + + this.isDefaultPrevented = returnTrue; + + if ( e && !this.isSimulated ) { + e.preventDefault(); + } + }, + stopPropagation: function() { + var e = this.originalEvent; + + this.isPropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopPropagation(); + } + }, + stopImmediatePropagation: function() { + var e = this.originalEvent; + + this.isImmediatePropagationStopped = returnTrue; + + if ( e && !this.isSimulated ) { + e.stopImmediatePropagation(); + } + + this.stopPropagation(); + } +}; + +// Includes all common event props including KeyEvent and MouseEvent specific props +jQuery.each( { + altKey: true, + bubbles: true, + cancelable: true, + changedTouches: true, + ctrlKey: true, + detail: true, + eventPhase: true, + metaKey: true, + pageX: true, + pageY: true, + shiftKey: true, + view: true, + "char": true, + code: true, + charCode: true, + key: true, + keyCode: true, + button: true, + buttons: true, + clientX: true, + clientY: true, + offsetX: true, + offsetY: true, + pointerId: true, + pointerType: true, + screenX: true, + screenY: true, + targetTouches: true, + toElement: true, + touches: true, + which: true +}, jQuery.event.addProp ); + +jQuery.each( { focus: "focusin", blur: "focusout" }, function( type, delegateType ) { + jQuery.event.special[ type ] = { + + // Utilize native event if possible so blur/focus sequence is correct + setup: function() { + + // Claim the first handler + // dataPriv.set( this, "focus", ... ) + // dataPriv.set( this, "blur", ... ) + leverageNative( this, type, expectSync ); + + // Return false to allow normal processing in the caller + return false; + }, + trigger: function() { + + // Force setup before trigger + leverageNative( this, type ); + + // Return non-false to allow normal event-path propagation + return true; + }, + + // Suppress native focus or blur as it's already being fired + // in leverageNative. + _default: function() { + return true; + }, + + delegateType: delegateType + }; +} ); + +// Create mouseenter/leave events using mouseover/out and event-time checks +// so that event delegation works in jQuery. +// Do the same for pointerenter/pointerleave and pointerover/pointerout +// +// Support: Safari 7 only +// Safari sends mouseenter too often; see: +// https://bugs.chromium.org/p/chromium/issues/detail?id=470258 +// for the description of the bug (it existed in older Chrome versions as well). +jQuery.each( { + mouseenter: "mouseover", + mouseleave: "mouseout", + pointerenter: "pointerover", + pointerleave: "pointerout" +}, function( orig, fix ) { + jQuery.event.special[ orig ] = { + delegateType: fix, + bindType: fix, + + handle: function( event ) { + var ret, + target = this, + related = event.relatedTarget, + handleObj = event.handleObj; + + // For mouseenter/leave call the handler if related is outside the target. + // NB: No relatedTarget if the mouse left/entered the browser window + if ( !related || ( related !== target && !jQuery.contains( target, related ) ) ) { + event.type = handleObj.origType; + ret = handleObj.handler.apply( this, arguments ); + event.type = fix; + } + return ret; + } + }; +} ); + +jQuery.fn.extend( { + + on: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn ); + }, + one: function( types, selector, data, fn ) { + return on( this, types, selector, data, fn, 1 ); + }, + off: function( types, selector, fn ) { + var handleObj, type; + if ( types && types.preventDefault && types.handleObj ) { + + // ( event ) dispatched jQuery.Event + handleObj = types.handleObj; + jQuery( types.delegateTarget ).off( + handleObj.namespace ? + handleObj.origType + "." + handleObj.namespace : + handleObj.origType, + handleObj.selector, + handleObj.handler + ); + return this; + } + if ( typeof types === "object" ) { + + // ( types-object [, selector] ) + for ( type in types ) { + this.off( type, selector, types[ type ] ); + } + return this; + } + if ( selector === false || typeof selector === "function" ) { + + // ( types [, fn] ) + fn = selector; + selector = undefined; + } + if ( fn === false ) { + fn = returnFalse; + } + return this.each( function() { + jQuery.event.remove( this, types, fn, selector ); + } ); + } +} ); + + +var + + // Support: IE <=10 - 11, Edge 12 - 13 only + // In IE/Edge using regex groups here causes severe slowdowns. + // See https://connect.microsoft.com/IE/feedback/details/1736512/ + rnoInnerhtml = /\s*$/g; + +// Prefer a tbody over its parent table for containing new rows +function manipulationTarget( elem, content ) { + if ( nodeName( elem, "table" ) && + nodeName( content.nodeType !== 11 ? content : content.firstChild, "tr" ) ) { + + return jQuery( elem ).children( "tbody" )[ 0 ] || elem; + } + + return elem; +} + +// Replace/restore the type attribute of script elements for safe DOM manipulation +function disableScript( elem ) { + elem.type = ( elem.getAttribute( "type" ) !== null ) + "/" + elem.type; + return elem; +} +function restoreScript( elem ) { + if ( ( elem.type || "" ).slice( 0, 5 ) === "true/" ) { + elem.type = elem.type.slice( 5 ); + } else { + elem.removeAttribute( "type" ); + } + + return elem; +} + +function cloneCopyEvent( src, dest ) { + var i, l, type, pdataOld, udataOld, udataCur, events; + + if ( dest.nodeType !== 1 ) { + return; + } + + // 1. Copy private data: events, handlers, etc. + if ( dataPriv.hasData( src ) ) { + pdataOld = dataPriv.get( src ); + events = pdataOld.events; + + if ( events ) { + dataPriv.remove( dest, "handle events" ); + + for ( type in events ) { + for ( i = 0, l = events[ type ].length; i < l; i++ ) { + jQuery.event.add( dest, type, events[ type ][ i ] ); + } + } + } + } + + // 2. Copy user data + if ( dataUser.hasData( src ) ) { + udataOld = dataUser.access( src ); + udataCur = jQuery.extend( {}, udataOld ); + + dataUser.set( dest, udataCur ); + } +} + +// Fix IE bugs, see support tests +function fixInput( src, dest ) { + var nodeName = dest.nodeName.toLowerCase(); + + // Fails to persist the checked state of a cloned checkbox or radio button. + if ( nodeName === "input" && rcheckableType.test( src.type ) ) { + dest.checked = src.checked; + + // Fails to return the selected option to the default selected state when cloning options + } else if ( nodeName === "input" || nodeName === "textarea" ) { + dest.defaultValue = src.defaultValue; + } +} + +function domManip( collection, args, callback, ignored ) { + + // Flatten any nested arrays + args = flat( args ); + + var fragment, first, scripts, hasScripts, node, doc, + i = 0, + l = collection.length, + iNoClone = l - 1, + value = args[ 0 ], + valueIsFunction = isFunction( value ); + + // We can't cloneNode fragments that contain checked, in WebKit + if ( valueIsFunction || + ( l > 1 && typeof value === "string" && + !support.checkClone && rchecked.test( value ) ) ) { + return collection.each( function( index ) { + var self = collection.eq( index ); + if ( valueIsFunction ) { + args[ 0 ] = value.call( this, index, self.html() ); + } + domManip( self, args, callback, ignored ); + } ); + } + + if ( l ) { + fragment = buildFragment( args, collection[ 0 ].ownerDocument, false, collection, ignored ); + first = fragment.firstChild; + + if ( fragment.childNodes.length === 1 ) { + fragment = first; + } + + // Require either new content or an interest in ignored elements to invoke the callback + if ( first || ignored ) { + scripts = jQuery.map( getAll( fragment, "script" ), disableScript ); + hasScripts = scripts.length; + + // Use the original fragment for the last item + // instead of the first because it can end up + // being emptied incorrectly in certain situations (#8070). + for ( ; i < l; i++ ) { + node = fragment; + + if ( i !== iNoClone ) { + node = jQuery.clone( node, true, true ); + + // Keep references to cloned scripts for later restoration + if ( hasScripts ) { + + // Support: Android <=4.0 only, PhantomJS 1 only + // push.apply(_, arraylike) throws on ancient WebKit + jQuery.merge( scripts, getAll( node, "script" ) ); + } + } + + callback.call( collection[ i ], node, i ); + } + + if ( hasScripts ) { + doc = scripts[ scripts.length - 1 ].ownerDocument; + + // Reenable scripts + jQuery.map( scripts, restoreScript ); + + // Evaluate executable scripts on first document insertion + for ( i = 0; i < hasScripts; i++ ) { + node = scripts[ i ]; + if ( rscriptType.test( node.type || "" ) && + !dataPriv.access( node, "globalEval" ) && + jQuery.contains( doc, node ) ) { + + if ( node.src && ( node.type || "" ).toLowerCase() !== "module" ) { + + // Optional AJAX dependency, but won't run scripts if not present + if ( jQuery._evalUrl && !node.noModule ) { + jQuery._evalUrl( node.src, { + nonce: node.nonce || node.getAttribute( "nonce" ) + }, doc ); + } + } else { + DOMEval( node.textContent.replace( rcleanScript, "" ), node, doc ); + } + } + } + } + } + } + + return collection; +} + +function remove( elem, selector, keepData ) { + var node, + nodes = selector ? jQuery.filter( selector, elem ) : elem, + i = 0; + + for ( ; ( node = nodes[ i ] ) != null; i++ ) { + if ( !keepData && node.nodeType === 1 ) { + jQuery.cleanData( getAll( node ) ); + } + + if ( node.parentNode ) { + if ( keepData && isAttached( node ) ) { + setGlobalEval( getAll( node, "script" ) ); + } + node.parentNode.removeChild( node ); + } + } + + return elem; +} + +jQuery.extend( { + htmlPrefilter: function( html ) { + return html; + }, + + clone: function( elem, dataAndEvents, deepDataAndEvents ) { + var i, l, srcElements, destElements, + clone = elem.cloneNode( true ), + inPage = isAttached( elem ); + + // Fix IE cloning issues + if ( !support.noCloneChecked && ( elem.nodeType === 1 || elem.nodeType === 11 ) && + !jQuery.isXMLDoc( elem ) ) { + + // We eschew Sizzle here for performance reasons: https://jsperf.com/getall-vs-sizzle/2 + destElements = getAll( clone ); + srcElements = getAll( elem ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + fixInput( srcElements[ i ], destElements[ i ] ); + } + } + + // Copy the events from the original to the clone + if ( dataAndEvents ) { + if ( deepDataAndEvents ) { + srcElements = srcElements || getAll( elem ); + destElements = destElements || getAll( clone ); + + for ( i = 0, l = srcElements.length; i < l; i++ ) { + cloneCopyEvent( srcElements[ i ], destElements[ i ] ); + } + } else { + cloneCopyEvent( elem, clone ); + } + } + + // Preserve script evaluation history + destElements = getAll( clone, "script" ); + if ( destElements.length > 0 ) { + setGlobalEval( destElements, !inPage && getAll( elem, "script" ) ); + } + + // Return the cloned set + return clone; + }, + + cleanData: function( elems ) { + var data, elem, type, + special = jQuery.event.special, + i = 0; + + for ( ; ( elem = elems[ i ] ) !== undefined; i++ ) { + if ( acceptData( elem ) ) { + if ( ( data = elem[ dataPriv.expando ] ) ) { + if ( data.events ) { + for ( type in data.events ) { + if ( special[ type ] ) { + jQuery.event.remove( elem, type ); + + // This is a shortcut to avoid jQuery.event.remove's overhead + } else { + jQuery.removeEvent( elem, type, data.handle ); + } + } + } + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataPriv.expando ] = undefined; + } + if ( elem[ dataUser.expando ] ) { + + // Support: Chrome <=35 - 45+ + // Assign undefined instead of using delete, see Data#remove + elem[ dataUser.expando ] = undefined; + } + } + } + } +} ); + +jQuery.fn.extend( { + detach: function( selector ) { + return remove( this, selector, true ); + }, + + remove: function( selector ) { + return remove( this, selector ); + }, + + text: function( value ) { + return access( this, function( value ) { + return value === undefined ? + jQuery.text( this ) : + this.empty().each( function() { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + this.textContent = value; + } + } ); + }, null, value, arguments.length ); + }, + + append: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.appendChild( elem ); + } + } ); + }, + + prepend: function() { + return domManip( this, arguments, function( elem ) { + if ( this.nodeType === 1 || this.nodeType === 11 || this.nodeType === 9 ) { + var target = manipulationTarget( this, elem ); + target.insertBefore( elem, target.firstChild ); + } + } ); + }, + + before: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this ); + } + } ); + }, + + after: function() { + return domManip( this, arguments, function( elem ) { + if ( this.parentNode ) { + this.parentNode.insertBefore( elem, this.nextSibling ); + } + } ); + }, + + empty: function() { + var elem, + i = 0; + + for ( ; ( elem = this[ i ] ) != null; i++ ) { + if ( elem.nodeType === 1 ) { + + // Prevent memory leaks + jQuery.cleanData( getAll( elem, false ) ); + + // Remove any remaining nodes + elem.textContent = ""; + } + } + + return this; + }, + + clone: function( dataAndEvents, deepDataAndEvents ) { + dataAndEvents = dataAndEvents == null ? false : dataAndEvents; + deepDataAndEvents = deepDataAndEvents == null ? dataAndEvents : deepDataAndEvents; + + return this.map( function() { + return jQuery.clone( this, dataAndEvents, deepDataAndEvents ); + } ); + }, + + html: function( value ) { + return access( this, function( value ) { + var elem = this[ 0 ] || {}, + i = 0, + l = this.length; + + if ( value === undefined && elem.nodeType === 1 ) { + return elem.innerHTML; + } + + // See if we can take a shortcut and just use innerHTML + if ( typeof value === "string" && !rnoInnerhtml.test( value ) && + !wrapMap[ ( rtagName.exec( value ) || [ "", "" ] )[ 1 ].toLowerCase() ] ) { + + value = jQuery.htmlPrefilter( value ); + + try { + for ( ; i < l; i++ ) { + elem = this[ i ] || {}; + + // Remove element nodes and prevent memory leaks + if ( elem.nodeType === 1 ) { + jQuery.cleanData( getAll( elem, false ) ); + elem.innerHTML = value; + } + } + + elem = 0; + + // If using innerHTML throws an exception, use the fallback method + } catch ( e ) {} + } + + if ( elem ) { + this.empty().append( value ); + } + }, null, value, arguments.length ); + }, + + replaceWith: function() { + var ignored = []; + + // Make the changes, replacing each non-ignored context element with the new content + return domManip( this, arguments, function( elem ) { + var parent = this.parentNode; + + if ( jQuery.inArray( this, ignored ) < 0 ) { + jQuery.cleanData( getAll( this ) ); + if ( parent ) { + parent.replaceChild( elem, this ); + } + } + + // Force callback invocation + }, ignored ); + } +} ); + +jQuery.each( { + appendTo: "append", + prependTo: "prepend", + insertBefore: "before", + insertAfter: "after", + replaceAll: "replaceWith" +}, function( name, original ) { + jQuery.fn[ name ] = function( selector ) { + var elems, + ret = [], + insert = jQuery( selector ), + last = insert.length - 1, + i = 0; + + for ( ; i <= last; i++ ) { + elems = i === last ? this : this.clone( true ); + jQuery( insert[ i ] )[ original ]( elems ); + + // Support: Android <=4.0 only, PhantomJS 1 only + // .get() because push.apply(_, arraylike) throws on ancient WebKit + push.apply( ret, elems.get() ); + } + + return this.pushStack( ret ); + }; +} ); +var rnumnonpx = new RegExp( "^(" + pnum + ")(?!px)[a-z%]+$", "i" ); + +var getStyles = function( elem ) { + + // Support: IE <=11 only, Firefox <=30 (#15098, #14150) + // IE throws on elements created in popups + // FF meanwhile throws on frame elements through "defaultView.getComputedStyle" + var view = elem.ownerDocument.defaultView; + + if ( !view || !view.opener ) { + view = window; + } + + return view.getComputedStyle( elem ); + }; + +var swap = function( elem, options, callback ) { + var ret, name, + old = {}; + + // Remember the old values, and insert the new ones + for ( name in options ) { + old[ name ] = elem.style[ name ]; + elem.style[ name ] = options[ name ]; + } + + ret = callback.call( elem ); + + // Revert the old values + for ( name in options ) { + elem.style[ name ] = old[ name ]; + } + + return ret; +}; + + +var rboxStyle = new RegExp( cssExpand.join( "|" ), "i" ); + + + +( function() { + + // Executing both pixelPosition & boxSizingReliable tests require only one layout + // so they're executed at the same time to save the second computation. + function computeStyleTests() { + + // This is a singleton, we need to execute it only once + if ( !div ) { + return; + } + + container.style.cssText = "position:absolute;left:-11111px;width:60px;" + + "margin-top:1px;padding:0;border:0"; + div.style.cssText = + "position:relative;display:block;box-sizing:border-box;overflow:scroll;" + + "margin:auto;border:1px;padding:1px;" + + "width:60%;top:1%"; + documentElement.appendChild( container ).appendChild( div ); + + var divStyle = window.getComputedStyle( div ); + pixelPositionVal = divStyle.top !== "1%"; + + // Support: Android 4.0 - 4.3 only, Firefox <=3 - 44 + reliableMarginLeftVal = roundPixelMeasures( divStyle.marginLeft ) === 12; + + // Support: Android 4.0 - 4.3 only, Safari <=9.1 - 10.1, iOS <=7.0 - 9.3 + // Some styles come back with percentage values, even though they shouldn't + div.style.right = "60%"; + pixelBoxStylesVal = roundPixelMeasures( divStyle.right ) === 36; + + // Support: IE 9 - 11 only + // Detect misreporting of content dimensions for box-sizing:border-box elements + boxSizingReliableVal = roundPixelMeasures( divStyle.width ) === 36; + + // Support: IE 9 only + // Detect overflow:scroll screwiness (gh-3699) + // Support: Chrome <=64 + // Don't get tricked when zoom affects offsetWidth (gh-4029) + div.style.position = "absolute"; + scrollboxSizeVal = roundPixelMeasures( div.offsetWidth / 3 ) === 12; + + documentElement.removeChild( container ); + + // Nullify the div so it wouldn't be stored in the memory and + // it will also be a sign that checks already performed + div = null; + } + + function roundPixelMeasures( measure ) { + return Math.round( parseFloat( measure ) ); + } + + var pixelPositionVal, boxSizingReliableVal, scrollboxSizeVal, pixelBoxStylesVal, + reliableTrDimensionsVal, reliableMarginLeftVal, + container = document.createElement( "div" ), + div = document.createElement( "div" ); + + // Finish early in limited (non-browser) environments + if ( !div.style ) { + return; + } + + // Support: IE <=9 - 11 only + // Style of cloned element affects source element cloned (#8908) + div.style.backgroundClip = "content-box"; + div.cloneNode( true ).style.backgroundClip = ""; + support.clearCloneStyle = div.style.backgroundClip === "content-box"; + + jQuery.extend( support, { + boxSizingReliable: function() { + computeStyleTests(); + return boxSizingReliableVal; + }, + pixelBoxStyles: function() { + computeStyleTests(); + return pixelBoxStylesVal; + }, + pixelPosition: function() { + computeStyleTests(); + return pixelPositionVal; + }, + reliableMarginLeft: function() { + computeStyleTests(); + return reliableMarginLeftVal; + }, + scrollboxSize: function() { + computeStyleTests(); + return scrollboxSizeVal; + }, + + // Support: IE 9 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Behavior in IE 9 is more subtle than in newer versions & it passes + // some versions of this test; make sure not to make it pass there! + // + // Support: Firefox 70+ + // Only Firefox includes border widths + // in computed dimensions. (gh-4529) + reliableTrDimensions: function() { + var table, tr, trChild, trStyle; + if ( reliableTrDimensionsVal == null ) { + table = document.createElement( "table" ); + tr = document.createElement( "tr" ); + trChild = document.createElement( "div" ); + + table.style.cssText = "position:absolute;left:-11111px;border-collapse:separate"; + tr.style.cssText = "border:1px solid"; + + // Support: Chrome 86+ + // Height set through cssText does not get applied. + // Computed height then comes back as 0. + tr.style.height = "1px"; + trChild.style.height = "9px"; + + // Support: Android 8 Chrome 86+ + // In our bodyBackground.html iframe, + // display for all div elements is set to "inline", + // which causes a problem only in Android 8 Chrome 86. + // Ensuring the div is display: block + // gets around this issue. + trChild.style.display = "block"; + + documentElement + .appendChild( table ) + .appendChild( tr ) + .appendChild( trChild ); + + trStyle = window.getComputedStyle( tr ); + reliableTrDimensionsVal = ( parseInt( trStyle.height, 10 ) + + parseInt( trStyle.borderTopWidth, 10 ) + + parseInt( trStyle.borderBottomWidth, 10 ) ) === tr.offsetHeight; + + documentElement.removeChild( table ); + } + return reliableTrDimensionsVal; + } + } ); +} )(); + + +function curCSS( elem, name, computed ) { + var width, minWidth, maxWidth, ret, + + // Support: Firefox 51+ + // Retrieving style before computed somehow + // fixes an issue with getting wrong values + // on detached elements + style = elem.style; + + computed = computed || getStyles( elem ); + + // getPropertyValue is needed for: + // .css('filter') (IE 9 only, #12537) + // .css('--customProperty) (#3144) + if ( computed ) { + ret = computed.getPropertyValue( name ) || computed[ name ]; + + if ( ret === "" && !isAttached( elem ) ) { + ret = jQuery.style( elem, name ); + } + + // A tribute to the "awesome hack by Dean Edwards" + // Android Browser returns percentage for some values, + // but width seems to be reliably pixels. + // This is against the CSSOM draft spec: + // https://drafts.csswg.org/cssom/#resolved-values + if ( !support.pixelBoxStyles() && rnumnonpx.test( ret ) && rboxStyle.test( name ) ) { + + // Remember the original values + width = style.width; + minWidth = style.minWidth; + maxWidth = style.maxWidth; + + // Put in the new values to get a computed value out + style.minWidth = style.maxWidth = style.width = ret; + ret = computed.width; + + // Revert the changed values + style.width = width; + style.minWidth = minWidth; + style.maxWidth = maxWidth; + } + } + + return ret !== undefined ? + + // Support: IE <=9 - 11 only + // IE returns zIndex value as an integer. + ret + "" : + ret; +} + + +function addGetHookIf( conditionFn, hookFn ) { + + // Define the hook, we'll check on the first run if it's really needed. + return { + get: function() { + if ( conditionFn() ) { + + // Hook not needed (or it's not possible to use it due + // to missing dependency), remove it. + delete this.get; + return; + } + + // Hook needed; redefine it so that the support test is not executed again. + return ( this.get = hookFn ).apply( this, arguments ); + } + }; +} + + +var cssPrefixes = [ "Webkit", "Moz", "ms" ], + emptyStyle = document.createElement( "div" ).style, + vendorProps = {}; + +// Return a vendor-prefixed property or undefined +function vendorPropName( name ) { + + // Check for vendor prefixed names + var capName = name[ 0 ].toUpperCase() + name.slice( 1 ), + i = cssPrefixes.length; + + while ( i-- ) { + name = cssPrefixes[ i ] + capName; + if ( name in emptyStyle ) { + return name; + } + } +} + +// Return a potentially-mapped jQuery.cssProps or vendor prefixed property +function finalPropName( name ) { + var final = jQuery.cssProps[ name ] || vendorProps[ name ]; + + if ( final ) { + return final; + } + if ( name in emptyStyle ) { + return name; + } + return vendorProps[ name ] = vendorPropName( name ) || name; +} + + +var + + // Swappable if display is none or starts with table + // except "table", "table-cell", or "table-caption" + // See here for display values: https://developer.mozilla.org/en-US/docs/CSS/display + rdisplayswap = /^(none|table(?!-c[ea]).+)/, + rcustomProp = /^--/, + cssShow = { position: "absolute", visibility: "hidden", display: "block" }, + cssNormalTransform = { + letterSpacing: "0", + fontWeight: "400" + }; + +function setPositiveNumber( _elem, value, subtract ) { + + // Any relative (+/-) values have already been + // normalized at this point + var matches = rcssNum.exec( value ); + return matches ? + + // Guard against undefined "subtract", e.g., when used as in cssHooks + Math.max( 0, matches[ 2 ] - ( subtract || 0 ) ) + ( matches[ 3 ] || "px" ) : + value; +} + +function boxModelAdjustment( elem, dimension, box, isBorderBox, styles, computedVal ) { + var i = dimension === "width" ? 1 : 0, + extra = 0, + delta = 0; + + // Adjustment may not be necessary + if ( box === ( isBorderBox ? "border" : "content" ) ) { + return 0; + } + + for ( ; i < 4; i += 2 ) { + + // Both box models exclude margin + if ( box === "margin" ) { + delta += jQuery.css( elem, box + cssExpand[ i ], true, styles ); + } + + // If we get here with a content-box, we're seeking "padding" or "border" or "margin" + if ( !isBorderBox ) { + + // Add padding + delta += jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + + // For "border" or "margin", add border + if ( box !== "padding" ) { + delta += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + + // But still keep track of it otherwise + } else { + extra += jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + + // If we get here with a border-box (content + padding + border), we're seeking "content" or + // "padding" or "margin" + } else { + + // For "content", subtract padding + if ( box === "content" ) { + delta -= jQuery.css( elem, "padding" + cssExpand[ i ], true, styles ); + } + + // For "content" or "padding", subtract border + if ( box !== "margin" ) { + delta -= jQuery.css( elem, "border" + cssExpand[ i ] + "Width", true, styles ); + } + } + } + + // Account for positive content-box scroll gutter when requested by providing computedVal + if ( !isBorderBox && computedVal >= 0 ) { + + // offsetWidth/offsetHeight is a rounded sum of content, padding, scroll gutter, and border + // Assuming integer scroll gutter, subtract the rest and round down + delta += Math.max( 0, Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + computedVal - + delta - + extra - + 0.5 + + // If offsetWidth/offsetHeight is unknown, then we can't determine content-box scroll gutter + // Use an explicit zero to avoid NaN (gh-3964) + ) ) || 0; + } + + return delta; +} + +function getWidthOrHeight( elem, dimension, extra ) { + + // Start with computed style + var styles = getStyles( elem ), + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-4322). + // Fake content-box until we know it's needed to know the true value. + boxSizingNeeded = !support.boxSizingReliable() || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + valueIsBorderBox = isBorderBox, + + val = curCSS( elem, dimension, styles ), + offsetProp = "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ); + + // Support: Firefox <=54 + // Return a confounding non-pixel value or feign ignorance, as appropriate. + if ( rnumnonpx.test( val ) ) { + if ( !extra ) { + return val; + } + val = "auto"; + } + + + // Support: IE 9 - 11 only + // Use offsetWidth/offsetHeight for when box sizing is unreliable. + // In those cases, the computed value can be trusted to be border-box. + if ( ( !support.boxSizingReliable() && isBorderBox || + + // Support: IE 10 - 11+, Edge 15 - 18+ + // IE/Edge misreport `getComputedStyle` of table rows with width/height + // set in CSS while `offset*` properties report correct values. + // Interestingly, in some cases IE 9 doesn't suffer from this issue. + !support.reliableTrDimensions() && nodeName( elem, "tr" ) || + + // Fall back to offsetWidth/offsetHeight when value is "auto" + // This happens for inline elements with no explicit setting (gh-3571) + val === "auto" || + + // Support: Android <=4.1 - 4.3 only + // Also use offsetWidth/offsetHeight for misreported inline dimensions (gh-3602) + !parseFloat( val ) && jQuery.css( elem, "display", false, styles ) === "inline" ) && + + // Make sure the element is visible & connected + elem.getClientRects().length ) { + + isBorderBox = jQuery.css( elem, "boxSizing", false, styles ) === "border-box"; + + // Where available, offsetWidth/offsetHeight approximate border box dimensions. + // Where not available (e.g., SVG), assume unreliable box-sizing and interpret the + // retrieved value as a content box dimension. + valueIsBorderBox = offsetProp in elem; + if ( valueIsBorderBox ) { + val = elem[ offsetProp ]; + } + } + + // Normalize "" and auto + val = parseFloat( val ) || 0; + + // Adjust for the element's box model + return ( val + + boxModelAdjustment( + elem, + dimension, + extra || ( isBorderBox ? "border" : "content" ), + valueIsBorderBox, + styles, + + // Provide the current computed size to request scroll gutter calculation (gh-3589) + val + ) + ) + "px"; +} + +jQuery.extend( { + + // Add in style property hooks for overriding the default + // behavior of getting and setting a style property + cssHooks: { + opacity: { + get: function( elem, computed ) { + if ( computed ) { + + // We should always get a number back from opacity + var ret = curCSS( elem, "opacity" ); + return ret === "" ? "1" : ret; + } + } + } + }, + + // Don't automatically add "px" to these possibly-unitless properties + cssNumber: { + "animationIterationCount": true, + "columnCount": true, + "fillOpacity": true, + "flexGrow": true, + "flexShrink": true, + "fontWeight": true, + "gridArea": true, + "gridColumn": true, + "gridColumnEnd": true, + "gridColumnStart": true, + "gridRow": true, + "gridRowEnd": true, + "gridRowStart": true, + "lineHeight": true, + "opacity": true, + "order": true, + "orphans": true, + "widows": true, + "zIndex": true, + "zoom": true + }, + + // Add in properties whose names you wish to fix before + // setting or getting the value + cssProps: {}, + + // Get and set the style property on a DOM Node + style: function( elem, name, value, extra ) { + + // Don't set styles on text and comment nodes + if ( !elem || elem.nodeType === 3 || elem.nodeType === 8 || !elem.style ) { + return; + } + + // Make sure that we're working with the right name + var ret, type, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ), + style = elem.style; + + // Make sure that we're working with the right name. We don't + // want to query the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Gets hook for the prefixed version, then unprefixed version + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // Check if we're setting a value + if ( value !== undefined ) { + type = typeof value; + + // Convert "+=" or "-=" to relative numbers (#7345) + if ( type === "string" && ( ret = rcssNum.exec( value ) ) && ret[ 1 ] ) { + value = adjustCSS( elem, name, ret ); + + // Fixes bug #9237 + type = "number"; + } + + // Make sure that null and NaN values aren't set (#7116) + if ( value == null || value !== value ) { + return; + } + + // If a number was passed in, add the unit (except for certain CSS properties) + // The isCustomProp check can be removed in jQuery 4.0 when we only auto-append + // "px" to a few hardcoded values. + if ( type === "number" && !isCustomProp ) { + value += ret && ret[ 3 ] || ( jQuery.cssNumber[ origName ] ? "" : "px" ); + } + + // background-* props affect original clone's values + if ( !support.clearCloneStyle && value === "" && name.indexOf( "background" ) === 0 ) { + style[ name ] = "inherit"; + } + + // If a hook was provided, use that value, otherwise just set the specified value + if ( !hooks || !( "set" in hooks ) || + ( value = hooks.set( elem, value, extra ) ) !== undefined ) { + + if ( isCustomProp ) { + style.setProperty( name, value ); + } else { + style[ name ] = value; + } + } + + } else { + + // If a hook was provided get the non-computed value from there + if ( hooks && "get" in hooks && + ( ret = hooks.get( elem, false, extra ) ) !== undefined ) { + + return ret; + } + + // Otherwise just get the value from the style object + return style[ name ]; + } + }, + + css: function( elem, name, extra, styles ) { + var val, num, hooks, + origName = camelCase( name ), + isCustomProp = rcustomProp.test( name ); + + // Make sure that we're working with the right name. We don't + // want to modify the value if it is a CSS custom property + // since they are user-defined. + if ( !isCustomProp ) { + name = finalPropName( origName ); + } + + // Try prefixed name followed by the unprefixed name + hooks = jQuery.cssHooks[ name ] || jQuery.cssHooks[ origName ]; + + // If a hook was provided get the computed value from there + if ( hooks && "get" in hooks ) { + val = hooks.get( elem, true, extra ); + } + + // Otherwise, if a way to get the computed value exists, use that + if ( val === undefined ) { + val = curCSS( elem, name, styles ); + } + + // Convert "normal" to computed value + if ( val === "normal" && name in cssNormalTransform ) { + val = cssNormalTransform[ name ]; + } + + // Make numeric if forced or a qualifier was provided and val looks numeric + if ( extra === "" || extra ) { + num = parseFloat( val ); + return extra === true || isFinite( num ) ? num || 0 : val; + } + + return val; + } +} ); + +jQuery.each( [ "height", "width" ], function( _i, dimension ) { + jQuery.cssHooks[ dimension ] = { + get: function( elem, computed, extra ) { + if ( computed ) { + + // Certain elements can have dimension info if we invisibly show them + // but it must have a current display style that would benefit + return rdisplayswap.test( jQuery.css( elem, "display" ) ) && + + // Support: Safari 8+ + // Table columns in Safari have non-zero offsetWidth & zero + // getBoundingClientRect().width unless display is changed. + // Support: IE <=11 only + // Running getBoundingClientRect on a disconnected node + // in IE throws an error. + ( !elem.getClientRects().length || !elem.getBoundingClientRect().width ) ? + swap( elem, cssShow, function() { + return getWidthOrHeight( elem, dimension, extra ); + } ) : + getWidthOrHeight( elem, dimension, extra ); + } + }, + + set: function( elem, value, extra ) { + var matches, + styles = getStyles( elem ), + + // Only read styles.position if the test has a chance to fail + // to avoid forcing a reflow. + scrollboxSizeBuggy = !support.scrollboxSize() && + styles.position === "absolute", + + // To avoid forcing a reflow, only fetch boxSizing if we need it (gh-3991) + boxSizingNeeded = scrollboxSizeBuggy || extra, + isBorderBox = boxSizingNeeded && + jQuery.css( elem, "boxSizing", false, styles ) === "border-box", + subtract = extra ? + boxModelAdjustment( + elem, + dimension, + extra, + isBorderBox, + styles + ) : + 0; + + // Account for unreliable border-box dimensions by comparing offset* to computed and + // faking a content-box to get border and padding (gh-3699) + if ( isBorderBox && scrollboxSizeBuggy ) { + subtract -= Math.ceil( + elem[ "offset" + dimension[ 0 ].toUpperCase() + dimension.slice( 1 ) ] - + parseFloat( styles[ dimension ] ) - + boxModelAdjustment( elem, dimension, "border", false, styles ) - + 0.5 + ); + } + + // Convert to pixels if value adjustment is needed + if ( subtract && ( matches = rcssNum.exec( value ) ) && + ( matches[ 3 ] || "px" ) !== "px" ) { + + elem.style[ dimension ] = value; + value = jQuery.css( elem, dimension ); + } + + return setPositiveNumber( elem, value, subtract ); + } + }; +} ); + +jQuery.cssHooks.marginLeft = addGetHookIf( support.reliableMarginLeft, + function( elem, computed ) { + if ( computed ) { + return ( parseFloat( curCSS( elem, "marginLeft" ) ) || + elem.getBoundingClientRect().left - + swap( elem, { marginLeft: 0 }, function() { + return elem.getBoundingClientRect().left; + } ) + ) + "px"; + } + } +); + +// These hooks are used by animate to expand properties +jQuery.each( { + margin: "", + padding: "", + border: "Width" +}, function( prefix, suffix ) { + jQuery.cssHooks[ prefix + suffix ] = { + expand: function( value ) { + var i = 0, + expanded = {}, + + // Assumes a single number if not a string + parts = typeof value === "string" ? value.split( " " ) : [ value ]; + + for ( ; i < 4; i++ ) { + expanded[ prefix + cssExpand[ i ] + suffix ] = + parts[ i ] || parts[ i - 2 ] || parts[ 0 ]; + } + + return expanded; + } + }; + + if ( prefix !== "margin" ) { + jQuery.cssHooks[ prefix + suffix ].set = setPositiveNumber; + } +} ); + +jQuery.fn.extend( { + css: function( name, value ) { + return access( this, function( elem, name, value ) { + var styles, len, + map = {}, + i = 0; + + if ( Array.isArray( name ) ) { + styles = getStyles( elem ); + len = name.length; + + for ( ; i < len; i++ ) { + map[ name[ i ] ] = jQuery.css( elem, name[ i ], false, styles ); + } + + return map; + } + + return value !== undefined ? + jQuery.style( elem, name, value ) : + jQuery.css( elem, name ); + }, name, value, arguments.length > 1 ); + } +} ); + + +function Tween( elem, options, prop, end, easing ) { + return new Tween.prototype.init( elem, options, prop, end, easing ); +} +jQuery.Tween = Tween; + +Tween.prototype = { + constructor: Tween, + init: function( elem, options, prop, end, easing, unit ) { + this.elem = elem; + this.prop = prop; + this.easing = easing || jQuery.easing._default; + this.options = options; + this.start = this.now = this.cur(); + this.end = end; + this.unit = unit || ( jQuery.cssNumber[ prop ] ? "" : "px" ); + }, + cur: function() { + var hooks = Tween.propHooks[ this.prop ]; + + return hooks && hooks.get ? + hooks.get( this ) : + Tween.propHooks._default.get( this ); + }, + run: function( percent ) { + var eased, + hooks = Tween.propHooks[ this.prop ]; + + if ( this.options.duration ) { + this.pos = eased = jQuery.easing[ this.easing ]( + percent, this.options.duration * percent, 0, 1, this.options.duration + ); + } else { + this.pos = eased = percent; + } + this.now = ( this.end - this.start ) * eased + this.start; + + if ( this.options.step ) { + this.options.step.call( this.elem, this.now, this ); + } + + if ( hooks && hooks.set ) { + hooks.set( this ); + } else { + Tween.propHooks._default.set( this ); + } + return this; + } +}; + +Tween.prototype.init.prototype = Tween.prototype; + +Tween.propHooks = { + _default: { + get: function( tween ) { + var result; + + // Use a property on the element directly when it is not a DOM element, + // or when there is no matching style property that exists. + if ( tween.elem.nodeType !== 1 || + tween.elem[ tween.prop ] != null && tween.elem.style[ tween.prop ] == null ) { + return tween.elem[ tween.prop ]; + } + + // Passing an empty string as a 3rd parameter to .css will automatically + // attempt a parseFloat and fallback to a string if the parse fails. + // Simple values such as "10px" are parsed to Float; + // complex values such as "rotate(1rad)" are returned as-is. + result = jQuery.css( tween.elem, tween.prop, "" ); + + // Empty strings, null, undefined and "auto" are converted to 0. + return !result || result === "auto" ? 0 : result; + }, + set: function( tween ) { + + // Use step hook for back compat. + // Use cssHook if its there. + // Use .style if available and use plain properties where available. + if ( jQuery.fx.step[ tween.prop ] ) { + jQuery.fx.step[ tween.prop ]( tween ); + } else if ( tween.elem.nodeType === 1 && ( + jQuery.cssHooks[ tween.prop ] || + tween.elem.style[ finalPropName( tween.prop ) ] != null ) ) { + jQuery.style( tween.elem, tween.prop, tween.now + tween.unit ); + } else { + tween.elem[ tween.prop ] = tween.now; + } + } + } +}; + +// Support: IE <=9 only +// Panic based approach to setting things on disconnected nodes +Tween.propHooks.scrollTop = Tween.propHooks.scrollLeft = { + set: function( tween ) { + if ( tween.elem.nodeType && tween.elem.parentNode ) { + tween.elem[ tween.prop ] = tween.now; + } + } +}; + +jQuery.easing = { + linear: function( p ) { + return p; + }, + swing: function( p ) { + return 0.5 - Math.cos( p * Math.PI ) / 2; + }, + _default: "swing" +}; + +jQuery.fx = Tween.prototype.init; + +// Back compat <1.8 extension point +jQuery.fx.step = {}; + + + + +var + fxNow, inProgress, + rfxtypes = /^(?:toggle|show|hide)$/, + rrun = /queueHooks$/; + +function schedule() { + if ( inProgress ) { + if ( document.hidden === false && window.requestAnimationFrame ) { + window.requestAnimationFrame( schedule ); + } else { + window.setTimeout( schedule, jQuery.fx.interval ); + } + + jQuery.fx.tick(); + } +} + +// Animations created synchronously will run synchronously +function createFxNow() { + window.setTimeout( function() { + fxNow = undefined; + } ); + return ( fxNow = Date.now() ); +} + +// Generate parameters to create a standard animation +function genFx( type, includeWidth ) { + var which, + i = 0, + attrs = { height: type }; + + // If we include width, step value is 1 to do all cssExpand values, + // otherwise step value is 2 to skip over Left and Right + includeWidth = includeWidth ? 1 : 0; + for ( ; i < 4; i += 2 - includeWidth ) { + which = cssExpand[ i ]; + attrs[ "margin" + which ] = attrs[ "padding" + which ] = type; + } + + if ( includeWidth ) { + attrs.opacity = attrs.width = type; + } + + return attrs; +} + +function createTween( value, prop, animation ) { + var tween, + collection = ( Animation.tweeners[ prop ] || [] ).concat( Animation.tweeners[ "*" ] ), + index = 0, + length = collection.length; + for ( ; index < length; index++ ) { + if ( ( tween = collection[ index ].call( animation, prop, value ) ) ) { + + // We're done with this property + return tween; + } + } +} + +function defaultPrefilter( elem, props, opts ) { + var prop, value, toggle, hooks, oldfire, propTween, restoreDisplay, display, + isBox = "width" in props || "height" in props, + anim = this, + orig = {}, + style = elem.style, + hidden = elem.nodeType && isHiddenWithinTree( elem ), + dataShow = dataPriv.get( elem, "fxshow" ); + + // Queue-skipping animations hijack the fx hooks + if ( !opts.queue ) { + hooks = jQuery._queueHooks( elem, "fx" ); + if ( hooks.unqueued == null ) { + hooks.unqueued = 0; + oldfire = hooks.empty.fire; + hooks.empty.fire = function() { + if ( !hooks.unqueued ) { + oldfire(); + } + }; + } + hooks.unqueued++; + + anim.always( function() { + + // Ensure the complete handler is called before this completes + anim.always( function() { + hooks.unqueued--; + if ( !jQuery.queue( elem, "fx" ).length ) { + hooks.empty.fire(); + } + } ); + } ); + } + + // Detect show/hide animations + for ( prop in props ) { + value = props[ prop ]; + if ( rfxtypes.test( value ) ) { + delete props[ prop ]; + toggle = toggle || value === "toggle"; + if ( value === ( hidden ? "hide" : "show" ) ) { + + // Pretend to be hidden if this is a "show" and + // there is still data from a stopped show/hide + if ( value === "show" && dataShow && dataShow[ prop ] !== undefined ) { + hidden = true; + + // Ignore all other no-op show/hide data + } else { + continue; + } + } + orig[ prop ] = dataShow && dataShow[ prop ] || jQuery.style( elem, prop ); + } + } + + // Bail out if this is a no-op like .hide().hide() + propTween = !jQuery.isEmptyObject( props ); + if ( !propTween && jQuery.isEmptyObject( orig ) ) { + return; + } + + // Restrict "overflow" and "display" styles during box animations + if ( isBox && elem.nodeType === 1 ) { + + // Support: IE <=9 - 11, Edge 12 - 15 + // Record all 3 overflow attributes because IE does not infer the shorthand + // from identically-valued overflowX and overflowY and Edge just mirrors + // the overflowX value there. + opts.overflow = [ style.overflow, style.overflowX, style.overflowY ]; + + // Identify a display type, preferring old show/hide data over the CSS cascade + restoreDisplay = dataShow && dataShow.display; + if ( restoreDisplay == null ) { + restoreDisplay = dataPriv.get( elem, "display" ); + } + display = jQuery.css( elem, "display" ); + if ( display === "none" ) { + if ( restoreDisplay ) { + display = restoreDisplay; + } else { + + // Get nonempty value(s) by temporarily forcing visibility + showHide( [ elem ], true ); + restoreDisplay = elem.style.display || restoreDisplay; + display = jQuery.css( elem, "display" ); + showHide( [ elem ] ); + } + } + + // Animate inline elements as inline-block + if ( display === "inline" || display === "inline-block" && restoreDisplay != null ) { + if ( jQuery.css( elem, "float" ) === "none" ) { + + // Restore the original display value at the end of pure show/hide animations + if ( !propTween ) { + anim.done( function() { + style.display = restoreDisplay; + } ); + if ( restoreDisplay == null ) { + display = style.display; + restoreDisplay = display === "none" ? "" : display; + } + } + style.display = "inline-block"; + } + } + } + + if ( opts.overflow ) { + style.overflow = "hidden"; + anim.always( function() { + style.overflow = opts.overflow[ 0 ]; + style.overflowX = opts.overflow[ 1 ]; + style.overflowY = opts.overflow[ 2 ]; + } ); + } + + // Implement show/hide animations + propTween = false; + for ( prop in orig ) { + + // General show/hide setup for this element animation + if ( !propTween ) { + if ( dataShow ) { + if ( "hidden" in dataShow ) { + hidden = dataShow.hidden; + } + } else { + dataShow = dataPriv.access( elem, "fxshow", { display: restoreDisplay } ); + } + + // Store hidden/visible for toggle so `.stop().toggle()` "reverses" + if ( toggle ) { + dataShow.hidden = !hidden; + } + + // Show elements before animating them + if ( hidden ) { + showHide( [ elem ], true ); + } + + /* eslint-disable no-loop-func */ + + anim.done( function() { + + /* eslint-enable no-loop-func */ + + // The final step of a "hide" animation is actually hiding the element + if ( !hidden ) { + showHide( [ elem ] ); + } + dataPriv.remove( elem, "fxshow" ); + for ( prop in orig ) { + jQuery.style( elem, prop, orig[ prop ] ); + } + } ); + } + + // Per-property setup + propTween = createTween( hidden ? dataShow[ prop ] : 0, prop, anim ); + if ( !( prop in dataShow ) ) { + dataShow[ prop ] = propTween.start; + if ( hidden ) { + propTween.end = propTween.start; + propTween.start = 0; + } + } + } +} + +function propFilter( props, specialEasing ) { + var index, name, easing, value, hooks; + + // camelCase, specialEasing and expand cssHook pass + for ( index in props ) { + name = camelCase( index ); + easing = specialEasing[ name ]; + value = props[ index ]; + if ( Array.isArray( value ) ) { + easing = value[ 1 ]; + value = props[ index ] = value[ 0 ]; + } + + if ( index !== name ) { + props[ name ] = value; + delete props[ index ]; + } + + hooks = jQuery.cssHooks[ name ]; + if ( hooks && "expand" in hooks ) { + value = hooks.expand( value ); + delete props[ name ]; + + // Not quite $.extend, this won't overwrite existing keys. + // Reusing 'index' because we have the correct "name" + for ( index in value ) { + if ( !( index in props ) ) { + props[ index ] = value[ index ]; + specialEasing[ index ] = easing; + } + } + } else { + specialEasing[ name ] = easing; + } + } +} + +function Animation( elem, properties, options ) { + var result, + stopped, + index = 0, + length = Animation.prefilters.length, + deferred = jQuery.Deferred().always( function() { + + // Don't match elem in the :animated selector + delete tick.elem; + } ), + tick = function() { + if ( stopped ) { + return false; + } + var currentTime = fxNow || createFxNow(), + remaining = Math.max( 0, animation.startTime + animation.duration - currentTime ), + + // Support: Android 2.3 only + // Archaic crash bug won't allow us to use `1 - ( 0.5 || 0 )` (#12497) + temp = remaining / animation.duration || 0, + percent = 1 - temp, + index = 0, + length = animation.tweens.length; + + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( percent ); + } + + deferred.notifyWith( elem, [ animation, percent, remaining ] ); + + // If there's more to do, yield + if ( percent < 1 && length ) { + return remaining; + } + + // If this was an empty animation, synthesize a final progress notification + if ( !length ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + } + + // Resolve the animation and report its conclusion + deferred.resolveWith( elem, [ animation ] ); + return false; + }, + animation = deferred.promise( { + elem: elem, + props: jQuery.extend( {}, properties ), + opts: jQuery.extend( true, { + specialEasing: {}, + easing: jQuery.easing._default + }, options ), + originalProperties: properties, + originalOptions: options, + startTime: fxNow || createFxNow(), + duration: options.duration, + tweens: [], + createTween: function( prop, end ) { + var tween = jQuery.Tween( elem, animation.opts, prop, end, + animation.opts.specialEasing[ prop ] || animation.opts.easing ); + animation.tweens.push( tween ); + return tween; + }, + stop: function( gotoEnd ) { + var index = 0, + + // If we are going to the end, we want to run all the tweens + // otherwise we skip this part + length = gotoEnd ? animation.tweens.length : 0; + if ( stopped ) { + return this; + } + stopped = true; + for ( ; index < length; index++ ) { + animation.tweens[ index ].run( 1 ); + } + + // Resolve when we played the last frame; otherwise, reject + if ( gotoEnd ) { + deferred.notifyWith( elem, [ animation, 1, 0 ] ); + deferred.resolveWith( elem, [ animation, gotoEnd ] ); + } else { + deferred.rejectWith( elem, [ animation, gotoEnd ] ); + } + return this; + } + } ), + props = animation.props; + + propFilter( props, animation.opts.specialEasing ); + + for ( ; index < length; index++ ) { + result = Animation.prefilters[ index ].call( animation, elem, props, animation.opts ); + if ( result ) { + if ( isFunction( result.stop ) ) { + jQuery._queueHooks( animation.elem, animation.opts.queue ).stop = + result.stop.bind( result ); + } + return result; + } + } + + jQuery.map( props, createTween, animation ); + + if ( isFunction( animation.opts.start ) ) { + animation.opts.start.call( elem, animation ); + } + + // Attach callbacks from options + animation + .progress( animation.opts.progress ) + .done( animation.opts.done, animation.opts.complete ) + .fail( animation.opts.fail ) + .always( animation.opts.always ); + + jQuery.fx.timer( + jQuery.extend( tick, { + elem: elem, + anim: animation, + queue: animation.opts.queue + } ) + ); + + return animation; +} + +jQuery.Animation = jQuery.extend( Animation, { + + tweeners: { + "*": [ function( prop, value ) { + var tween = this.createTween( prop, value ); + adjustCSS( tween.elem, prop, rcssNum.exec( value ), tween ); + return tween; + } ] + }, + + tweener: function( props, callback ) { + if ( isFunction( props ) ) { + callback = props; + props = [ "*" ]; + } else { + props = props.match( rnothtmlwhite ); + } + + var prop, + index = 0, + length = props.length; + + for ( ; index < length; index++ ) { + prop = props[ index ]; + Animation.tweeners[ prop ] = Animation.tweeners[ prop ] || []; + Animation.tweeners[ prop ].unshift( callback ); + } + }, + + prefilters: [ defaultPrefilter ], + + prefilter: function( callback, prepend ) { + if ( prepend ) { + Animation.prefilters.unshift( callback ); + } else { + Animation.prefilters.push( callback ); + } + } +} ); + +jQuery.speed = function( speed, easing, fn ) { + var opt = speed && typeof speed === "object" ? jQuery.extend( {}, speed ) : { + complete: fn || !fn && easing || + isFunction( speed ) && speed, + duration: speed, + easing: fn && easing || easing && !isFunction( easing ) && easing + }; + + // Go to the end state if fx are off + if ( jQuery.fx.off ) { + opt.duration = 0; + + } else { + if ( typeof opt.duration !== "number" ) { + if ( opt.duration in jQuery.fx.speeds ) { + opt.duration = jQuery.fx.speeds[ opt.duration ]; + + } else { + opt.duration = jQuery.fx.speeds._default; + } + } + } + + // Normalize opt.queue - true/undefined/null -> "fx" + if ( opt.queue == null || opt.queue === true ) { + opt.queue = "fx"; + } + + // Queueing + opt.old = opt.complete; + + opt.complete = function() { + if ( isFunction( opt.old ) ) { + opt.old.call( this ); + } + + if ( opt.queue ) { + jQuery.dequeue( this, opt.queue ); + } + }; + + return opt; +}; + +jQuery.fn.extend( { + fadeTo: function( speed, to, easing, callback ) { + + // Show any hidden elements after setting opacity to 0 + return this.filter( isHiddenWithinTree ).css( "opacity", 0 ).show() + + // Animate to the value specified + .end().animate( { opacity: to }, speed, easing, callback ); + }, + animate: function( prop, speed, easing, callback ) { + var empty = jQuery.isEmptyObject( prop ), + optall = jQuery.speed( speed, easing, callback ), + doAnimation = function() { + + // Operate on a copy of prop so per-property easing won't be lost + var anim = Animation( this, jQuery.extend( {}, prop ), optall ); + + // Empty animations, or finishing resolves immediately + if ( empty || dataPriv.get( this, "finish" ) ) { + anim.stop( true ); + } + }; + + doAnimation.finish = doAnimation; + + return empty || optall.queue === false ? + this.each( doAnimation ) : + this.queue( optall.queue, doAnimation ); + }, + stop: function( type, clearQueue, gotoEnd ) { + var stopQueue = function( hooks ) { + var stop = hooks.stop; + delete hooks.stop; + stop( gotoEnd ); + }; + + if ( typeof type !== "string" ) { + gotoEnd = clearQueue; + clearQueue = type; + type = undefined; + } + if ( clearQueue ) { + this.queue( type || "fx", [] ); + } + + return this.each( function() { + var dequeue = true, + index = type != null && type + "queueHooks", + timers = jQuery.timers, + data = dataPriv.get( this ); + + if ( index ) { + if ( data[ index ] && data[ index ].stop ) { + stopQueue( data[ index ] ); + } + } else { + for ( index in data ) { + if ( data[ index ] && data[ index ].stop && rrun.test( index ) ) { + stopQueue( data[ index ] ); + } + } + } + + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && + ( type == null || timers[ index ].queue === type ) ) { + + timers[ index ].anim.stop( gotoEnd ); + dequeue = false; + timers.splice( index, 1 ); + } + } + + // Start the next in the queue if the last step wasn't forced. + // Timers currently will call their complete callbacks, which + // will dequeue but only if they were gotoEnd. + if ( dequeue || !gotoEnd ) { + jQuery.dequeue( this, type ); + } + } ); + }, + finish: function( type ) { + if ( type !== false ) { + type = type || "fx"; + } + return this.each( function() { + var index, + data = dataPriv.get( this ), + queue = data[ type + "queue" ], + hooks = data[ type + "queueHooks" ], + timers = jQuery.timers, + length = queue ? queue.length : 0; + + // Enable finishing flag on private data + data.finish = true; + + // Empty the queue first + jQuery.queue( this, type, [] ); + + if ( hooks && hooks.stop ) { + hooks.stop.call( this, true ); + } + + // Look for any active animations, and finish them + for ( index = timers.length; index--; ) { + if ( timers[ index ].elem === this && timers[ index ].queue === type ) { + timers[ index ].anim.stop( true ); + timers.splice( index, 1 ); + } + } + + // Look for any animations in the old queue and finish them + for ( index = 0; index < length; index++ ) { + if ( queue[ index ] && queue[ index ].finish ) { + queue[ index ].finish.call( this ); + } + } + + // Turn off finishing flag + delete data.finish; + } ); + } +} ); + +jQuery.each( [ "toggle", "show", "hide" ], function( _i, name ) { + var cssFn = jQuery.fn[ name ]; + jQuery.fn[ name ] = function( speed, easing, callback ) { + return speed == null || typeof speed === "boolean" ? + cssFn.apply( this, arguments ) : + this.animate( genFx( name, true ), speed, easing, callback ); + }; +} ); + +// Generate shortcuts for custom animations +jQuery.each( { + slideDown: genFx( "show" ), + slideUp: genFx( "hide" ), + slideToggle: genFx( "toggle" ), + fadeIn: { opacity: "show" }, + fadeOut: { opacity: "hide" }, + fadeToggle: { opacity: "toggle" } +}, function( name, props ) { + jQuery.fn[ name ] = function( speed, easing, callback ) { + return this.animate( props, speed, easing, callback ); + }; +} ); + +jQuery.timers = []; +jQuery.fx.tick = function() { + var timer, + i = 0, + timers = jQuery.timers; + + fxNow = Date.now(); + + for ( ; i < timers.length; i++ ) { + timer = timers[ i ]; + + // Run the timer and safely remove it when done (allowing for external removal) + if ( !timer() && timers[ i ] === timer ) { + timers.splice( i--, 1 ); + } + } + + if ( !timers.length ) { + jQuery.fx.stop(); + } + fxNow = undefined; +}; + +jQuery.fx.timer = function( timer ) { + jQuery.timers.push( timer ); + jQuery.fx.start(); +}; + +jQuery.fx.interval = 13; +jQuery.fx.start = function() { + if ( inProgress ) { + return; + } + + inProgress = true; + schedule(); +}; + +jQuery.fx.stop = function() { + inProgress = null; +}; + +jQuery.fx.speeds = { + slow: 600, + fast: 200, + + // Default speed + _default: 400 +}; + + +// Based off of the plugin by Clint Helfers, with permission. +// https://web.archive.org/web/20100324014747/http://blindsignals.com/index.php/2009/07/jquery-delay/ +jQuery.fn.delay = function( time, type ) { + time = jQuery.fx ? jQuery.fx.speeds[ time ] || time : time; + type = type || "fx"; + + return this.queue( type, function( next, hooks ) { + var timeout = window.setTimeout( next, time ); + hooks.stop = function() { + window.clearTimeout( timeout ); + }; + } ); +}; + + +( function() { + var input = document.createElement( "input" ), + select = document.createElement( "select" ), + opt = select.appendChild( document.createElement( "option" ) ); + + input.type = "checkbox"; + + // Support: Android <=4.3 only + // Default value for a checkbox should be "on" + support.checkOn = input.value !== ""; + + // Support: IE <=11 only + // Must access selectedIndex to make default options select + support.optSelected = opt.selected; + + // Support: IE <=11 only + // An input loses its value after becoming a radio + input = document.createElement( "input" ); + input.value = "t"; + input.type = "radio"; + support.radioValue = input.value === "t"; +} )(); + + +var boolHook, + attrHandle = jQuery.expr.attrHandle; + +jQuery.fn.extend( { + attr: function( name, value ) { + return access( this, jQuery.attr, name, value, arguments.length > 1 ); + }, + + removeAttr: function( name ) { + return this.each( function() { + jQuery.removeAttr( this, name ); + } ); + } +} ); + +jQuery.extend( { + attr: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set attributes on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + // Fallback to prop when attributes are not supported + if ( typeof elem.getAttribute === "undefined" ) { + return jQuery.prop( elem, name, value ); + } + + // Attribute hooks are determined by the lowercase version + // Grab necessary hook if one is defined + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + hooks = jQuery.attrHooks[ name.toLowerCase() ] || + ( jQuery.expr.match.bool.test( name ) ? boolHook : undefined ); + } + + if ( value !== undefined ) { + if ( value === null ) { + jQuery.removeAttr( elem, name ); + return; + } + + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + elem.setAttribute( name, value + "" ); + return value; + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + ret = jQuery.find.attr( elem, name ); + + // Non-existent attributes return null, we normalize to undefined + return ret == null ? undefined : ret; + }, + + attrHooks: { + type: { + set: function( elem, value ) { + if ( !support.radioValue && value === "radio" && + nodeName( elem, "input" ) ) { + var val = elem.value; + elem.setAttribute( "type", value ); + if ( val ) { + elem.value = val; + } + return value; + } + } + } + }, + + removeAttr: function( elem, value ) { + var name, + i = 0, + + // Attribute names can contain non-HTML whitespace characters + // https://html.spec.whatwg.org/multipage/syntax.html#attributes-2 + attrNames = value && value.match( rnothtmlwhite ); + + if ( attrNames && elem.nodeType === 1 ) { + while ( ( name = attrNames[ i++ ] ) ) { + elem.removeAttribute( name ); + } + } + } +} ); + +// Hooks for boolean attributes +boolHook = { + set: function( elem, value, name ) { + if ( value === false ) { + + // Remove boolean attributes when set to false + jQuery.removeAttr( elem, name ); + } else { + elem.setAttribute( name, name ); + } + return name; + } +}; + +jQuery.each( jQuery.expr.match.bool.source.match( /\w+/g ), function( _i, name ) { + var getter = attrHandle[ name ] || jQuery.find.attr; + + attrHandle[ name ] = function( elem, name, isXML ) { + var ret, handle, + lowercaseName = name.toLowerCase(); + + if ( !isXML ) { + + // Avoid an infinite loop by temporarily removing this function from the getter + handle = attrHandle[ lowercaseName ]; + attrHandle[ lowercaseName ] = ret; + ret = getter( elem, name, isXML ) != null ? + lowercaseName : + null; + attrHandle[ lowercaseName ] = handle; + } + return ret; + }; +} ); + + + + +var rfocusable = /^(?:input|select|textarea|button)$/i, + rclickable = /^(?:a|area)$/i; + +jQuery.fn.extend( { + prop: function( name, value ) { + return access( this, jQuery.prop, name, value, arguments.length > 1 ); + }, + + removeProp: function( name ) { + return this.each( function() { + delete this[ jQuery.propFix[ name ] || name ]; + } ); + } +} ); + +jQuery.extend( { + prop: function( elem, name, value ) { + var ret, hooks, + nType = elem.nodeType; + + // Don't get/set properties on text, comment and attribute nodes + if ( nType === 3 || nType === 8 || nType === 2 ) { + return; + } + + if ( nType !== 1 || !jQuery.isXMLDoc( elem ) ) { + + // Fix name and attach hooks + name = jQuery.propFix[ name ] || name; + hooks = jQuery.propHooks[ name ]; + } + + if ( value !== undefined ) { + if ( hooks && "set" in hooks && + ( ret = hooks.set( elem, value, name ) ) !== undefined ) { + return ret; + } + + return ( elem[ name ] = value ); + } + + if ( hooks && "get" in hooks && ( ret = hooks.get( elem, name ) ) !== null ) { + return ret; + } + + return elem[ name ]; + }, + + propHooks: { + tabIndex: { + get: function( elem ) { + + // Support: IE <=9 - 11 only + // elem.tabIndex doesn't always return the + // correct value when it hasn't been explicitly set + // https://web.archive.org/web/20141116233347/http://fluidproject.org/blog/2008/01/09/getting-setting-and-removing-tabindex-values-with-javascript/ + // Use proper attribute retrieval(#12072) + var tabindex = jQuery.find.attr( elem, "tabindex" ); + + if ( tabindex ) { + return parseInt( tabindex, 10 ); + } + + if ( + rfocusable.test( elem.nodeName ) || + rclickable.test( elem.nodeName ) && + elem.href + ) { + return 0; + } + + return -1; + } + } + }, + + propFix: { + "for": "htmlFor", + "class": "className" + } +} ); + +// Support: IE <=11 only +// Accessing the selectedIndex property +// forces the browser to respect setting selected +// on the option +// The getter ensures a default option is selected +// when in an optgroup +// eslint rule "no-unused-expressions" is disabled for this code +// since it considers such accessions noop +if ( !support.optSelected ) { + jQuery.propHooks.selected = { + get: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent && parent.parentNode ) { + parent.parentNode.selectedIndex; + } + return null; + }, + set: function( elem ) { + + /* eslint no-unused-expressions: "off" */ + + var parent = elem.parentNode; + if ( parent ) { + parent.selectedIndex; + + if ( parent.parentNode ) { + parent.parentNode.selectedIndex; + } + } + } + }; +} + +jQuery.each( [ + "tabIndex", + "readOnly", + "maxLength", + "cellSpacing", + "cellPadding", + "rowSpan", + "colSpan", + "useMap", + "frameBorder", + "contentEditable" +], function() { + jQuery.propFix[ this.toLowerCase() ] = this; +} ); + + + + + // Strip and collapse whitespace according to HTML spec + // https://infra.spec.whatwg.org/#strip-and-collapse-ascii-whitespace + function stripAndCollapse( value ) { + var tokens = value.match( rnothtmlwhite ) || []; + return tokens.join( " " ); + } + + +function getClass( elem ) { + return elem.getAttribute && elem.getAttribute( "class" ) || ""; +} + +function classesToArray( value ) { + if ( Array.isArray( value ) ) { + return value; + } + if ( typeof value === "string" ) { + return value.match( rnothtmlwhite ) || []; + } + return []; +} + +jQuery.fn.extend( { + addClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).addClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + if ( cur.indexOf( " " + clazz + " " ) < 0 ) { + cur += clazz + " "; + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + removeClass: function( value ) { + var classes, elem, cur, curValue, clazz, j, finalValue, + i = 0; + + if ( isFunction( value ) ) { + return this.each( function( j ) { + jQuery( this ).removeClass( value.call( this, j, getClass( this ) ) ); + } ); + } + + if ( !arguments.length ) { + return this.attr( "class", "" ); + } + + classes = classesToArray( value ); + + if ( classes.length ) { + while ( ( elem = this[ i++ ] ) ) { + curValue = getClass( elem ); + + // This expression is here for better compressibility (see addClass) + cur = elem.nodeType === 1 && ( " " + stripAndCollapse( curValue ) + " " ); + + if ( cur ) { + j = 0; + while ( ( clazz = classes[ j++ ] ) ) { + + // Remove *all* instances + while ( cur.indexOf( " " + clazz + " " ) > -1 ) { + cur = cur.replace( " " + clazz + " ", " " ); + } + } + + // Only assign if different to avoid unneeded rendering. + finalValue = stripAndCollapse( cur ); + if ( curValue !== finalValue ) { + elem.setAttribute( "class", finalValue ); + } + } + } + } + + return this; + }, + + toggleClass: function( value, stateVal ) { + var type = typeof value, + isValidValue = type === "string" || Array.isArray( value ); + + if ( typeof stateVal === "boolean" && isValidValue ) { + return stateVal ? this.addClass( value ) : this.removeClass( value ); + } + + if ( isFunction( value ) ) { + return this.each( function( i ) { + jQuery( this ).toggleClass( + value.call( this, i, getClass( this ), stateVal ), + stateVal + ); + } ); + } + + return this.each( function() { + var className, i, self, classNames; + + if ( isValidValue ) { + + // Toggle individual class names + i = 0; + self = jQuery( this ); + classNames = classesToArray( value ); + + while ( ( className = classNames[ i++ ] ) ) { + + // Check each className given, space separated list + if ( self.hasClass( className ) ) { + self.removeClass( className ); + } else { + self.addClass( className ); + } + } + + // Toggle whole class name + } else if ( value === undefined || type === "boolean" ) { + className = getClass( this ); + if ( className ) { + + // Store className if set + dataPriv.set( this, "__className__", className ); + } + + // If the element has a class name or if we're passed `false`, + // then remove the whole classname (if there was one, the above saved it). + // Otherwise bring back whatever was previously saved (if anything), + // falling back to the empty string if nothing was stored. + if ( this.setAttribute ) { + this.setAttribute( "class", + className || value === false ? + "" : + dataPriv.get( this, "__className__" ) || "" + ); + } + } + } ); + }, + + hasClass: function( selector ) { + var className, elem, + i = 0; + + className = " " + selector + " "; + while ( ( elem = this[ i++ ] ) ) { + if ( elem.nodeType === 1 && + ( " " + stripAndCollapse( getClass( elem ) ) + " " ).indexOf( className ) > -1 ) { + return true; + } + } + + return false; + } +} ); + + + + +var rreturn = /\r/g; + +jQuery.fn.extend( { + val: function( value ) { + var hooks, ret, valueIsFunction, + elem = this[ 0 ]; + + if ( !arguments.length ) { + if ( elem ) { + hooks = jQuery.valHooks[ elem.type ] || + jQuery.valHooks[ elem.nodeName.toLowerCase() ]; + + if ( hooks && + "get" in hooks && + ( ret = hooks.get( elem, "value" ) ) !== undefined + ) { + return ret; + } + + ret = elem.value; + + // Handle most common string cases + if ( typeof ret === "string" ) { + return ret.replace( rreturn, "" ); + } + + // Handle cases where value is null/undef or number + return ret == null ? "" : ret; + } + + return; + } + + valueIsFunction = isFunction( value ); + + return this.each( function( i ) { + var val; + + if ( this.nodeType !== 1 ) { + return; + } + + if ( valueIsFunction ) { + val = value.call( this, i, jQuery( this ).val() ); + } else { + val = value; + } + + // Treat null/undefined as ""; convert numbers to string + if ( val == null ) { + val = ""; + + } else if ( typeof val === "number" ) { + val += ""; + + } else if ( Array.isArray( val ) ) { + val = jQuery.map( val, function( value ) { + return value == null ? "" : value + ""; + } ); + } + + hooks = jQuery.valHooks[ this.type ] || jQuery.valHooks[ this.nodeName.toLowerCase() ]; + + // If set returns undefined, fall back to normal setting + if ( !hooks || !( "set" in hooks ) || hooks.set( this, val, "value" ) === undefined ) { + this.value = val; + } + } ); + } +} ); + +jQuery.extend( { + valHooks: { + option: { + get: function( elem ) { + + var val = jQuery.find.attr( elem, "value" ); + return val != null ? + val : + + // Support: IE <=10 - 11 only + // option.text throws exceptions (#14686, #14858) + // Strip and collapse whitespace + // https://html.spec.whatwg.org/#strip-and-collapse-whitespace + stripAndCollapse( jQuery.text( elem ) ); + } + }, + select: { + get: function( elem ) { + var value, option, i, + options = elem.options, + index = elem.selectedIndex, + one = elem.type === "select-one", + values = one ? null : [], + max = one ? index + 1 : options.length; + + if ( index < 0 ) { + i = max; + + } else { + i = one ? index : 0; + } + + // Loop through all the selected options + for ( ; i < max; i++ ) { + option = options[ i ]; + + // Support: IE <=9 only + // IE8-9 doesn't update selected after form reset (#2551) + if ( ( option.selected || i === index ) && + + // Don't return options that are disabled or in a disabled optgroup + !option.disabled && + ( !option.parentNode.disabled || + !nodeName( option.parentNode, "optgroup" ) ) ) { + + // Get the specific value for the option + value = jQuery( option ).val(); + + // We don't need an array for one selects + if ( one ) { + return value; + } + + // Multi-Selects return an array + values.push( value ); + } + } + + return values; + }, + + set: function( elem, value ) { + var optionSet, option, + options = elem.options, + values = jQuery.makeArray( value ), + i = options.length; + + while ( i-- ) { + option = options[ i ]; + + /* eslint-disable no-cond-assign */ + + if ( option.selected = + jQuery.inArray( jQuery.valHooks.option.get( option ), values ) > -1 + ) { + optionSet = true; + } + + /* eslint-enable no-cond-assign */ + } + + // Force browsers to behave consistently when non-matching value is set + if ( !optionSet ) { + elem.selectedIndex = -1; + } + return values; + } + } + } +} ); + +// Radios and checkboxes getter/setter +jQuery.each( [ "radio", "checkbox" ], function() { + jQuery.valHooks[ this ] = { + set: function( elem, value ) { + if ( Array.isArray( value ) ) { + return ( elem.checked = jQuery.inArray( jQuery( elem ).val(), value ) > -1 ); + } + } + }; + if ( !support.checkOn ) { + jQuery.valHooks[ this ].get = function( elem ) { + return elem.getAttribute( "value" ) === null ? "on" : elem.value; + }; + } +} ); + + + + +// Return jQuery for attributes-only inclusion + + +support.focusin = "onfocusin" in window; + + +var rfocusMorph = /^(?:focusinfocus|focusoutblur)$/, + stopPropagationCallback = function( e ) { + e.stopPropagation(); + }; + +jQuery.extend( jQuery.event, { + + trigger: function( event, data, elem, onlyHandlers ) { + + var i, cur, tmp, bubbleType, ontype, handle, special, lastElement, + eventPath = [ elem || document ], + type = hasOwn.call( event, "type" ) ? event.type : event, + namespaces = hasOwn.call( event, "namespace" ) ? event.namespace.split( "." ) : []; + + cur = lastElement = tmp = elem = elem || document; + + // Don't do events on text and comment nodes + if ( elem.nodeType === 3 || elem.nodeType === 8 ) { + return; + } + + // focus/blur morphs to focusin/out; ensure we're not firing them right now + if ( rfocusMorph.test( type + jQuery.event.triggered ) ) { + return; + } + + if ( type.indexOf( "." ) > -1 ) { + + // Namespaced trigger; create a regexp to match event type in handle() + namespaces = type.split( "." ); + type = namespaces.shift(); + namespaces.sort(); + } + ontype = type.indexOf( ":" ) < 0 && "on" + type; + + // Caller can pass in a jQuery.Event object, Object, or just an event type string + event = event[ jQuery.expando ] ? + event : + new jQuery.Event( type, typeof event === "object" && event ); + + // Trigger bitmask: & 1 for native handlers; & 2 for jQuery (always true) + event.isTrigger = onlyHandlers ? 2 : 3; + event.namespace = namespaces.join( "." ); + event.rnamespace = event.namespace ? + new RegExp( "(^|\\.)" + namespaces.join( "\\.(?:.*\\.|)" ) + "(\\.|$)" ) : + null; + + // Clean up the event in case it is being reused + event.result = undefined; + if ( !event.target ) { + event.target = elem; + } + + // Clone any incoming data and prepend the event, creating the handler arg list + data = data == null ? + [ event ] : + jQuery.makeArray( data, [ event ] ); + + // Allow special events to draw outside the lines + special = jQuery.event.special[ type ] || {}; + if ( !onlyHandlers && special.trigger && special.trigger.apply( elem, data ) === false ) { + return; + } + + // Determine event propagation path in advance, per W3C events spec (#9951) + // Bubble up to document, then to window; watch for a global ownerDocument var (#9724) + if ( !onlyHandlers && !special.noBubble && !isWindow( elem ) ) { + + bubbleType = special.delegateType || type; + if ( !rfocusMorph.test( bubbleType + type ) ) { + cur = cur.parentNode; + } + for ( ; cur; cur = cur.parentNode ) { + eventPath.push( cur ); + tmp = cur; + } + + // Only add window if we got to document (e.g., not plain obj or detached DOM) + if ( tmp === ( elem.ownerDocument || document ) ) { + eventPath.push( tmp.defaultView || tmp.parentWindow || window ); + } + } + + // Fire handlers on the event path + i = 0; + while ( ( cur = eventPath[ i++ ] ) && !event.isPropagationStopped() ) { + lastElement = cur; + event.type = i > 1 ? + bubbleType : + special.bindType || type; + + // jQuery handler + handle = ( dataPriv.get( cur, "events" ) || Object.create( null ) )[ event.type ] && + dataPriv.get( cur, "handle" ); + if ( handle ) { + handle.apply( cur, data ); + } + + // Native handler + handle = ontype && cur[ ontype ]; + if ( handle && handle.apply && acceptData( cur ) ) { + event.result = handle.apply( cur, data ); + if ( event.result === false ) { + event.preventDefault(); + } + } + } + event.type = type; + + // If nobody prevented the default action, do it now + if ( !onlyHandlers && !event.isDefaultPrevented() ) { + + if ( ( !special._default || + special._default.apply( eventPath.pop(), data ) === false ) && + acceptData( elem ) ) { + + // Call a native DOM method on the target with the same name as the event. + // Don't do default actions on window, that's where global variables be (#6170) + if ( ontype && isFunction( elem[ type ] ) && !isWindow( elem ) ) { + + // Don't re-trigger an onFOO event when we call its FOO() method + tmp = elem[ ontype ]; + + if ( tmp ) { + elem[ ontype ] = null; + } + + // Prevent re-triggering of the same event, since we already bubbled it above + jQuery.event.triggered = type; + + if ( event.isPropagationStopped() ) { + lastElement.addEventListener( type, stopPropagationCallback ); + } + + elem[ type ](); + + if ( event.isPropagationStopped() ) { + lastElement.removeEventListener( type, stopPropagationCallback ); + } + + jQuery.event.triggered = undefined; + + if ( tmp ) { + elem[ ontype ] = tmp; + } + } + } + } + + return event.result; + }, + + // Piggyback on a donor event to simulate a different one + // Used only for `focus(in | out)` events + simulate: function( type, elem, event ) { + var e = jQuery.extend( + new jQuery.Event(), + event, + { + type: type, + isSimulated: true + } + ); + + jQuery.event.trigger( e, null, elem ); + } + +} ); + +jQuery.fn.extend( { + + trigger: function( type, data ) { + return this.each( function() { + jQuery.event.trigger( type, data, this ); + } ); + }, + triggerHandler: function( type, data ) { + var elem = this[ 0 ]; + if ( elem ) { + return jQuery.event.trigger( type, data, elem, true ); + } + } +} ); + + +// Support: Firefox <=44 +// Firefox doesn't have focus(in | out) events +// Related ticket - https://bugzilla.mozilla.org/show_bug.cgi?id=687787 +// +// Support: Chrome <=48 - 49, Safari <=9.0 - 9.1 +// focus(in | out) events fire after focus & blur events, +// which is spec violation - http://www.w3.org/TR/DOM-Level-3-Events/#events-focusevent-event-order +// Related ticket - https://bugs.chromium.org/p/chromium/issues/detail?id=449857 +if ( !support.focusin ) { + jQuery.each( { focus: "focusin", blur: "focusout" }, function( orig, fix ) { + + // Attach a single capturing handler on the document while someone wants focusin/focusout + var handler = function( event ) { + jQuery.event.simulate( fix, event.target, jQuery.event.fix( event ) ); + }; + + jQuery.event.special[ fix ] = { + setup: function() { + + // Handle: regular nodes (via `this.ownerDocument`), window + // (via `this.document`) & document (via `this`). + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ); + + if ( !attaches ) { + doc.addEventListener( orig, handler, true ); + } + dataPriv.access( doc, fix, ( attaches || 0 ) + 1 ); + }, + teardown: function() { + var doc = this.ownerDocument || this.document || this, + attaches = dataPriv.access( doc, fix ) - 1; + + if ( !attaches ) { + doc.removeEventListener( orig, handler, true ); + dataPriv.remove( doc, fix ); + + } else { + dataPriv.access( doc, fix, attaches ); + } + } + }; + } ); +} +var location = window.location; + +var nonce = { guid: Date.now() }; + +var rquery = ( /\?/ ); + + + +// Cross-browser xml parsing +jQuery.parseXML = function( data ) { + var xml, parserErrorElem; + if ( !data || typeof data !== "string" ) { + return null; + } + + // Support: IE 9 - 11 only + // IE throws on parseFromString with invalid input. + try { + xml = ( new window.DOMParser() ).parseFromString( data, "text/xml" ); + } catch ( e ) {} + + parserErrorElem = xml && xml.getElementsByTagName( "parsererror" )[ 0 ]; + if ( !xml || parserErrorElem ) { + jQuery.error( "Invalid XML: " + ( + parserErrorElem ? + jQuery.map( parserErrorElem.childNodes, function( el ) { + return el.textContent; + } ).join( "\n" ) : + data + ) ); + } + return xml; +}; + + +var + rbracket = /\[\]$/, + rCRLF = /\r?\n/g, + rsubmitterTypes = /^(?:submit|button|image|reset|file)$/i, + rsubmittable = /^(?:input|select|textarea|keygen)/i; + +function buildParams( prefix, obj, traditional, add ) { + var name; + + if ( Array.isArray( obj ) ) { + + // Serialize array item. + jQuery.each( obj, function( i, v ) { + if ( traditional || rbracket.test( prefix ) ) { + + // Treat each array item as a scalar. + add( prefix, v ); + + } else { + + // Item is non-scalar (array or object), encode its numeric index. + buildParams( + prefix + "[" + ( typeof v === "object" && v != null ? i : "" ) + "]", + v, + traditional, + add + ); + } + } ); + + } else if ( !traditional && toType( obj ) === "object" ) { + + // Serialize object item. + for ( name in obj ) { + buildParams( prefix + "[" + name + "]", obj[ name ], traditional, add ); + } + + } else { + + // Serialize scalar item. + add( prefix, obj ); + } +} + +// Serialize an array of form elements or a set of +// key/values into a query string +jQuery.param = function( a, traditional ) { + var prefix, + s = [], + add = function( key, valueOrFunction ) { + + // If value is a function, invoke it and use its return value + var value = isFunction( valueOrFunction ) ? + valueOrFunction() : + valueOrFunction; + + s[ s.length ] = encodeURIComponent( key ) + "=" + + encodeURIComponent( value == null ? "" : value ); + }; + + if ( a == null ) { + return ""; + } + + // If an array was passed in, assume that it is an array of form elements. + if ( Array.isArray( a ) || ( a.jquery && !jQuery.isPlainObject( a ) ) ) { + + // Serialize the form elements + jQuery.each( a, function() { + add( this.name, this.value ); + } ); + + } else { + + // If traditional, encode the "old" way (the way 1.3.2 or older + // did it), otherwise encode params recursively. + for ( prefix in a ) { + buildParams( prefix, a[ prefix ], traditional, add ); + } + } + + // Return the resulting serialization + return s.join( "&" ); +}; + +jQuery.fn.extend( { + serialize: function() { + return jQuery.param( this.serializeArray() ); + }, + serializeArray: function() { + return this.map( function() { + + // Can add propHook for "elements" to filter or add form elements + var elements = jQuery.prop( this, "elements" ); + return elements ? jQuery.makeArray( elements ) : this; + } ).filter( function() { + var type = this.type; + + // Use .is( ":disabled" ) so that fieldset[disabled] works + return this.name && !jQuery( this ).is( ":disabled" ) && + rsubmittable.test( this.nodeName ) && !rsubmitterTypes.test( type ) && + ( this.checked || !rcheckableType.test( type ) ); + } ).map( function( _i, elem ) { + var val = jQuery( this ).val(); + + if ( val == null ) { + return null; + } + + if ( Array.isArray( val ) ) { + return jQuery.map( val, function( val ) { + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ); + } + + return { name: elem.name, value: val.replace( rCRLF, "\r\n" ) }; + } ).get(); + } +} ); + + +var + r20 = /%20/g, + rhash = /#.*$/, + rantiCache = /([?&])_=[^&]*/, + rheaders = /^(.*?):[ \t]*([^\r\n]*)$/mg, + + // #7653, #8125, #8152: local protocol detection + rlocalProtocol = /^(?:about|app|app-storage|.+-extension|file|res|widget):$/, + rnoContent = /^(?:GET|HEAD)$/, + rprotocol = /^\/\//, + + /* Prefilters + * 1) They are useful to introduce custom dataTypes (see ajax/jsonp.js for an example) + * 2) These are called: + * - BEFORE asking for a transport + * - AFTER param serialization (s.data is a string if s.processData is true) + * 3) key is the dataType + * 4) the catchall symbol "*" can be used + * 5) execution will start with transport dataType and THEN continue down to "*" if needed + */ + prefilters = {}, + + /* Transports bindings + * 1) key is the dataType + * 2) the catchall symbol "*" can be used + * 3) selection will start with transport dataType and THEN go to "*" if needed + */ + transports = {}, + + // Avoid comment-prolog char sequence (#10098); must appease lint and evade compression + allTypes = "*/".concat( "*" ), + + // Anchor tag for parsing the document origin + originAnchor = document.createElement( "a" ); + +originAnchor.href = location.href; + +// Base "constructor" for jQuery.ajaxPrefilter and jQuery.ajaxTransport +function addToPrefiltersOrTransports( structure ) { + + // dataTypeExpression is optional and defaults to "*" + return function( dataTypeExpression, func ) { + + if ( typeof dataTypeExpression !== "string" ) { + func = dataTypeExpression; + dataTypeExpression = "*"; + } + + var dataType, + i = 0, + dataTypes = dataTypeExpression.toLowerCase().match( rnothtmlwhite ) || []; + + if ( isFunction( func ) ) { + + // For each dataType in the dataTypeExpression + while ( ( dataType = dataTypes[ i++ ] ) ) { + + // Prepend if requested + if ( dataType[ 0 ] === "+" ) { + dataType = dataType.slice( 1 ) || "*"; + ( structure[ dataType ] = structure[ dataType ] || [] ).unshift( func ); + + // Otherwise append + } else { + ( structure[ dataType ] = structure[ dataType ] || [] ).push( func ); + } + } + } + }; +} + +// Base inspection function for prefilters and transports +function inspectPrefiltersOrTransports( structure, options, originalOptions, jqXHR ) { + + var inspected = {}, + seekingTransport = ( structure === transports ); + + function inspect( dataType ) { + var selected; + inspected[ dataType ] = true; + jQuery.each( structure[ dataType ] || [], function( _, prefilterOrFactory ) { + var dataTypeOrTransport = prefilterOrFactory( options, originalOptions, jqXHR ); + if ( typeof dataTypeOrTransport === "string" && + !seekingTransport && !inspected[ dataTypeOrTransport ] ) { + + options.dataTypes.unshift( dataTypeOrTransport ); + inspect( dataTypeOrTransport ); + return false; + } else if ( seekingTransport ) { + return !( selected = dataTypeOrTransport ); + } + } ); + return selected; + } + + return inspect( options.dataTypes[ 0 ] ) || !inspected[ "*" ] && inspect( "*" ); +} + +// A special extend for ajax options +// that takes "flat" options (not to be deep extended) +// Fixes #9887 +function ajaxExtend( target, src ) { + var key, deep, + flatOptions = jQuery.ajaxSettings.flatOptions || {}; + + for ( key in src ) { + if ( src[ key ] !== undefined ) { + ( flatOptions[ key ] ? target : ( deep || ( deep = {} ) ) )[ key ] = src[ key ]; + } + } + if ( deep ) { + jQuery.extend( true, target, deep ); + } + + return target; +} + +/* Handles responses to an ajax request: + * - finds the right dataType (mediates between content-type and expected dataType) + * - returns the corresponding response + */ +function ajaxHandleResponses( s, jqXHR, responses ) { + + var ct, type, finalDataType, firstDataType, + contents = s.contents, + dataTypes = s.dataTypes; + + // Remove auto dataType and get content-type in the process + while ( dataTypes[ 0 ] === "*" ) { + dataTypes.shift(); + if ( ct === undefined ) { + ct = s.mimeType || jqXHR.getResponseHeader( "Content-Type" ); + } + } + + // Check if we're dealing with a known content-type + if ( ct ) { + for ( type in contents ) { + if ( contents[ type ] && contents[ type ].test( ct ) ) { + dataTypes.unshift( type ); + break; + } + } + } + + // Check to see if we have a response for the expected dataType + if ( dataTypes[ 0 ] in responses ) { + finalDataType = dataTypes[ 0 ]; + } else { + + // Try convertible dataTypes + for ( type in responses ) { + if ( !dataTypes[ 0 ] || s.converters[ type + " " + dataTypes[ 0 ] ] ) { + finalDataType = type; + break; + } + if ( !firstDataType ) { + firstDataType = type; + } + } + + // Or just use first one + finalDataType = finalDataType || firstDataType; + } + + // If we found a dataType + // We add the dataType to the list if needed + // and return the corresponding response + if ( finalDataType ) { + if ( finalDataType !== dataTypes[ 0 ] ) { + dataTypes.unshift( finalDataType ); + } + return responses[ finalDataType ]; + } +} + +/* Chain conversions given the request and the original response + * Also sets the responseXXX fields on the jqXHR instance + */ +function ajaxConvert( s, response, jqXHR, isSuccess ) { + var conv2, current, conv, tmp, prev, + converters = {}, + + // Work with a copy of dataTypes in case we need to modify it for conversion + dataTypes = s.dataTypes.slice(); + + // Create converters map with lowercased keys + if ( dataTypes[ 1 ] ) { + for ( conv in s.converters ) { + converters[ conv.toLowerCase() ] = s.converters[ conv ]; + } + } + + current = dataTypes.shift(); + + // Convert to each sequential dataType + while ( current ) { + + if ( s.responseFields[ current ] ) { + jqXHR[ s.responseFields[ current ] ] = response; + } + + // Apply the dataFilter if provided + if ( !prev && isSuccess && s.dataFilter ) { + response = s.dataFilter( response, s.dataType ); + } + + prev = current; + current = dataTypes.shift(); + + if ( current ) { + + // There's only work to do if current dataType is non-auto + if ( current === "*" ) { + + current = prev; + + // Convert response if prev dataType is non-auto and differs from current + } else if ( prev !== "*" && prev !== current ) { + + // Seek a direct converter + conv = converters[ prev + " " + current ] || converters[ "* " + current ]; + + // If none found, seek a pair + if ( !conv ) { + for ( conv2 in converters ) { + + // If conv2 outputs current + tmp = conv2.split( " " ); + if ( tmp[ 1 ] === current ) { + + // If prev can be converted to accepted input + conv = converters[ prev + " " + tmp[ 0 ] ] || + converters[ "* " + tmp[ 0 ] ]; + if ( conv ) { + + // Condense equivalence converters + if ( conv === true ) { + conv = converters[ conv2 ]; + + // Otherwise, insert the intermediate dataType + } else if ( converters[ conv2 ] !== true ) { + current = tmp[ 0 ]; + dataTypes.unshift( tmp[ 1 ] ); + } + break; + } + } + } + } + + // Apply converter (if not an equivalence) + if ( conv !== true ) { + + // Unless errors are allowed to bubble, catch and return them + if ( conv && s.throws ) { + response = conv( response ); + } else { + try { + response = conv( response ); + } catch ( e ) { + return { + state: "parsererror", + error: conv ? e : "No conversion from " + prev + " to " + current + }; + } + } + } + } + } + } + + return { state: "success", data: response }; +} + +jQuery.extend( { + + // Counter for holding the number of active queries + active: 0, + + // Last-Modified header cache for next request + lastModified: {}, + etag: {}, + + ajaxSettings: { + url: location.href, + type: "GET", + isLocal: rlocalProtocol.test( location.protocol ), + global: true, + processData: true, + async: true, + contentType: "application/x-www-form-urlencoded; charset=UTF-8", + + /* + timeout: 0, + data: null, + dataType: null, + username: null, + password: null, + cache: null, + throws: false, + traditional: false, + headers: {}, + */ + + accepts: { + "*": allTypes, + text: "text/plain", + html: "text/html", + xml: "application/xml, text/xml", + json: "application/json, text/javascript" + }, + + contents: { + xml: /\bxml\b/, + html: /\bhtml/, + json: /\bjson\b/ + }, + + responseFields: { + xml: "responseXML", + text: "responseText", + json: "responseJSON" + }, + + // Data converters + // Keys separate source (or catchall "*") and destination types with a single space + converters: { + + // Convert anything to text + "* text": String, + + // Text to html (true = no transformation) + "text html": true, + + // Evaluate text as a json expression + "text json": JSON.parse, + + // Parse text as xml + "text xml": jQuery.parseXML + }, + + // For options that shouldn't be deep extended: + // you can add your own custom options here if + // and when you create one that shouldn't be + // deep extended (see ajaxExtend) + flatOptions: { + url: true, + context: true + } + }, + + // Creates a full fledged settings object into target + // with both ajaxSettings and settings fields. + // If target is omitted, writes into ajaxSettings. + ajaxSetup: function( target, settings ) { + return settings ? + + // Building a settings object + ajaxExtend( ajaxExtend( target, jQuery.ajaxSettings ), settings ) : + + // Extending ajaxSettings + ajaxExtend( jQuery.ajaxSettings, target ); + }, + + ajaxPrefilter: addToPrefiltersOrTransports( prefilters ), + ajaxTransport: addToPrefiltersOrTransports( transports ), + + // Main method + ajax: function( url, options ) { + + // If url is an object, simulate pre-1.5 signature + if ( typeof url === "object" ) { + options = url; + url = undefined; + } + + // Force options to be an object + options = options || {}; + + var transport, + + // URL without anti-cache param + cacheURL, + + // Response headers + responseHeadersString, + responseHeaders, + + // timeout handle + timeoutTimer, + + // Url cleanup var + urlAnchor, + + // Request state (becomes false upon send and true upon completion) + completed, + + // To know if global events are to be dispatched + fireGlobals, + + // Loop variable + i, + + // uncached part of the url + uncached, + + // Create the final options object + s = jQuery.ajaxSetup( {}, options ), + + // Callbacks context + callbackContext = s.context || s, + + // Context for global events is callbackContext if it is a DOM node or jQuery collection + globalEventContext = s.context && + ( callbackContext.nodeType || callbackContext.jquery ) ? + jQuery( callbackContext ) : + jQuery.event, + + // Deferreds + deferred = jQuery.Deferred(), + completeDeferred = jQuery.Callbacks( "once memory" ), + + // Status-dependent callbacks + statusCode = s.statusCode || {}, + + // Headers (they are sent all at once) + requestHeaders = {}, + requestHeadersNames = {}, + + // Default abort message + strAbort = "canceled", + + // Fake xhr + jqXHR = { + readyState: 0, + + // Builds headers hashtable if needed + getResponseHeader: function( key ) { + var match; + if ( completed ) { + if ( !responseHeaders ) { + responseHeaders = {}; + while ( ( match = rheaders.exec( responseHeadersString ) ) ) { + responseHeaders[ match[ 1 ].toLowerCase() + " " ] = + ( responseHeaders[ match[ 1 ].toLowerCase() + " " ] || [] ) + .concat( match[ 2 ] ); + } + } + match = responseHeaders[ key.toLowerCase() + " " ]; + } + return match == null ? null : match.join( ", " ); + }, + + // Raw string + getAllResponseHeaders: function() { + return completed ? responseHeadersString : null; + }, + + // Caches the header + setRequestHeader: function( name, value ) { + if ( completed == null ) { + name = requestHeadersNames[ name.toLowerCase() ] = + requestHeadersNames[ name.toLowerCase() ] || name; + requestHeaders[ name ] = value; + } + return this; + }, + + // Overrides response content-type header + overrideMimeType: function( type ) { + if ( completed == null ) { + s.mimeType = type; + } + return this; + }, + + // Status-dependent callbacks + statusCode: function( map ) { + var code; + if ( map ) { + if ( completed ) { + + // Execute the appropriate callbacks + jqXHR.always( map[ jqXHR.status ] ); + } else { + + // Lazy-add the new callbacks in a way that preserves old ones + for ( code in map ) { + statusCode[ code ] = [ statusCode[ code ], map[ code ] ]; + } + } + } + return this; + }, + + // Cancel the request + abort: function( statusText ) { + var finalText = statusText || strAbort; + if ( transport ) { + transport.abort( finalText ); + } + done( 0, finalText ); + return this; + } + }; + + // Attach deferreds + deferred.promise( jqXHR ); + + // Add protocol if not provided (prefilters might expect it) + // Handle falsy url in the settings object (#10093: consistency with old signature) + // We also use the url parameter if available + s.url = ( ( url || s.url || location.href ) + "" ) + .replace( rprotocol, location.protocol + "//" ); + + // Alias method option to type as per ticket #12004 + s.type = options.method || options.type || s.method || s.type; + + // Extract dataTypes list + s.dataTypes = ( s.dataType || "*" ).toLowerCase().match( rnothtmlwhite ) || [ "" ]; + + // A cross-domain request is in order when the origin doesn't match the current origin. + if ( s.crossDomain == null ) { + urlAnchor = document.createElement( "a" ); + + // Support: IE <=8 - 11, Edge 12 - 15 + // IE throws exception on accessing the href property if url is malformed, + // e.g. http://example.com:80x/ + try { + urlAnchor.href = s.url; + + // Support: IE <=8 - 11 only + // Anchor's host property isn't correctly set when s.url is relative + urlAnchor.href = urlAnchor.href; + s.crossDomain = originAnchor.protocol + "//" + originAnchor.host !== + urlAnchor.protocol + "//" + urlAnchor.host; + } catch ( e ) { + + // If there is an error parsing the URL, assume it is crossDomain, + // it can be rejected by the transport if it is invalid + s.crossDomain = true; + } + } + + // Convert data if not already a string + if ( s.data && s.processData && typeof s.data !== "string" ) { + s.data = jQuery.param( s.data, s.traditional ); + } + + // Apply prefilters + inspectPrefiltersOrTransports( prefilters, s, options, jqXHR ); + + // If request was aborted inside a prefilter, stop there + if ( completed ) { + return jqXHR; + } + + // We can fire global events as of now if asked to + // Don't fire events if jQuery.event is undefined in an AMD-usage scenario (#15118) + fireGlobals = jQuery.event && s.global; + + // Watch for a new set of requests + if ( fireGlobals && jQuery.active++ === 0 ) { + jQuery.event.trigger( "ajaxStart" ); + } + + // Uppercase the type + s.type = s.type.toUpperCase(); + + // Determine if request has content + s.hasContent = !rnoContent.test( s.type ); + + // Save the URL in case we're toying with the If-Modified-Since + // and/or If-None-Match header later on + // Remove hash to simplify url manipulation + cacheURL = s.url.replace( rhash, "" ); + + // More options handling for requests with no content + if ( !s.hasContent ) { + + // Remember the hash so we can put it back + uncached = s.url.slice( cacheURL.length ); + + // If data is available and should be processed, append data to url + if ( s.data && ( s.processData || typeof s.data === "string" ) ) { + cacheURL += ( rquery.test( cacheURL ) ? "&" : "?" ) + s.data; + + // #9682: remove data so that it's not used in an eventual retry + delete s.data; + } + + // Add or update anti-cache param if needed + if ( s.cache === false ) { + cacheURL = cacheURL.replace( rantiCache, "$1" ); + uncached = ( rquery.test( cacheURL ) ? "&" : "?" ) + "_=" + ( nonce.guid++ ) + + uncached; + } + + // Put hash and anti-cache on the URL that will be requested (gh-1732) + s.url = cacheURL + uncached; + + // Change '%20' to '+' if this is encoded form body content (gh-2658) + } else if ( s.data && s.processData && + ( s.contentType || "" ).indexOf( "application/x-www-form-urlencoded" ) === 0 ) { + s.data = s.data.replace( r20, "+" ); + } + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + if ( jQuery.lastModified[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-Modified-Since", jQuery.lastModified[ cacheURL ] ); + } + if ( jQuery.etag[ cacheURL ] ) { + jqXHR.setRequestHeader( "If-None-Match", jQuery.etag[ cacheURL ] ); + } + } + + // Set the correct header, if data is being sent + if ( s.data && s.hasContent && s.contentType !== false || options.contentType ) { + jqXHR.setRequestHeader( "Content-Type", s.contentType ); + } + + // Set the Accepts header for the server, depending on the dataType + jqXHR.setRequestHeader( + "Accept", + s.dataTypes[ 0 ] && s.accepts[ s.dataTypes[ 0 ] ] ? + s.accepts[ s.dataTypes[ 0 ] ] + + ( s.dataTypes[ 0 ] !== "*" ? ", " + allTypes + "; q=0.01" : "" ) : + s.accepts[ "*" ] + ); + + // Check for headers option + for ( i in s.headers ) { + jqXHR.setRequestHeader( i, s.headers[ i ] ); + } + + // Allow custom headers/mimetypes and early abort + if ( s.beforeSend && + ( s.beforeSend.call( callbackContext, jqXHR, s ) === false || completed ) ) { + + // Abort if not done already and return + return jqXHR.abort(); + } + + // Aborting is no longer a cancellation + strAbort = "abort"; + + // Install callbacks on deferreds + completeDeferred.add( s.complete ); + jqXHR.done( s.success ); + jqXHR.fail( s.error ); + + // Get transport + transport = inspectPrefiltersOrTransports( transports, s, options, jqXHR ); + + // If no transport, we auto-abort + if ( !transport ) { + done( -1, "No Transport" ); + } else { + jqXHR.readyState = 1; + + // Send global event + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxSend", [ jqXHR, s ] ); + } + + // If request was aborted inside ajaxSend, stop there + if ( completed ) { + return jqXHR; + } + + // Timeout + if ( s.async && s.timeout > 0 ) { + timeoutTimer = window.setTimeout( function() { + jqXHR.abort( "timeout" ); + }, s.timeout ); + } + + try { + completed = false; + transport.send( requestHeaders, done ); + } catch ( e ) { + + // Rethrow post-completion exceptions + if ( completed ) { + throw e; + } + + // Propagate others as results + done( -1, e ); + } + } + + // Callback for when everything is done + function done( status, nativeStatusText, responses, headers ) { + var isSuccess, success, error, response, modified, + statusText = nativeStatusText; + + // Ignore repeat invocations + if ( completed ) { + return; + } + + completed = true; + + // Clear timeout if it exists + if ( timeoutTimer ) { + window.clearTimeout( timeoutTimer ); + } + + // Dereference transport for early garbage collection + // (no matter how long the jqXHR object will be used) + transport = undefined; + + // Cache response headers + responseHeadersString = headers || ""; + + // Set readyState + jqXHR.readyState = status > 0 ? 4 : 0; + + // Determine if successful + isSuccess = status >= 200 && status < 300 || status === 304; + + // Get response data + if ( responses ) { + response = ajaxHandleResponses( s, jqXHR, responses ); + } + + // Use a noop converter for missing script but not if jsonp + if ( !isSuccess && + jQuery.inArray( "script", s.dataTypes ) > -1 && + jQuery.inArray( "json", s.dataTypes ) < 0 ) { + s.converters[ "text script" ] = function() {}; + } + + // Convert no matter what (that way responseXXX fields are always set) + response = ajaxConvert( s, response, jqXHR, isSuccess ); + + // If successful, handle type chaining + if ( isSuccess ) { + + // Set the If-Modified-Since and/or If-None-Match header, if in ifModified mode. + if ( s.ifModified ) { + modified = jqXHR.getResponseHeader( "Last-Modified" ); + if ( modified ) { + jQuery.lastModified[ cacheURL ] = modified; + } + modified = jqXHR.getResponseHeader( "etag" ); + if ( modified ) { + jQuery.etag[ cacheURL ] = modified; + } + } + + // if no content + if ( status === 204 || s.type === "HEAD" ) { + statusText = "nocontent"; + + // if not modified + } else if ( status === 304 ) { + statusText = "notmodified"; + + // If we have data, let's convert it + } else { + statusText = response.state; + success = response.data; + error = response.error; + isSuccess = !error; + } + } else { + + // Extract error from statusText and normalize for non-aborts + error = statusText; + if ( status || !statusText ) { + statusText = "error"; + if ( status < 0 ) { + status = 0; + } + } + } + + // Set data for the fake xhr object + jqXHR.status = status; + jqXHR.statusText = ( nativeStatusText || statusText ) + ""; + + // Success/Error + if ( isSuccess ) { + deferred.resolveWith( callbackContext, [ success, statusText, jqXHR ] ); + } else { + deferred.rejectWith( callbackContext, [ jqXHR, statusText, error ] ); + } + + // Status-dependent callbacks + jqXHR.statusCode( statusCode ); + statusCode = undefined; + + if ( fireGlobals ) { + globalEventContext.trigger( isSuccess ? "ajaxSuccess" : "ajaxError", + [ jqXHR, s, isSuccess ? success : error ] ); + } + + // Complete + completeDeferred.fireWith( callbackContext, [ jqXHR, statusText ] ); + + if ( fireGlobals ) { + globalEventContext.trigger( "ajaxComplete", [ jqXHR, s ] ); + + // Handle the global AJAX counter + if ( !( --jQuery.active ) ) { + jQuery.event.trigger( "ajaxStop" ); + } + } + } + + return jqXHR; + }, + + getJSON: function( url, data, callback ) { + return jQuery.get( url, data, callback, "json" ); + }, + + getScript: function( url, callback ) { + return jQuery.get( url, undefined, callback, "script" ); + } +} ); + +jQuery.each( [ "get", "post" ], function( _i, method ) { + jQuery[ method ] = function( url, data, callback, type ) { + + // Shift arguments if data argument was omitted + if ( isFunction( data ) ) { + type = type || callback; + callback = data; + data = undefined; + } + + // The url can be an options object (which then must have .url) + return jQuery.ajax( jQuery.extend( { + url: url, + type: method, + dataType: type, + data: data, + success: callback + }, jQuery.isPlainObject( url ) && url ) ); + }; +} ); + +jQuery.ajaxPrefilter( function( s ) { + var i; + for ( i in s.headers ) { + if ( i.toLowerCase() === "content-type" ) { + s.contentType = s.headers[ i ] || ""; + } + } +} ); + + +jQuery._evalUrl = function( url, options, doc ) { + return jQuery.ajax( { + url: url, + + // Make this explicit, since user can override this through ajaxSetup (#11264) + type: "GET", + dataType: "script", + cache: true, + async: false, + global: false, + + // Only evaluate the response if it is successful (gh-4126) + // dataFilter is not invoked for failure responses, so using it instead + // of the default converter is kludgy but it works. + converters: { + "text script": function() {} + }, + dataFilter: function( response ) { + jQuery.globalEval( response, options, doc ); + } + } ); +}; + + +jQuery.fn.extend( { + wrapAll: function( html ) { + var wrap; + + if ( this[ 0 ] ) { + if ( isFunction( html ) ) { + html = html.call( this[ 0 ] ); + } + + // The elements to wrap the target around + wrap = jQuery( html, this[ 0 ].ownerDocument ).eq( 0 ).clone( true ); + + if ( this[ 0 ].parentNode ) { + wrap.insertBefore( this[ 0 ] ); + } + + wrap.map( function() { + var elem = this; + + while ( elem.firstElementChild ) { + elem = elem.firstElementChild; + } + + return elem; + } ).append( this ); + } + + return this; + }, + + wrapInner: function( html ) { + if ( isFunction( html ) ) { + return this.each( function( i ) { + jQuery( this ).wrapInner( html.call( this, i ) ); + } ); + } + + return this.each( function() { + var self = jQuery( this ), + contents = self.contents(); + + if ( contents.length ) { + contents.wrapAll( html ); + + } else { + self.append( html ); + } + } ); + }, + + wrap: function( html ) { + var htmlIsFunction = isFunction( html ); + + return this.each( function( i ) { + jQuery( this ).wrapAll( htmlIsFunction ? html.call( this, i ) : html ); + } ); + }, + + unwrap: function( selector ) { + this.parent( selector ).not( "body" ).each( function() { + jQuery( this ).replaceWith( this.childNodes ); + } ); + return this; + } +} ); + + +jQuery.expr.pseudos.hidden = function( elem ) { + return !jQuery.expr.pseudos.visible( elem ); +}; +jQuery.expr.pseudos.visible = function( elem ) { + return !!( elem.offsetWidth || elem.offsetHeight || elem.getClientRects().length ); +}; + + + + +jQuery.ajaxSettings.xhr = function() { + try { + return new window.XMLHttpRequest(); + } catch ( e ) {} +}; + +var xhrSuccessStatus = { + + // File protocol always yields status code 0, assume 200 + 0: 200, + + // Support: IE <=9 only + // #1450: sometimes IE returns 1223 when it should be 204 + 1223: 204 + }, + xhrSupported = jQuery.ajaxSettings.xhr(); + +support.cors = !!xhrSupported && ( "withCredentials" in xhrSupported ); +support.ajax = xhrSupported = !!xhrSupported; + +jQuery.ajaxTransport( function( options ) { + var callback, errorCallback; + + // Cross domain only allowed if supported through XMLHttpRequest + if ( support.cors || xhrSupported && !options.crossDomain ) { + return { + send: function( headers, complete ) { + var i, + xhr = options.xhr(); + + xhr.open( + options.type, + options.url, + options.async, + options.username, + options.password + ); + + // Apply custom fields if provided + if ( options.xhrFields ) { + for ( i in options.xhrFields ) { + xhr[ i ] = options.xhrFields[ i ]; + } + } + + // Override mime type if needed + if ( options.mimeType && xhr.overrideMimeType ) { + xhr.overrideMimeType( options.mimeType ); + } + + // X-Requested-With header + // For cross-domain requests, seeing as conditions for a preflight are + // akin to a jigsaw puzzle, we simply never set it to be sure. + // (it can always be set on a per-request basis or even using ajaxSetup) + // For same-domain requests, won't change header if already provided. + if ( !options.crossDomain && !headers[ "X-Requested-With" ] ) { + headers[ "X-Requested-With" ] = "XMLHttpRequest"; + } + + // Set headers + for ( i in headers ) { + xhr.setRequestHeader( i, headers[ i ] ); + } + + // Callback + callback = function( type ) { + return function() { + if ( callback ) { + callback = errorCallback = xhr.onload = + xhr.onerror = xhr.onabort = xhr.ontimeout = + xhr.onreadystatechange = null; + + if ( type === "abort" ) { + xhr.abort(); + } else if ( type === "error" ) { + + // Support: IE <=9 only + // On a manual native abort, IE9 throws + // errors on any property access that is not readyState + if ( typeof xhr.status !== "number" ) { + complete( 0, "error" ); + } else { + complete( + + // File: protocol always yields status 0; see #8605, #14207 + xhr.status, + xhr.statusText + ); + } + } else { + complete( + xhrSuccessStatus[ xhr.status ] || xhr.status, + xhr.statusText, + + // Support: IE <=9 only + // IE9 has no XHR2 but throws on binary (trac-11426) + // For XHR2 non-text, let the caller handle it (gh-2498) + ( xhr.responseType || "text" ) !== "text" || + typeof xhr.responseText !== "string" ? + { binary: xhr.response } : + { text: xhr.responseText }, + xhr.getAllResponseHeaders() + ); + } + } + }; + }; + + // Listen to events + xhr.onload = callback(); + errorCallback = xhr.onerror = xhr.ontimeout = callback( "error" ); + + // Support: IE 9 only + // Use onreadystatechange to replace onabort + // to handle uncaught aborts + if ( xhr.onabort !== undefined ) { + xhr.onabort = errorCallback; + } else { + xhr.onreadystatechange = function() { + + // Check readyState before timeout as it changes + if ( xhr.readyState === 4 ) { + + // Allow onerror to be called first, + // but that will not handle a native abort + // Also, save errorCallback to a variable + // as xhr.onerror cannot be accessed + window.setTimeout( function() { + if ( callback ) { + errorCallback(); + } + } ); + } + }; + } + + // Create the abort callback + callback = callback( "abort" ); + + try { + + // Do send the request (this may raise an exception) + xhr.send( options.hasContent && options.data || null ); + } catch ( e ) { + + // #14683: Only rethrow if this hasn't been notified as an error yet + if ( callback ) { + throw e; + } + } + }, + + abort: function() { + if ( callback ) { + callback(); + } + } + }; + } +} ); + + + + +// Prevent auto-execution of scripts when no explicit dataType was provided (See gh-2432) +jQuery.ajaxPrefilter( function( s ) { + if ( s.crossDomain ) { + s.contents.script = false; + } +} ); + +// Install script dataType +jQuery.ajaxSetup( { + accepts: { + script: "text/javascript, application/javascript, " + + "application/ecmascript, application/x-ecmascript" + }, + contents: { + script: /\b(?:java|ecma)script\b/ + }, + converters: { + "text script": function( text ) { + jQuery.globalEval( text ); + return text; + } + } +} ); + +// Handle cache's special case and crossDomain +jQuery.ajaxPrefilter( "script", function( s ) { + if ( s.cache === undefined ) { + s.cache = false; + } + if ( s.crossDomain ) { + s.type = "GET"; + } +} ); + +// Bind script tag hack transport +jQuery.ajaxTransport( "script", function( s ) { + + // This transport only deals with cross domain or forced-by-attrs requests + if ( s.crossDomain || s.scriptAttrs ) { + var script, callback; + return { + send: function( _, complete ) { + script = jQuery( " + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Accounts

+

The accounts section is for the management of user accounts, access permissions, and tenants.

+
+

Users

+

Description of User account fields:

+
    +
  • Username - Unique username.

  • +
  • Full Name - Full Name of the user.

  • +
  • E-mail - The email address of the user. Also used for system notifications and for password retrieval.

  • +
  • E-mail CC - Send copies of email notifications to this address.

  • +
  • Phone Number - User’s phone number.

  • +
  • Company - Company the user works for. Usually useful for multi-tenant systems where the company provides Netris Controller access to customers.

  • +
  • Position - Position within the company.

  • +
  • User Role - When using a User Role object to define RBAC (role-based access control), Permissions Group and Tenant fields will deactivate.

  • +
  • Permission Group - User permissions for viewing and editing parts of the Netris Controller. (if User Role is not used)

  • +
  • +Tenant - User permissions for viewing and editing services using Switch Port and IP resources assigned to various Tenants. (if User Role is not used)

  • +
+

Example: Creating a user with full access to all sections of Netris Controller, read-only access to resources managed by any Tenant, and full access to resources assigned to the Tenant Admin.

+User Management +

Password: To set a password or email the user for a password form, go to the listing of usernames and click the menu on the right side.

+List User Accounts +
+
+

Tenants

+

IP addresses and Network Interfaces are resources that can be assigned to different Tenants for their management. Admin is the default tenant, and by default, it owns all the resources. The concept of Tenants can be used for sharing and delegation of control over the network resources, typically used by network teams to grant access to other teams for requesting & managing network services using the Netris Controller as a self service portal or programmatically (with Kubernetes CRDs or Terraform) as part of DevOps/NetOps pipeline.

+

A Tenant has just two fields, the unique name and custom description.

+

Example: Adding a tenant.

+Adding Tenants +
+
+

Permission Groups

+

Permission Groups are a list of permissions on a per section basis that can be attached individually to a User or a User Role. Every section has a View and Edit attribute. The view defines if users with this Permission Group can see the particular section at all. Edit defines if users with this Permission Group can edit services and policies in specific sections.

+

Example: Permission Group.

+Managing Permissions +
+
+

User Roles

+

Permission Groups and Tenants can be either linked directly to an individual username or can be linked to a User Role object which then can be linked to an individual username.

+User Roles +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/acls.html b/en/latest/acls.html new file mode 100644 index 0000000000..234e1aa823 --- /dev/null +++ b/en/latest/acls.html @@ -0,0 +1,928 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Access Control Lists (ACL) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Access Control Lists (ACL)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Access Control Lists (ACL)

+

Netris supports ACLs for switch network access control. (ACL and ACL2.0) ACL is for defining network access lists in a source IP: Port, destination IP: Port format. ACL2.0 is an object-oriented service way of describing network access.

+

Both ACL and ACL2.0 services support tenant/RBAC based approval workflows. Access control lists execute in switch hardware providing line-rate performance for security enforcement. It’s important to keep in mind that the number of ACLs is limited to the limited size of TCAM of network switches.

+

Screenshot: TCAM utilization can be seen under Net→Inventory

+_images/TCAM.png +

Netris is applying several optimization algorithms to minimize the usage of TCAM while achieving the user-defined requirements.

+
+

ACL Default Policy

+

The ACL default policy is to permit all hosts to communicate with each other. You can change the default policy on a per Site basis by editing the Site features under Net→Sites. Once the “ACL Default Policy” is changed to “Deny,” the given site will start dropping any traffic unless specific communication is permitted through ACL or ACL2.0 rules.

+

Example: Changing “ACL Default Policy” for the site “siteDefault”.

+_images/siteDefault.png +
+
+

ACL Rules

+

ACL rules can be created, listed, edited, approved under Services→ACL.

+

Description of ACL fields. +General

+
    +
  • Name - Unique name for the ACL entry.

  • +
  • Protocol - IP protocol to match.

    +
      +
    • All - Any IP protocols.

    • +
    • IP - Specific IP protocol number.

    • +
    • TCP - TCP.

    • +
    • UDP - UDP.

    • +
    • ICMP ALL - Any IPv4 ICMP protocol.

    • +
    • ICMP Custom - Custom IPv4 ICMP code.

    • +
    • ICMPv6 ALL - Any IPv6 ICMP protocol.

    • +
    • ICMPv6 Custom - Custom IPv6 ICMP code.

    • +
    +
  • +
  • Active Until - Disable this rule at the defined date/time.

  • +
  • Action - Permit or Deny forwarding of matched packets.

  • +
  • Established/Reverse - For TCP, also match reverse packets except with TCP SYN flag. For non-TCP, also generate a reverse rule with swapped source/destination.

  • +
+

Source/Destination - Source and destination addresses and ports to match.

+
    +
  • Source IPv4/IPv6 - IPv4/IPv6 address.

  • +
  • Ports Type

    +
      +
    • Port Range - Match on the port or a port range defined in this window.

    • +
    • Port Group - Match on a group of ports defined under Services→ ACL Port Group.

    • +
    +
  • +
  • From Port - Port range starting from.

  • +
  • To Port - Port range ending with.

  • +
  • Comment - Descriptive comment, commonly used for approval workflows.

  • +
  • Check button - Check if Another ACL on the system already permits the described network access.

  • +
+

Example: Permit hosts in 10.0.3.0/24 to access hosts in 10.0.5.0/24 by SSH, also permit the return traffic (Established).

+_images/action_permit.png +

Example: “Check” shows that requested access is already provided by a broader ACL rule.

+_images/ACL_rule.png +
+
+

ACL Approval Workflow

+

When one tenant (one team) needs to get network access to resources under the responsibility of another tenant (another team), an ACL can be created but will activate only after approval of the tenant responsible for the destination address resources. See the below example.

+

Example: User representing QA_tenant is creating an ACL where source belongs to QA_tenant, but destination belongs to the Admin tenant.

+_images/ACL_approval.png +

Screenshot: ACL stays in “waiting for approval” state until approved.

+_images/waiting_approval.png +

Screenshot: Users of tenant Admin, receive a notification in the GUI, and optionally by email. Then one can review the access request and either approve or reject it.

+_images/approve_reject.png +

Screenshot: Once approved, users of both tenants will see the ACL in the “Active” state, and soon Netris Agents will push the appropriate config throughout the switch fabric.

+_images/ACL_active.png +
+
+

ACL Processing Order

+
    +
  1. User-defined Deny Rules

  2. +
  3. User-defined Permit Rules

  4. +
  5. Deny the rest

  6. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/controller-k3s-installation.html b/en/latest/controller-k3s-installation.html new file mode 100644 index 0000000000..dd5591e3e6 --- /dev/null +++ b/en/latest/controller-k3s-installation.html @@ -0,0 +1,1027 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Netris Controller installation on a generic Linux host — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Controller installation on a generic Linux host

+
+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+
+

Note

+

K3s is expected to work on most modern Linux systems.

+

Some OSs have specific requirements:

+
    +
  • If you are using Raspbian Buster, follow these steps to switch to legacy iptables.

  • +
  • If you are using Alpine Linux, follow these steps for additional setup.

  • +
  • If you are using (Red Hat/CentOS) Enterprise Linux, follow these steps for additional setup.

  • +
+
+
+
+

Installation

+

The following command will install the Netris Controller on your Linux server:

+
curl -sfL https://get.netris.io | sh -
+
+
+

Once installed, you will be able to log in to Netris Controller using your host’s IP address.

+
+

Note

+

The installation script does the following:

+ +
+
+

Installation with the specific host name

+

In order to set the specific ingress host name to the Netris Controller, use the --ctl-hostname installation argument:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com
+
+
+

A self-signed SSL certificate will be generated from that host name.

+
+
+

Installation with the Let’s Encrypt SSL

+

The installation script supports Let’s Encrypt SSL generation out-of-box. To instruct the installation script to do that use --ctl-ssl-issuer argument.

+
+

Note

+
+
The argument --ctl-ssl-issuer is passing cert-manager.io/cluster-issuer value to the ingress resource of the Netris Controller. The installation script can create two types of ClusterIssuer resource: selfsigned or letsencrypt, where selfsigned is just Cert-Manager self-signed SSL and the letsencrypt is the ACME issuer with HTTP01 challenge validation.
+
If the --ctl-ssl-issuer argument is not set, the installation script will proceed with selfsigned ClusterIssuer type.
+
+
+

Run the following command to install Netris Controller and use letsencrypt ClusterIssuer for SSL generation:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

To successfully validate and complete Let’s Encrypt SSL generation, a valid A/CNAME record for the domain/subdomain name should exist prior, and that name must be accessible from the Internet.

+
+
+
+

Installation with the Custom SSL Issuer

+

The HTTP01 challenge validation is the simplest way of issuing the Let’s Encrypt SSL, but it does not work when the host behind the FQDN is not accessible from the public internet. +The common approach of validating and completing Let’s Encrypt SSL generation for private deployments is DNS01 challenge validation. +If the DNS01 does not work for you either, Cert-Manager supports a number of certificate issuers, get familiar with all types of issuers here.

+

In order to install Netris Controller with the custom SSL issuer, you need to run installation script with the specified host name:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com
+
+
+

Once the installation is complete, create a yaml file with the ClusterIssuer resource, suitable for your requirements, and apply it:

+
kubectl apply -f my-cluster-issuer.yaml
+
+
+

Then rerun the installation script with the --ctl-ssl-issuer argument:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-ssl-issuer <Your ClusterIssuer resource name>
+
+
+
+
+
+

Upgrading

+

To upgrade the Netris Controller to the latest version simply run the script:

+
curl -sfL https://get.netris.io | sh -
+
+
+

If a newer version of Netris Controller is available, it will be updated in a few minutes.

+
+
+

Uninstalling

+

To uninstall Netris Controller and K3s from a server node, run:

+
/usr/local/bin/k3s-uninstall.sh
+
+
+
+
+

Backup and Restore

+

Netris Controller stores all critical data in MariaDB. It’s highly recommended to create a cronjob with mysqldump.

+
+

Backup

+

To take database snapshot run the following command:

+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot-$(date +%Y-%m-%d-%H-%M-%S).sql
+
+
+

After command execution, you can find db-snapshot-YYYY-MM-DD-HH-MM-SS.sql file in the current working directory.

+
+

Backup the Secret Key

+

Netris Controller generates a unique secret key at the first installation. If you’re moving or reinstalling your controller, it makes sense to take note of the secret key for restoring purpose in the future. Overwise, you have to reinitiate all devices connected to the controller.

+
kubectl -n netris-controller get secret netris-controller-grpc-secret -o jsonpath='{.data.secret-key}{"\n"}'
+
+
+
+
+
+

Restore

+

In order to restore DB from a database snapshot, follow these steps:

+
    +
  1. Drop the current database by running the following command:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"'
+
+
+
    +
  1. Create a new database:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"'
+
+
+
    +
  1. Copy snapshot file to the MariaDB container:

  2. +
+
kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql
+
+
+
    +
  1. Run the restore command:

  2. +
+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql'
+
+
+
+

Note

+

In this example the snapshot file name is db-snapshot.sql and it’s located in the current working directory

+
+
+

Restore the Secret Key

+

If you want to restore the controller secret key too (you might want to do that if you’re reinstalling or moving the controller to the other place), follow these steps:

+
    +
  1. Set OLD_SECRET environment variable (the secret key taken from the old controller):

  2. +
+
export OLD_SECRET=<Your old secret key>
+
+
+

example: export OLD_SECRET=VUdodFFSakJCU2lFVVA4T1c0cnpuUmdiMkQxem85Y2dnS3pkajlNSg==

+
    +
  1. Update the secret key of the new controller:

  2. +
+
kubectl -n netris-controller patch secret netris-controller-grpc-secret --type='json' -p='[{"op" : "replace" ,"path" : "/data/secret-key" ,"value" : "'$OLD_SECRET'"}]'
+
+
+
    +
  1. Restart Netris Controller’s all microservices

  2. +
+
kubectl -n netris-controller rollout restart deployments
+
+
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/controller-k8s-installation.html b/en/latest/controller-k8s-installation.html new file mode 100644 index 0000000000..ceb4303a7e --- /dev/null +++ b/en/latest/controller-k8s-installation.html @@ -0,0 +1,902 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Helm Chart Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Helm Chart Installation

+
+

Requirements

+
    +
  • Kubernetes 1.12+

  • +
  • Helm 3.1+

  • +
  • PV provisioner support in the underlying infrastructure

  • +
+
+
+

Get Repo Info

+

Add the Netris Helm repository:

+
helm repo add netrisai https://netrisai.github.io/charts
+helm repo update
+
+
+
+
+

Installing the Chart

+

In order to install the Helm chart, you must follow these steps:

+
    +
  1. Create the namespace for netris-controller:

  2. +
+
kubectl create namespace netris-controller
+
+
+
    +
  1. Install helm chart with netris-controller:

  2. +
+
helm install netris-controller netrisai/netris-controller \
+  --namespace netris-controller \
+  --set ingress.hosts={my.domain.com}
+
+
+
+
+

Uninstalling the Chart

+

To uninstall/delete the netris-controller helm release:

+
helm uninstall netris-controller
+
+
+
+
+

Chart Configuration

+

See the netris-controller README for details about configurable parameters and their default values.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/controller-k8s-quickstart.html b/en/latest/controller-k8s-quickstart.html new file mode 100644 index 0000000000..3cda90600a --- /dev/null +++ b/en/latest/controller-k8s-quickstart.html @@ -0,0 +1,870 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Quickstart Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Quickstart Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Quickstart Installation

+

Netris offers a simplified deployment model for users who want to quickly install the Netris Controller in the shortest amount of time.

+

This installation process is streamlined for Linux servers that do not already have Kubernetes running. The install does the following:

+ +

If you wish to install the controller on an existing Kubernetes cluster, follow these instructions instead of this Quickstart.

+
+

Quickstart Process

+
    +
  1. Install the Netris Controller w/ k3s by running the following command on your Linux server:

  2. +
+
curl -sfL https://get.netris.io | sh -
+
+
+
    +
  1. When the installation completes, you will be provided with the login for the web UI. Login with the provided credentials.

  2. +
  3. Navigate in the UI to Net→IPAM and add a new subnet that contains the desired management IP addresses you wish to use for your SoftGates and switches.

    +

    For example, if you are planning on using 192.168.1.100 as the IP address of your Ubuntu server, then create a subnet in Netris UI for 192.168.1.0/24.

    +

    Detailed configuration documentation is available here: Netris IPAM.

    +
  4. +
  5. Navigate in the UI to Topology

  6. +
  7. Click the Add in the upper right

  8. +
  9. Fill out the fields for the SoftGate you wish to add

  10. +
  11. Select the proper Management IP address from the subnet selector

  12. +
  13. Once the SoftGate is created in the Topology, right-click on the SoftGate and select the Install Agent option

  14. +
  15. Copy the agent install command to your clipboard and run the command on the Ubuntu server you are using as your SoftGate

  16. +
  17. Congratulations. The SoftGate should now be connected to your controller.

  18. +
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/controller-vm-installation.html b/en/latest/controller-vm-installation.html new file mode 100644 index 0000000000..5d278ccf3e --- /dev/null +++ b/en/latest/controller-vm-installation.html @@ -0,0 +1,1005 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Virtual Machine Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Virtual Machine Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Virtual Machine Installation

+
+

Requirements

+

Minimal system requirements for the VM:

+
    +
  • CPU - 4 Core

  • +
  • RAM - 4 Gb

  • +
  • Disk - 100Gb

  • +
  • Network - 1 virtual NIC

  • +
+

Recommended system requirements for the VM:

+
    +
  • CPU - 8 Core

  • +
  • RAM - 16 Gb

  • +
  • Disk - 100Gb

  • +
  • Network - 1 virtual NIC

  • +
+
+
+

KVM Hypervisor Installation

+

If KVM is not already installed, install Qemu/KVM on the host machine (example provided for Ubuntu Linux 18.04)

+
sudo apt-get install virt-manager
+
+
+
+
+

VM Controller Installation

+
    +
  1. Download the Netris Controller image. (contact Netris support for repository access permissions).

  2. +
+
cd /var/lib/libvirt/images
+
+sudo wget http://img.netris.io/netris-controller3.qcow2
+
+
+
    +
  1. Download VM definition file.

  2. +
+
cd /etc/libvirt/qemu
+
+sudo wget http://img.netris.io/netris-controller3.xml
+
+
+
    +
  1. Define the KVM virtual machine

  2. +
+
sudo virsh define netris-controller3.xml
+
+
+
+

Note

+

Netris Controller virtual NIC will bind to the “br-mgmt” interface on the KVM host machine. See below for the network interface configuration example.

+
+

Example: Network configuration on host (hypervisor) machine.

+
+

Note

+

replace <Physical NIC>, <host server management IP/prefix length> and <host server default gateway> +with the correct NIC and IP for your host machine.

+
+
sudo vim /etc/network/interfaces
+
+
+
#Physical NIC connected to the management network
+auto <Physical NIC>
+iface <Physical NIC> inet static
+                        address 0.0.0.0/0
+
+#bridge interface
+auto br-mgmt
+iface br-mgmt inet static
+                        address <host server management IP/prefix length>
+                        gateway <host server default gateway>
+                        bridge-ports <Physical NIC>
+
+source /etc/network/interfaces.d/*
+
+
+
sudo ifreload -a
+
+
+
    +
  1. Set the virtual machine to autostart and start it.

  2. +
+
sudo virsh autostart netris-controller
+
+
+
sudo virsh start netris-controller
+
+
+
+
+

Accessing the Netris Controller

+

By default, Netris Controller will obtain an IP address from a DHCP server.

+

Below steps describe how to configure a Static IP address for the Netris Controller.

+
    +
  1. Connecting to the VM console.

  2. +
+

default credentials. login: netris password: newNet0ps

+
sudo virsh console netris-controller
+
+
+
+

Note

+

Do not forget to change the default password (using passwd command).

+
+
    +
  1. Setting a static IP address.

  2. +
+

Edit network configuration file.

+
sudo vim /etc/network/interfaces
+
+
+

Example: IP configuration file.

+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+
+# The primary network interface
+auto eth0
+iface eth0 inet static
+        address <Netris Controller IP/prefix length>
+        gateway <Netris Controller default gateway>
+        dns-nameserver <a DNS server address>
+
+source /etc/network/interfaces.d/*
+
+
+

Reload the network config.

+
sudo ifreload -a
+
+
+
+

Note

+

Make sure Netris Controller has Internet access.

+
+
    +
  1. Reboot the controller

  2. +
+
sudo reboot
+
+
+

After reboot, the Netris Controller GUI should be accessible using a browser. Use netris/newNet0ps credentials.

+Netris Credentials +
+
+

Replacing the SSL certificate

+
    +
  1. Replace the below file with your SSL certificate file.

  2. +
+
/etc/nginx/ssl/controller.cert.pem;
+
+
+
    +
  1. Replace the below file with your SSL private key.

  2. +
+
/etc/nginx/ssl/controller.key.pem;
+
+
+
    +
  1. Restart Nginx service.

  2. +
+
systemctl restart nginx.service
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/definitions.html b/en/latest/definitions.html new file mode 100644 index 0000000000..9b4317a4d6 --- /dev/null +++ b/en/latest/definitions.html @@ -0,0 +1,868 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Definitions — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Definitions

+

When configuring and operating a Netris system, the following nomenclature is important to understand:

+
    +
  • User - A user account for accessing Netris Controller through GUI, RestAPI, and Kubernetes. The default username is netris, with password newNet0ps.

  • +
  • The Netris VPC - logically segregated virtual network.The VPC acts as a VRF in traditional networking, providing the flexibility to employ overlapping IP ranges across various VPCs while maintaining secure management and operation of resources.

  • +
  • Tenant - IP addresses and Switch Ports are network resources assigned to different Tenants to have under their management. Admin is the default tenant, and by default, it owns all the resources. You can use different Tenants for sharing and delegation of control over the network resources. Network teams typically use Tenants to grant access to other groups to request and manage network services using the Netris Controller as a self-service portal or programmatically (with Kubernetes CRDs or Terraform) via a DevOps/NetOps pipeline.

  • +
  • Permission Group - List of permissions on a per section basis can be attached individually to a User or a User Role.

  • +
  • User Role - Group of user permissions and tenants for role-based access control (RBAC).

  • +
  • Site - Each separate deployment (each data center) should be defined as a Site. All network units and resources are attached to a site. Site entry defines global attributes such as; AS numbers, default ACL policy, Site Mesh (site to site VPN), and other site-level parameters.

  • +
  • Subnet - IPv4/IPv6 address resources linked to Sites and Tenants

  • +
  • Switch Port - Physical ports of all switches attached to the system, or server endpoints in a Bare Metal Cloud environment.

  • +
  • Inventory - Inventory of all network units that are operated using Netris Agent.

  • +
  • E-BGP - Defines all External BGP peers (iBGP and eBGP).

  • +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/genindex.html b/en/latest/genindex.html new file mode 100644 index 0000000000..76288c3aff --- /dev/null +++ b/en/latest/genindex.html @@ -0,0 +1,842 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Index — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ + +

Index

+ +
+ +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/index.html b/en/latest/index.html new file mode 100644 index 0000000000..ca63c0a297 --- /dev/null +++ b/en/latest/index.html @@ -0,0 +1,1207 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Documentation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Documentation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Welcome to Netris Documentation

+

Learn how to get started with Netris VPC Networking for your network environment.

+

You are welcome to join our Slack channel to get additional support from our engineers and community.

+ + +
+

Tutorials

+ +
+ + + + + + + +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/installation.html b/en/latest/installation.html new file mode 100644 index 0000000000..f55e98ab98 --- /dev/null +++ b/en/latest/installation.html @@ -0,0 +1,888 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Controller Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Controller Installation
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Controller Installation

+

Netris Controller can be installed locally as a VM, deployed as a Kubernetes application, or hosted in the Netris Cloud. All three options provide the same functionality. Cloud-hosted Controller can be moved into on-prem anytime.

+
+

Note

+

Select ONE installation option below.

+
+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/installing-netris-controller.html b/en/latest/installing-netris-controller.html new file mode 100644 index 0000000000..1a69ea21cf --- /dev/null +++ b/en/latest/installing-netris-controller.html @@ -0,0 +1,860 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Installing a Netris Controller
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Installing a Netris Controller

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where to host the Netris controller. What matters is that the Netris controller needs to be accessible over the Internet. So you can access the console, and nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface.

+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

In this example I am running my Netris controller on an AWS hosted virtual machine (EC2) which has got a public IP address 54.219.211.71. While it is OK for users and nodes to refer to the Netris Controller through an IP address, I like using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

I’m using Cloudflare to create this “example-netris-controller.netris.dev” DNS record to point to the public IP address of my EC2 : 54.219.211.71.

+_images/cloudflare-dns-record.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+

To install Netris Controller on a freshly installed Linux you only need to run below one-liner command. Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-hostname ” will instruct the installer to generate a Let’s Encrypt SSL certificate for the provided domain name. That’s why it is important to create the DNS record before this step.

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com
+
+
+

Once installation process is finished you will be able to access your newly installed Netris Controller web consonle using netris/newNet0ps credentials.

+

Please immediately change the default password to something strong in Setting → My Account → Change Password. +You can also use Settings → Login whitelist to restrict web console access to the controller.

+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/introduction.html b/en/latest/introduction.html new file mode 100644 index 0000000000..a1b8f1264b --- /dev/null +++ b/en/latest/introduction.html @@ -0,0 +1,857 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Introduction to Netris — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Introduction to Netris
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Introduction to Netris

+

Netris is a network automation and abstraction software for cloud builders. Netris brings cloud-like VPC abstractions for operating physical networks like it is a cloud. Netris automatically configures switching, routing, load-balancing, and network security based on user-defined services and policies. Netris continuously monitors the network’s health and either applies software remediation or informs you of necessary actions if human intervention is required. Netris abstracts away the complexities of detailed network configuration, letting you perform efficiently by operating your physical network in a top down approach like a cloud – instead of the legacy box by box operation.

+_images/private-cloud-enterprise-dc-2.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/inventory-profiles.html b/en/latest/inventory-profiles.html new file mode 100644 index 0000000000..27497a8de7 --- /dev/null +++ b/en/latest/inventory-profiles.html @@ -0,0 +1,880 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Inventory Profiles — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +
+

Inventory Profiles

+

Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/softgate is allowed. +As soon as the inventory profile is attached to a device it denies all traffic destined to the device except netris-defined and user-defined custom flows. Generated rules include:

+
    +
  • SSH from user defined subnets

  • +
  • NTP from user defined ntp services

  • +
  • DNS from user defined DNS servers

  • +
  • Custom user defined rules

  • +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + +
Inventory Profile Fields

Name

Profile name

Description

Free text description

Allow SSH from IPv4

List of IPv4 subnets allowed to ssh (one address per line)

Allow SSH from IPv6

List of IPv6 subnets allowed to ssh (one address per line)

Timezone

Devices using this inventory profile will adjust their system time to the selected timezone.

NTP servers

List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate.

DNS servers

List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate.

+

Example: In this example Netris Controller is used to provide NTP and DNS services to the switches (common setup).

+Adding an inventory profile +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/ipam.html b/en/latest/ipam.html new file mode 100644 index 0000000000..19bc0c47c3 --- /dev/null +++ b/en/latest/ipam.html @@ -0,0 +1,925 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + IP Address Management (IPAM) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • IP Address Management (IPAM)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

IP Address Management (IPAM)

+

Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting.

+

Purpose: +Users define specific roles(purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-net, NAT, etc…

+
+

Allocations and Subnets

+

There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets.

+IPAM Tree View +

IPAM Tree View

+
+
+
+

Add an Allocation

+
    +
  1. Navigate to Net→IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Allocation from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + +
Allocation Fields

Name

Unique name for current allocation.

Prefix

Unique prefix for allocation, must not overlap with other allocations.

Tenant

Owner of the allocation.

+Add a New IP Allocation +

Add Allocation Window

+
+
+
+

Add a Subnet

+
    +
  1. Navigate to Net→IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Subnet from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + + + + +
Subnet fields

Name

Unique name for current subnet.

Prefix

Unique prefix for subnet, ust be included in one of allocations.

Tenant

Owner of the subnet.

Purpose

This field describes for what kind of services the current subnet can be used. It can have the following values:

+
+
    +
  • common - ordinary subnet, can be used in v-nets and ROH.

  • +
  • loopback - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates).

  • +
  • management - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates).

  • +
  • load-balancer - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience.

  • +
  • nat - hosts of this subnet or subnet itself can be used to define NAT services.

  • +
  • inactive - can’t be used in any services, useful for reserving/documenting prefixes for future use.

  • +
+
+
+Add a New Subnet +

Add Subnet Window

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/kubernetes-integration.html b/en/latest/kubernetes-integration.html new file mode 100644 index 0000000000..529959bbf2 --- /dev/null +++ b/en/latest/kubernetes-integration.html @@ -0,0 +1,1221 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Kubernetes Integration — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Kubernetes Integration
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Kubernetes Integration

+

Netris integrates with Kube API to provide on-demand load balancer and other Kubernetes specific networking features. Netris-Kubernetes integration is designed to complement Kubernetes CNI networking and provide a cloud-like user experience to local Kubernetes clusters.

+
+

Install Netris Operator

+

Integration between the Netris Controller and the Kubernetes API is completed by installing the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart:

+ +
+

Regular Manifest Method

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='http://**your-netris-controller-ip-or-host**' \
+--from-literal=login='**your-netris-admin-username**' --from-literal=password='**your-netris-admin-password**'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+
+

Using Type ‘LoadBalancer’

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services / L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox pod provisioning +

After provisioning has finished, inspect the service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.202   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9898   active   50.117.59.202   9898/TCP   US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9898   active   50.117.59.202   9898/TCP   US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.202   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.203   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.203
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+View L4 LB instances +
+
+

V-Net Custom Resource

+

You can also create Netris V-Nets (L2 segments) via Kubernetes with a simple manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our VNet resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new VNet has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+
+
+

BGP Custom Resource

+

You can create BGP peers via Kubernetes manifests:

+
    +
  1. Create a yaml file:

  2. +
+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1092
+  localIP: 50.117.59.118/30
+  remoteIP: 50.117.59.117/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.192/28 le 32
+EOF
+
+
+
    +
  1. Apply the manifest file:

  2. +
+
kubectl apply -f isp2-customer.yaml
+
+
+
    +
  1. Check created BGP:

  2. +
+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.118/30   50.117.59.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.118/30   50.117.59.117/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the controller web interface previously.

+

Return to the Netris UI and navigate to Net / Topology to see the new BGP neighbor you created.

+
+
+

Importing existing resources from Netris Controller to Kubernetes

+

You can import any custom resources, already created from the Netris Controller to k8s by adding this annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them w/out “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Calico CNI Integration

+

Netris Operator can integrate with Calico CNI. This annotation will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.118/30   50.117.59.117/30    7m59s
+sandbox9-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox9-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox9-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As seen our BGP peers are established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.118/30   50.117.59.117/30    8m41s
+sandbox9-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox9-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox9-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until k8s cluster all nodes’ BGP peers are being established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.202
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/l3-load-balancer.html b/en/latest/l3-load-balancer.html new file mode 100644 index 0000000000..b742619b5c --- /dev/null +++ b/en/latest/l3-load-balancer.html @@ -0,0 +1,874 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + L3 Load Balancer (Anycast LB) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • L3 Load Balancer (Anycast LB)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

L3 Load Balancer (Anycast LB)

+

L3 (Anycast) load balancer leverages ECMP load balancing and hashing capability of spine and leaf switches to deliver line-rate server load balancing with health checks.

+

ROH servers, besides advertising their unicast (unique) loopback IP address, need to configure and advertise an additional anycast (the same IP) IP address. Unicast IP address is used for connecting to each individual server.

+

End-user traffic should be destined to the anycast IP address. The switch fabric uses ECMP to load balance the traffic towards every server, and will hash sessions based on IP/Protocol/Port such that TCP sessions will exist between the given end-user and server pair for the lifetime of the session. Optional health checks are used to identify application failures and reroute traffic in the case of a server outage.

+
+

Creating an L3 Load Balancer

+

To configure L3 (Anycast) load balancing:

+
    +
  1. Navigate to Services→Instances (ROH) and locate an existing ROH instance

  2. +
  3. Click the ellipses and then the Edit button

  4. +
  5. Select an extra IPv4 address from the select box at the bottom, and check the Anycast option.

  6. +
  7. This will create a service under Services→Load Balancer and permit using the Anycast IP address in multiple ROH instances.

  8. +
+Add a L3 LB +

Example: Adding an Anycast IPv4 address

+List L3 LBs +

Example: Under Services→Load Balancer, you can find the listing of L3 (Anycast) Load Balancers, service statuses, and you can add/remove more ROH instances and/or health checks.

+List L3 LB Details +

Screenshot: L3 (Anycast) Load Balancer Detail

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/l4-load-balancer.html b/en/latest/l4-load-balancer.html new file mode 100644 index 0000000000..5334eaf4f8 --- /dev/null +++ b/en/latest/l4-load-balancer.html @@ -0,0 +1,907 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + L4 Load Balancer (L4LB) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • L4 Load Balancer (L4LB)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

L4 Load Balancer (L4LB)

+

Netris L4 Load Balancer (L4LB) leverages SoftGate(Linux router) nodes to provide Layer-4 load balancing services, including on-demand cloud load balancing with native integration with Kubernetes.

+
+

Enabling L4LB service

+

L4 Load Balancer service requires at least one SoftGate node to be available in a given Site, as well as at least one IP address assignment (purpose=load balancer).

+

The IP address pool for L4LB can be defined in the Net→IPAM section by adding an Allocation and setting the purpose field to ‘load-balancer’. You can define multiple IP pools for L4LB at any given site. See the below example.

+

Example: Adding a load-balancer IP pool assignment.

+Add an IP Allocation +

Screenshot: Listing of Net→IPAM after adding a load-balancer assignment

+List IP Subnets +
+
+

Consuming L4LB service

+

This guide describes how to request an L4 Load Balancer using GUI. For Kubernetes integration, check the Kubenet section.

+

Click +add under Services→L4 Load Balancer to request an L4LB service.

+

Add new L4 Load Balancer fields are described below:

+

General fields

+
    +
  • Name - Unique name.

  • +
  • Protocol - TCP or UDP.

  • +
  • Tenant - Requestor Tenant should have access to the backend IP space.

  • +
  • Site - Site where L4LB service is being requested for. Backends should belong on this site.

  • +
  • State - Administrative state.

  • +
+

Frontend

+
    +
  • Address - Frontend IP address to be exposed for this L4LB service. “Assign automatically” will provide the next available IP address from the defined load-balancer pool. Alternatively, users can select manually from the list of available addresses.

  • +
  • Port - TCP or UDP port to be exposed.

  • +
+

Health-check

+
    +
  • Type - Probe backends on service availability.

    +
      +
    • None - load balance unconditionally.

    • +
    • TCP - probe backend service availability through TCP connect checks.

    • +
    • HTTP - probe backend service availability through HTTP GET checks.

    • +
    +
  • +
  • Timeout(ms) - Probe timeout in milliseconds.

  • +
  • Request path - HTTP request path.

  • +
+

Backend

+
    +
  • +Add - add a backend host.

  • +
  • Address - IP address of the backend host.

  • +
  • Port - Service port on the backend host.

  • +
  • Enabled - Administrative state of particular backend.

  • +
+Request an L4 Load Balancer +

Example: Requesting an L4 Load Balancer service.

+List L4 Load Balancers +

Example: Listing of L4 Load Balancer services

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/lag.html b/en/latest/lag.html new file mode 100644 index 0000000000..37c348056a --- /dev/null +++ b/en/latest/lag.html @@ -0,0 +1,892 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Link Aggregation (LAG) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Link Aggregation (LAG)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ + + + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/maintenance-mode.html b/en/latest/maintenance-mode.html new file mode 100644 index 0000000000..a7099acce7 --- /dev/null +++ b/en/latest/maintenance-mode.html @@ -0,0 +1,899 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Maintenance Mode — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Maintenance Mode

+
+

Overview

+

Maintenance mode is intended to assist in smoothly redirecting traffic away from a particular device for a maintenance to be carried out with minimal impact on your network. It’s advisable to activate Maintenance Mode, wait a few minutes and ensure that traffic has been re-routed, prior to initiating maintenance procedures on the device. Once the maintenance is completed, you should deactivate Maintenance Mode to switchover the traffic back to normal. +To toggle Maintenance Mode on or off, navigate to the Inventory section, or Topology manager (more convenient for switch-fabric). Edit the devices, and use the Maintenance Mode checkbox to enable/disable.

+_images/maintenance-mode.png +
+

Note

+

Maintenance mode strives to redirect traffic away from the current device; however, please note that it does not guarantee the complete offloading of traffic.

+
+
+
+

Maintenance Mode for Softgate - What’s happening behind the scenes?

+

When you activate Maintenance Mode for the softgate, several automatic actions are undertaken behind the scenes to redirect traffic away from the softgate:

+
+
    +
  • The BGP local preference attribute is lowered for both external and internal peers.

  • +
  • Route information is prepended tenfold for outbound direction to all external and internal peers.

  • +
  • The BGP MED attribute is increased for all external and internal peers.

  • +
  • The BGP origin attribute is decreased for all external and internal peers.

  • +
  • Connection-oriented services such as SNAT and L4LB will be transferred to the second softgate, resulting in a reestablishment of TCP connections.

  • +
+
+
+

Note

+

Before activating the maintenance mode, make sure that the second softgate is operating correctly.

+
+
+
+

Maintenance Mode for Switch - What’s happening behind the scenes?

+

When you activate Maintenance Mode for the switch, several automatic actions are undertaken behind the scenes to redirect traffic away from the switch:

+
+
    +
  • The BGP local preference attribute is lowered for both external and internal peers.

  • +
  • Route information is prepended tenfold for outbound direction to all external and internal peers.

  • +
  • The BGP MED attribute is increased for all external and internal peers.

  • +
  • The BGP origin attribute is decreased for all external and internal peers.

  • +
  • The LACP system ID will undergo modification for EVPN Multihomed hosts, resulting in a reorganization of traffic from ports that connect to these multihomed hosts. The traffic will be redirected to other switches where the hosts are also linked.

  • +
+
+
+

Note

+

If a host is connected to two switches, and the first switch is in Maintenance Mode, if the link connected to the second switch experiences instability (link flapping), the connection linked to the first switch will become active. However, this connection will not automatically revert back to the second switch once it stabilizes (no preemption).

+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/netris-architecture.html b/en/latest/netris-architecture.html new file mode 100644 index 0000000000..df8f6959b4 --- /dev/null +++ b/en/latest/netris-architecture.html @@ -0,0 +1,885 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Netris Architecture — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Architecture

+

A Netris system is composed of 3 elements:

+
    +
  • Netris Controller

  • +
  • Netris Switch Agent

  • +
  • Netris SoftGate

  • +
+
+

Netris Controller

+

Netris Controller is the main operations control center for engineers using GUI/RestAPI/Kubernetes, systems, and network devices. The Netris Controller stores the data representing the user-defined network services and policies, health, statistics, analytics received from the network devices, and information from integration modules with external systems (Kubernetes, Terraform, etc.). Netris Controller can run as a VM or container, on/off-prem, or in Netris cloud.

+

Diagram: High level Netris architecture

+_images/netris_controller_diagram.png +
    +
  • Controller HA We highly recommend running more than one copy of the controller for database replication. Find here backup and restore procedure.

  • +
  • Multiple sites Netris is designed to operate multiple sites with just a single controller with HA

  • +
  • What if the controller is unreachable. Netris operated switches/routers can tolerate the unreachability of the Netris Controller. Changes and stats collection will be unavailable during the controller unavailability window; however, switches/routers core operations will not be affected.

  • +
+
+
+

Netris Switch Agent

+

Netris Switch Agent is software running in the user space of the network operating system (NOS) of the switch and is responsible for automatically generating the particular switch configuration according to service requirements and policies defined in the Netris Controller. Netris Switch Agent uses an encrypted GRPC protocol for secure communication with the Netris Controller accessible through a local management network or over the Internet.

+
+
+

Netris SoftGate

+

Netris SoftGate is a software for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), DHCP, Firewall, and site-to-site VPN function on a regular x86 server with a SmartNIC (optionally) card.

+

Netris SoftGate supports a high-performance DPDK data plane running in the user-space. It configures the system so that packets entering the NIC (network interface card) bypass Linux Kernel and go directly to the user space application. So traffic from the NIC travels through the PCIe bus to the closest CPU’s last level cache and then into one of 8 cores, all reserved for the data-plane application. DPDK data-plane software processes the traffic for routing, load-balancing, NAT and makes necessary changes in the packet header (rewrites mac/VLAN-id) then returns the packet to the NIC, which sends it further into the switch for traveling further in Layer-2.

+

The SoftGate PRO (the 100Gbps version) server has to have 2 x Intel CPUs (8+ cores each). One CPU (closest to the SmartNIC card) is reserved for the data-plane process only (OS will report 100% CPU usage). Another CPU is used for running Linux OS, routing control plane (FRR), Netris agent, and other standard Linux utilities.

+

Netris agents can also configure Wireguard to form full mesh VPN tunnels between customer sites and then run necessary dynamic routing. So, servers and applications in multiple data centers can communicate over the Internet using encrypted tunnels.

+

Diagram: Netris SoftGate high level architecture.

+_images/softgate_diagram.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/network-policies.html b/en/latest/network-policies.html new file mode 100644 index 0000000000..12842c3394 --- /dev/null +++ b/en/latest/network-policies.html @@ -0,0 +1,1230 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + VPC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

VPC

+

Netris Controller is preconfigured with a default system VPC-1. Use the default VPC, and create additional VPCs as needed in the future.

+

The VPC acts as a VRF in traditional networking, providing the ability to use overlapping IP ranges across various VPCs while maintaining safe management and operation of services.

+

VPC can be created in the Network → VPC section.

+
+

Adding new VPC

+
    +
  1. Navigate to Network → VPC in the web UI.

  2. +
  3. Click Add button.

  4. +
+_images/vpc_empty.png +
+
+
+

IP Address Management

+

Netris IPAM allows users to document their IP addresses and track pool usage. It is designed to have a tree-like view to provide opportunity to perform any kind of subnetting.

+

Purpose: Users define specific roles (purpose) for each subnet/address and only after that are allowed to use those subnets in services like V-Net, NAT, etc…

+

Each VPC has its own IPAM table.

+
+

Allocations and Subnets

+

There are 2 main types of IP prefixes - allocation and subnet. Allocations are IP ranges allocated to an organization via RIR/LIR or private IP ranges that are going to be used by the network. Subnets are prefixes which are going to be used in services. Subnets are always childs of allocation. Allocations do not have parent subnets.

+IPAM Tree View +
+
+

IPAM Tree View

+
+
+

Add an Allocation

+
    +
  1. Navigate to Network → IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Allocation from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + + + + +
Allocation Fields

Prefix

Unique prefix for allocation, must not overlap with other allocations.

Name

Unique name for current allocation.

VPC

Select a VPC to which the allocation belongs.

Tenant

Owner of the allocation.

+Add a New IP Allocation +

Add Allocation Window

+
+
+
+

Add a Subnet

+
    +
  1. Navigate to Network → IPAM

  2. +
  3. Click the Add button

  4. +
  5. Select Subnet from the bottom select box

  6. +
  7. Fill in the rest of the fields based on the requirements listed below

  8. +
  9. Click the Add button

  10. +
+ + ++++ + + + + + + + + + + + + + + + + + +
Subnet fields

Prefix

Unique prefix for subnet, ust be included in one of allocations.

Name

Unique name for current subnet.

VPC

Select a VPC to which the subnet belongs.

Tenant

Owner of the subnet.

Purpose

This field describes for what kind of services the current subnet can be used. It can have the following values:

+
+
    +
  • common - ordinary subnet, can be used in v-nets and ROH.

  • +
  • loopback - hosts of this subnet can be used only as loopback IP addresses for Netris hardware (switches and/or softgates).

  • +
  • management - subnet which specifies the out-of-band management IP addresses for Netris hardware (switches and softgates).

  • +
  • load-balancer - hosts of this subnet are used in L4LB services only. Useful for deploying on-prem kubernetes with cloud-like experience.

  • +
  • nat - hosts of this subnet or subnet itself can be used to define NAT services.

  • +
  • inactive - can’t be used in any services, useful for reserving/documenting prefixes for future use.

  • +
+
+
+Add a New Subnet +

Add Subnet Window

+
+
+
+

Basic BGP

+

BGP neighbors can be declared in the Network → E-BGP section. Netris software will automatically generate and program the network configuration to meet the requirements.

+
+

Adding BGP Peers

+
    +
  1. Navigate to Network → E-BGP in the web UI.

  2. +
  3. Click the Add button.

  4. +
  5. Fill in the fields as described in the table below.

  6. +
  7. Click the Add button.

  8. +
+

Example: Declare a basic BGP neighbor.

+_images/bgp_empty.png +
+
+
+

Advanced BGP

+

BGP neighbor declaration can optionally include advanced BGP attributes and BGP route-maps for fine-tuning of BGP policies.

+

Click Advanced to expand the BGP neighbor add/edit window.

+
+
+

BGP Objects

+
+
Under Network → E-BGP objects, you can define various BGP objects referenced from a route-map to declare a dynamic BGP policy.
+
Supported objects include:
+
+
    +
  • IPv4 Prefix

  • +
  • IPv6 Prefix

  • +
  • AS-PATH

  • +
  • Community

  • +
  • Extended Community

  • +
  • Large Community

  • +
+
+

IPv4 Prefix

+
+
The rules are defined one per line.
+
Each line in IPv4 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv4 prefix (mandatory).

  • +
  • Length - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv4 Prefix list.

+_images/ipv4_prefix.png +
+
+

IPv6 Prefix

+
+
Rules defined one per line.
+
Each line in IPv6 prefix list field consists of three parts:
+
+
    +
  • Action - Possible values are: permit or deny (mandatory).

  • +
  • IP Prefix - Any valid IPv6 prefix (mandatory).

  • +
  • Keyword - Possible values are: le <len>, ge <len> or ge <len> le <len>.

  • +
+

Example: Creating an IPv6 Prefix list.

+_images/ipv6_prefix.png +
+
+

Community

+
+
Community field has two parts:
+
+
    +
  • Action - Possible values: permit or deny (mandatory).

  • +
  • Community string - format is AA:NN, where AA and NN are any number from 0 to 65535 range or alternatively well known string (local-AS|no-advertise|no-export|internet|additive).

  • +
+

Example: Creating community.

+_images/community.png +
+
+
+
+

BGP route-maps

+
+
Under the Network → E-BGP Route-maps section, you can define route-map policies, which can be associated with the BGP neighbors inbound or outbound.
+
+
+
Description of route-map fields:
+
+
    +
  • Sequence Number - Automatically assigned a sequence number. Drag and move sequences to organize the order.

  • +
  • Description - Free description.

  • +
  • Policy - Permit or deny the routes which match below all match clauses within the current sequence.

  • +
  • Match - Rules for route matching.

    +
      +
    • Type - Type of the object to match: AS-Path, Community, Extended Community, Large Community, IPv4 prefix-list, IPv4 next-hop, Route Source, IPv6 prefix-list. IPv6 next-hop, local-preference, MED, Origin, Route Tag.

    • +
    • Object - Select an object from the list.

    • +
    +
  • +
  • Action - Action when all match clauses are met.

    +
      +
    • Action type - Define whether to manipulate a particular BGP attribute or go to another sequence.

    • +
    • Attribute - The attribute to be manipulated.

    • +
    • Value - New attribute value.

    • +
    +
  • +
+

Example: route-map

+_images/route-map.png +
+
+
+
+

Static Routing

+

Located under Network → Routes is a method for describing static routing policies that Netris will dynamically inject on switches and/or SoftGate where appropriate. We recommend using the Routes only if BGP is not supported by the remote end.

+
+
Typical use cases for static routing:
+
+
    +
  • To connect the switch fabric to an ISP or upstream router in a situation where BGP and dual-homing are not supported.

  • +
  • Temporary interconnection with the old network for a migration.

  • +
  • Routing a subnet behind a VM hypervisor machine for an internal VM network.

  • +
  • Specifically routing traffic destined to a particular prefix through an out-of-band management network.

  • +
+
+
Add new static route fields description:
+
+
    +
  • Prefix - Route destination to match.

  • +
  • Next-Hop - Traffic destined to the Prefix will be routed towards the Next-Hop. Note that static routes will be injected only on units that have the Next-Hop as a connected network.

  • +
  • Description - Free description.

  • +
  • VPC - Select a VPC to which the static route belongs.

  • +
  • Site - Site where Route belongs.

  • +
  • State - Administrative (enable/disable) state of the Route.

  • +
  • Apply to - Limit the scope to particular units. It’s typically used for Null routes.

  • +
+

Example: Default route pointing to a Next-Hop that belongs to one of V-Nets.

+_images/static_route_empty.png +

Example: Adding a back route to 10.254.0.0/16 through an out-of-band management network.

+_images/static_route2_empty.png +

Screenshot shows that the back route is actually applied on Softgate1 and Softgate2 .

+_images/static_route3.png +
+
+
+

NAT

+

Netris SoftGate nodes are required forNAT (Network Address Translation) functionality to work.

+

Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).

+
+

Enabling NAT

+

To enable NAT for a given site, you first need to create a subnet with NAT purpose in the IPAM section. The NAT IP addresses can be used for SNAT or DNAT as a global IP address (the public IP visible on the Internet). NAT IP pools are IP address ranges that SNAT can use as a rolling global IP (for a larger scale, similar to carrier-grade SNAT). SNAT is always overloading the ports, so many local hosts can share one or just a few public IP addresses. You can add as many NAT IP addresses and NAT pools as you need. +Adding an IP Subnet under Network → IPAM.

+
    +
  1. Allocate a public IP subnet for NAT under Net→IPAM.

  2. +
+

Example: Adding an IP allocation under Net→Subnets.

+_images/nat_subnet_empty.png +
+
+

Defining NAT rules

+

NAT rules are defined under Network → NAT.

+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
NAT Rule Fields

Name

Unique name.

State

State of rule (enabled or disabled).

Site

Site to apply the rule.

Action

SNAT - Replace the source IP address with specified NAT IP along with port overloading. +DNAT - Replace the destination IP address and/or destination port with specified NAT IP. +ACCEPT - Silently forward, typically used to add an exclusion to broader SNAT or DNAT rule. +MASQUERADE - Replace the source IP address with the IP address of the exit interface.

Protocol

All - Match any IP protocol. +TCP - Match TCP traffic and ports. +UDP - Match UDP traffic and ports. +ICMP - Match ICMP traffic.

Source

Address - Source IP address to match. +Port - Source ports range to match with this value (TCP/UDP).

Destination

Address - Destination IP address to match. In the case of DNAT it should be one of the predefined NAT IP addresses. +Port - For DNAT only, to match a single destination port. +Ports - For SNAT/ACCEPT only. Destination ports range to match with this value (TCP/UDP).

DNAT to IP

The global IP address for SNAT to be visible on the Public Internet. The internal IP address for DNAT to replace the original destination address with.

DNAT to Port

The Port to which destination Port of the packet should be NAT’d.

Status

Administrative state (enable/disable).

Comment

Free optional comment.

+

Example: SNAT all hosts on 10.0.1.0/24subnet to the Internet using 192.0.2.128as a global IP.

+_images/create_snat_empty.png +

Example: Port forwarding. DNAT the traffic destined to 192.0.2.130:8080 to be forwarded to the host 10.0.1.100 on port tcp/80.

+_images/create_dnat_empty.png +
+
+
+
+

SiteMesh

+

SiteMesh is a Netris service for site-to-site interconnection over the public Internet. SiteMesh automatically generates configuration for WireGuard to create encrypted tunnels between participating sites and automatically generates a configuration for FRR to run dynamic routing. Hence, sites learn how to reach each other over the mesh WireGuard tunnels. The SiteMesh feature requires a SoftGate node at each participating site.

+

Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).

+

Edit Network → Sites, do declare what sites should form a SiteMesh. See SiteMesh types described below.

+
    +
  • Disabled - Do not participate in SiteMesh.

  • +
  • Hub - Hub sites form full-mesh tunnels with all other sites (Hub and non-Hub) and can carry transit traffic for non-Hub sites. (usually major data center sites)

  • +
  • Spoke - Spoke sites form tunnels with all Hub sites. Spoke to Spoke traffic will transit a Hub site. (small data center sites or major office sites)

  • +
  • Dynamic Spoke - Dynamic Spoke is like Spoke, but it will maintain a tunnel only with one Hub site, based on dynamic connectivity measurements underneath and mathematical modeling. (small office sites)

  • +
+

Screenshot: Site Mesh parameter editing a Site under Network → Sites.

+_images/sitemesh_edit.png +

You only need to define your site-to-site VPN architecture policy by selecting SiteMesh mode for every site. Netris will generate the WireGuard tunnels (using randomly generated keys, and generate FRR rules to get the dynamic routing to converge.

+_images/SiteMesh_modes.png +

Check the Network → Site Mesh section for the listing of tunnel statuses.

+

Screenshot: Listing of SiteMesh tunnels and BGP statuses (Net→Site Mesh)

+_images/SiteMesh_listing.png +
+
+
+

Looking Glass

+

The Looking Glass Is a GUI-based tool for looking up routing information from a switch or SoftGate perspective. You can access the Looking Glass either from Topology, individually for every device (right click on device → details → Looking Glass), or by navigating to Network → Looking Glass then selecting the device from the top-left dropdown menu.

+

Looking Glass controls described for IPv4/IPv6 protocol families.

+
    +
  • VPC - select a VPC.

  • +
  • BGP Summary - Shows the summary of BGP adjacencies with neighbors, interface names, prefixes received. You can click on the neighbor name then query for the list of advertised/received prefixes.

  • +
  • BGP Route - Lookup the BGP table (RIB) for the given address.

  • +
  • Route - Lookup switch routing table for the given address.

  • +
  • Traceroute - Conduct a traceroute from the selected device towards the given destination, optionally allowing to determine the source IP address.

  • +
  • Ping - Execute a ping on the selected device towards the given destination, optionally allowing to select the source IP address.

  • +
+

Example: listing BGP neighbors of a switch and number of received prefixes for the Underlay VPC.

+_images/lg_summary.png +

Example: BGP Route - looking up V-Net subnet from switch11 perspective. Switch11 is load balancing between four available paths.

+_images/lg_bgp_route.png +

Example: Ping.

+_images/lg_ping.png +
+
Looking Glass controls described for the EVPN family.
+
+
    +
  • VPC - select a VPC.

  • +
  • BGP Summary - Show brief summary of BGP adjacencies with neighbors, interface names, and EVPN prefixes received.

  • +
  • VNI - List VNIs learned.

  • +
  • BGP EVPN - List detailed EVPN routing information optionally for the given route distinguisher.

  • +
  • MAC table - List MAC address table for the given VNI.

  • +
+

Example: Listing MAC addresses on VNI 50.

+_images/lg_mac.png +

Example: EVPN routing information listing for a specified route distinguisher.

+_images/lg_rd.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/objects.inv b/en/latest/objects.inv new file mode 100644 index 0000000000..12690e1506 Binary files /dev/null and b/en/latest/objects.inv differ diff --git a/en/latest/reference-designs.html b/en/latest/reference-designs.html new file mode 100644 index 0000000000..fdaf0e9e46 --- /dev/null +++ b/en/latest/reference-designs.html @@ -0,0 +1,849 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Network Reference Designs — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Network Reference Designs
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Network Reference Designs

+

Netris can support any type of standard network design.

+

The majority of designs fall into one of four patterns:

+
    +
  1. Unmanaged Switch

  2. +
  3. Single L2 Domain

  4. +
  5. Collapsed Core

  6. +
  7. Data Center Leaf/Spine Designs (Modern High Throughput Redundant Design)

  8. +
  9. Data Center Core/Distribution/Access (Legacy High Throughput Redundant Design)

  10. +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/release-notes.html b/en/latest/release-notes.html new file mode 100644 index 0000000000..8cb147c65b --- /dev/null +++ b/en/latest/release-notes.html @@ -0,0 +1,852 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Release notes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + +
+
+
+
+ +
+

Release notes

+
    +
  • DPDK​ ​data plane support for SoftGate nodes​. - Provides higher SoftGate performance. Up to 27Mpps, 100Gbps for L3 routing, 12Mpps with NAT rules on.

  • +
  • L4 Load Balancer​. - In addition to switch-based Anycast Load Balancer, we now support a SoftGate/DPDK-based L4 Load Balancer. L4LB integrates with Kubernetes providing cloud-like load balancer service (type: load-balancer).

  • +
  • Kubenet​ - a network service purpose-built for Kubernetes cluster nodes. Kubenet integrates with Kube API to provide an on-demand load balancer andother Kubernetes specific networking features. Netris Kubenet is designed to complement Kubernetes CNI networking with modern physical networking.

  • +
  • API logs​ - Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type.

  • +
  • Site Mesh​ - a Netris service for automatically configuring site-to-site interconnect over the public Internet. Site Mesh supports configuration for WireGuard to create encrypted tunnels between participating sites andautomatically generates configuration for FRR to run dynamic routing. In a few clicks, services in one site get connectivity to services in other sites over a mesh of WireGuard tunnels.

  • +
  • Ubuntu/SwitchDev updates​ - Removed the requirement for a hairpin loop cable. Removed the need for IP address reservation for V-NET, switching entirely to the anycast default gateway.

  • +
  • Controller distributions​ - Netris controller, is now available in three deployment forms. 1) On-prem KVM virtual machine. 2) Kubernetes application. 3) Managed/Hosted in the cloud.

  • +
  • Inventory Profiles​ - A construct for defining access security, timezone, DNS,NTP settings profiles for network switches and SoftGate nodes.

  • +
  • Switch/SoftGate agents​ - New installer with easy initial config tool. Support for IP and FQDN as a controller address. Authentication key.

  • +
  • GUI​ - Improved Net→Topology section, becoming the main and required place for defining the network topology. All sections got a column organizer, so every user can order and hide/show columns to their comfort.

  • +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/roh.html b/en/latest/roh.html new file mode 100644 index 0000000000..dc55c41456 --- /dev/null +++ b/en/latest/roh.html @@ -0,0 +1,895 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + ROH (Routing on the Host) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • ROH (Routing on the Host)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

ROH (Routing on the Host)

+

To create more resilient and higher-performance data centers, some companies leverage the Linux ecosystem to run routing protocols directly on their servers. This is commonly known as ROH (Routing on the Host).

+

In ROH architectures, servers use a routing daemon to establish a BGP adjacency with the switch fabric on every physical link. ROH can run on bare metal servers, VMs, and even containers. The most commonly used routing daemon/suite is FRR.

+

Hosts connected to the network in ROH architecture don’t have IP addresses on a shared Ethernet segment; instead an IP address is configured on the loopback interface and advertised over all BGP links towards switch fabric. This is a modern and optimal design, leveraging Layer-3 networking from the fabric to the servers.

+

By using only Layer-3 interfaces, Layer-2 protocols such as Spanning Tree (STP) can be minized and the reliability of the network increases.

+

The ROH architecture that is configured by Netris allows for leveraging ECMP load balancing capabilities of the switching hardware for the high-performance server load balancing (described in L3 Load Balancer section). For each instance of ROH, you’ll need to create a ROH entry in Netris Controller.

+
+

Adding ROH Hosts

+
    +
  1. Navigate in the Netris UI to Services→Instances (ROH)

  2. +
  3. Click the Add button

  4. +
  5. Fill out the form based on the fields in the table below.

  6. +
  7. Click the Add button

  8. +
+

Description of ROH instance fields:

+
    +
  • Name - Unique name for the ROH instance

  • +
  • Site - Site where the current ROH instance belongs

  • +
  • Type - Physical Server, for all servers forming a BGP adjacency directly with the switch fabric. Hypervisor, for using the hypervisor as an interim router. Proxmox is currently the only supported hypervisor.

  • +
  • ROH Routing Profile - ROH Routing profile defines what set of routing prefixes to be advertised to ROH instances

    +
      +
    • Default route only (most common design) - Will advertise 0.0.0.0/0 + loopback address of the physically connected switch

    • +
    • Default + Aggregate - Will add prefixes of defined assignments + “Default” profile

    • +
    • Full table - Will advertise all prefixes available in the routing table of the connected switch

    • +
    • Inherit - Will inherit policy from site objects defined under Net→Sites

    • +
    +
  • +
  • Legacy Mode - Switch from default zero-config mode to using /30 IP addresses. Used for MSFT Windows Servers or other OS that doesn’t support FRR.

  • +
  • +Port - Physical Switch Ports anywhere on the network.

  • +
  • +IPv4 - IPv4 addresses for the loopback interface.

  • +
  • +Inbound Prefix List - List of additional prefixes that the ROH server may advertise. Sometimes used to advertise container or VM networks.

  • +
+
+

Tip

+

Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gpbs switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+
+ROH Instances +

Example: Adding an ROH instance. (Yes, you can use A.B.C.0/32 and A.B.C.255/32)

+ROH Listings +

Expanded view of ROH listing. BGP sessions are up, and the expected IP is in fact received from the actual ROH server. Traffic stats are available per port.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox1/configurations.html b/en/latest/sandbox/Sandbox1/configurations.html new file mode 100644 index 0000000000..6eb08c16a4 --- /dev/null +++ b/en/latest/sandbox/Sandbox1/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox1.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc1::1) from the “2607:f358:11:ffc1::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.0)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.201 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox1/creating-services.html b/en/latest/sandbox/Sandbox1/creating-services.html new file mode 100644 index 0000000000..c8a270cb7b --- /dev/null +++ b/en/latest/sandbox/Sandbox1/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.201 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1012 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.22.

    6. +
    7. In the Remote IP field, type in 45.38.161.21.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.0/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.1)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.201 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.4/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.4/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.4/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.8/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.8”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.8” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.8) into the browser’s address bar or simply visit http://45.38.161.8/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.8 (name_45.38.161.8)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.8 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.201 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox1.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox1/index.html b/en/latest/sandbox/Sandbox1/index.html new file mode 100644 index 0000000000..ab9e29bf2f --- /dev/null +++ b/en/latest/sandbox/Sandbox1/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox1 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox1/onprem-k8s.html b/en/latest/sandbox/Sandbox1/onprem-k8s.html new file mode 100644 index 0000000000..58c6ffde25 --- /dev/null +++ b/en/latest/sandbox/Sandbox1/onprem-k8s.html @@ -0,0 +1,1361 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox1.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox1.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox1.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox1.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.13   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.13:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.13
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.13   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.13   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.13   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.13   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.14   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.14
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.14
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.14
+
+SRV05-NYC
+curl 45.38.161.14
+
+SRV04-NYC
+curl 45.38.161.14
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1012
+  localIP: 45.38.161.22/30
+  remoteIP: 45.38.161.21/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.0/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.22/30   45.38.161.21/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.22/30 45.38.161.21/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.22/30   45.38.161.21/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.22/30   45.38.161.21/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.13
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox1/sandbox-info.html b/en/latest/sandbox/Sandbox1/sandbox-info.html new file mode 100644 index 0000000000..615301b473 --- /dev/null +++ b/en/latest/sandbox/Sandbox1/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.201 -p 30061
+srv02-nyc: ssh demo@216.172.128.201 -p 30062
+srv03-nyc: ssh demo@216.172.128.201 -p 30063
+srv04-nyc: ssh demo@216.172.128.201 -p 30064
+srv05-nyc: ssh demo@216.172.128.201 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1011
+Local Address:                  45.38.161.18/30
+Remote Address:                 45.38.161.17/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.0/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1011
+Local Address:                  2607:f358:11:ffc0::3/127
+Remote Address:                 2607:f358:11:ffc0::2/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc1::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1012
+Local Address:                  45.38.161.22/30
+Remote Address:                 45.38.161.21/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.0/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.0/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.0/30
+|___ NAT Subnet:               45.38.161.4/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.8/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.12/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc1::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc1::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox10/configurations.html b/en/latest/sandbox/Sandbox10/configurations.html new file mode 100644 index 0000000000..b6f42d8677 --- /dev/null +++ b/en/latest/sandbox/Sandbox10/configurations.html @@ -0,0 +1,902 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

After logging into the Netris Controller by visiting https://sandbox10.netris.io and navigating to Services → V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example.

+

If you examine the particular service settings ( select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service), you will find that the services is configured on the second port of switch 21 named “swp2(swp2)@sw21-nyc (Admin)”.

+

The V-Net servicers is also configured with both an IPv4 and IPv6 gateway, 192.168.45.1 (from the “192.168.45.0/24 (EXAMPLE)” subnet) and 2607:f358:11:ffca::1 (from the “2607:f358:11:ffca::/64 (EXAMPLE IPv6)” subnet) respectively.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net → Looking Glass.

  2. +
  3. Select switch “sw21-nyc(10.254.46.21)” (the switch the “vnet-example” service is configured on) from the Select device drop-down menu.

  4. +
  5. Select “Ping” from the Command drop-down menu.

  6. +
  7. Type 192.168.45.64 (the IP address of srv04-nyc connected to swp2@sw21-nyc) in the field labeled IPv4 address.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between switch sw21-nyc and server srv04-nyc is working properly thanks to the configured V-Net service.

+
sw21-nyc# ip vrf exec Vrf_netris ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.562 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.745 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.690 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.737 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.666 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4092ms
+rtt min/avg/max/mdev = 0.562/0.680/0.745/0.065 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net → E-BGP. Here, aside from the necessary system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as example with IRIS ISP1. This ensures communication between the internal network with the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.210 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Net → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox10/creating-services.html b/en/latest/sandbox/Sandbox10/creating-services.html new file mode 100644 index 0000000000..300f1d8193 --- /dev/null +++ b/en/latest/sandbox/Sandbox10/creating-services.html @@ -0,0 +1,1054 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.210 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the Owner drop-down menu, select “Demo”.

    10. +
    11. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    12. +
    13. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    14. +
    15. From the Add Network Interface drop-down menu put a check mark next to switch port “swp2(swp2 | srv05-nyc)@sw22-nyc (Demo)”, which we can see is the the port where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    16. +
    +
    +
      +
    • The drop-down menu only contains this single switch port as it is the only port that has been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Check the Untag check-box and click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is reachable from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-NET” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Net → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, uncheck the Untag check-box and type in 1102.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.126.

    6. +
    7. In the Remote IP field, type in 50.117.59.125.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Inbound field, type in permit 0.0.0.0/0

    12. +
    13. In the Prefix List Outbound field, type in permit 50.117.59.208/28 le 32

    14. +
    15. And finally click Add

    16. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Select “SoftGate2(50.117.59.209)” (the border router where our newly created BGP session is terminated on) from the Select device drop-down menu.

  2. +
  3. Leaving the Family drop-down menu on IPv4 and the Command drop-down menu on “BGP Summary”, click on the Submit button.

  4. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.210 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Net → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. From the Protocol drop-down menu, select “ALL”.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, type in 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.212/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.212/32” IP address.

    22. +
    +
    +
      +
    • This public IP is part of 50.117.59.212/30 (NAT) subnet which is configured in the NET → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”..

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 load balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.216/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.216”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.216” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.216) into the browser’s address bar or simply visit http://50.117.59.216/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+../../_images/l3lb_srv01.png +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > to the left of “50.117.59.216 (name_50.117.59.216)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.216 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+../../_images/l3lb_srv02.png +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.210 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox10.netris.io and navigate to Net → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable.Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots () on the right side of the newly created “V-Net Customer to WAN” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, we can see in the terminal window that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox10/index.html b/en/latest/sandbox/Sandbox10/index.html new file mode 100644 index 0000000000..e928f65739 --- /dev/null +++ b/en/latest/sandbox/Sandbox10/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox10 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox10/onprem-k8s.html b/en/latest/sandbox/Sandbox10/onprem-k8s.html new file mode 100644 index 0000000000..ae3f34e2f9 --- /dev/null +++ b/en/latest/sandbox/Sandbox10/onprem-k8s.html @@ -0,0 +1,1338 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox10.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox10.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.221   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.221:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.221
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.221   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.221   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.221   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.221   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.222   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.222
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.222
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.222
+
+SRV05-NYC
+curl 50.117.59.222
+
+SRV05-NYC
+curl 50.117.59.222
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Demo
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1102
+  localIP: 50.117.59.126/30
+  remoteIP: 50.117.59.125/30
+  description: Example BGP to ISP2
+  prefixListInbound:
+    - permit 0.0.0.0/0
+  prefixListOutbound:
+    - permit 50.117.59.208/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.126/30   50.117.59.125/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME                      STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled   bgp: Established; prefix: 160; time: 00:01:27   Link Up      65007         50.117.59.126/30   50.117.59.125/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                 STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:06:18  Link Up      65007         50.117.59.126/30   50.117.59.125/30    7m59s
+sandbox10-srv06-nyc-192.168.110.66   enabled                                                               4230000000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox10-srv07-nyc-192.168.110.67   enabled                                                               4230000001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox10-srv08-nyc-192.168.110.68   enabled                                                               4230000002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                 STATE     BGP STATE                                       PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer              enabled   bgp: Established; prefix: 160; time: 00:07:48   Link Up      65007         50.117.59.126/30   50.117.59.125/30    8m41s
+sandbox10-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox10-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19     N/A          4230000001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox10-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44     N/A          4230000002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.221
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox10/sandbox-info.html b/en/latest/sandbox/Sandbox10/sandbox-info.html new file mode 100644 index 0000000000..bb3f8fd427 --- /dev/null +++ b/en/latest/sandbox/Sandbox10/sandbox-info.html @@ -0,0 +1,952 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+Sandbox Topology +
+
+

Netris Controller

+

https://sandbox10.netris.io

+
+
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.210 -p 30061
+srv02-nyc: ssh demo@216.172.128.210 -p 30062
+srv03-nyc: ssh demo@216.172.128.210 -p 30063
+srv04-nyc: ssh demo@216.172.128.210 -p 30064
+srv05-nyc: ssh demo@216.172.128.210 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under NET → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1101
+Local Address:                  50.117.59.122/30
+Remote Address:                 50.117.59.121/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.208/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1101
+Local Address:                  2607:f358:11:ffc0::15/127
+Remote Address:                 2607:f358:11:ffc0::14/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffca::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1102
+Local Address:                  50.117.59.126/30
+Remote Address:                 50.117.59.125/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.208/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Net → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.208/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.208/30
+|___ NAT Subnet:               50.117.59.212/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.216/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.220/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffca::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffca::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox11/configurations.html b/en/latest/sandbox/Sandbox11/configurations.html new file mode 100644 index 0000000000..50777764d2 --- /dev/null +++ b/en/latest/sandbox/Sandbox11/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox11.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffcb::1) from the “2607:f358:11:ffcb::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.96)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.211 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox11/creating-services.html b/en/latest/sandbox/Sandbox11/creating-services.html new file mode 100644 index 0000000000..639cee3d05 --- /dev/null +++ b/en/latest/sandbox/Sandbox11/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.211 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1112 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.118.

    6. +
    7. In the Remote IP field, type in 45.38.161.117.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.96/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.97)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.211 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.100/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.100/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.100/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.104/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.104”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.104” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.104) into the browser’s address bar or simply visit http://45.38.161.104/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.104 (name_45.38.161.104)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.104 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.211 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox11.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox11/index.html b/en/latest/sandbox/Sandbox11/index.html new file mode 100644 index 0000000000..670d32a438 --- /dev/null +++ b/en/latest/sandbox/Sandbox11/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox11 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox11/onprem-k8s.html b/en/latest/sandbox/Sandbox11/onprem-k8s.html new file mode 100644 index 0000000000..8d6906f202 --- /dev/null +++ b/en/latest/sandbox/Sandbox11/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox11.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox11.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox11.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox11.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.109   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.109:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.109
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.109   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.109   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.109   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.109   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.110   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.110
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.110
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.110
+
+SRV05-NYC
+curl 45.38.161.110
+
+SRV04-NYC
+curl 45.38.161.110
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1112
+  localIP: 45.38.161.118/30
+  remoteIP: 45.38.161.117/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.96/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.118/30   45.38.161.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.118/30 45.38.161.117/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.118/30   45.38.161.117/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.118/30   45.38.161.117/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.109
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox11/sandbox-info.html b/en/latest/sandbox/Sandbox11/sandbox-info.html new file mode 100644 index 0000000000..9852fef4b0 --- /dev/null +++ b/en/latest/sandbox/Sandbox11/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.211 -p 30061
+srv02-nyc: ssh demo@216.172.128.211 -p 30062
+srv03-nyc: ssh demo@216.172.128.211 -p 30063
+srv04-nyc: ssh demo@216.172.128.211 -p 30064
+srv05-nyc: ssh demo@216.172.128.211 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1111
+Local Address:                  45.38.161.114/30
+Remote Address:                 45.38.161.113/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.96/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1111
+Local Address:                  2607:f358:11:ffc0::17/127
+Remote Address:                 2607:f358:11:ffc0::16/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcb::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1112
+Local Address:                  45.38.161.118/30
+Remote Address:                 45.38.161.117/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.96/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.96/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.96/30
+|___ NAT Subnet:               45.38.161.100/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.104/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.108/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcb::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcb::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox12/configurations.html b/en/latest/sandbox/Sandbox12/configurations.html new file mode 100644 index 0000000000..cb131a4f05 --- /dev/null +++ b/en/latest/sandbox/Sandbox12/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://Sandbox12.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (45.38.161.1251) from the “45.38.161.125/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(1122)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@sandbox12 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox12/creating-services.html b/en/latest/sandbox/Sandbox12/creating-services.html new file mode 100644 index 0000000000..a6123ef009 --- /dev/null +++ b/en/latest/sandbox/Sandbox12/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@sandbox12 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://Sandbox12.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://Sandbox12.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1121 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.121.

    6. +
    7. In the Remote IP field, type in 45.38.161.126.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 1122/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.128)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@sandbox12 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://Sandbox12.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.129/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.129/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.129/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://Sandbox12.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.132/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.132”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.132” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.132) into the browser’s address bar or simply visit http://45.38.161.132/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.132 (name_45.38.161.132)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.132 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@sandbox12 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://Sandbox12.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox12/index.html b/en/latest/sandbox/Sandbox12/index.html new file mode 100644 index 0000000000..d78887024e --- /dev/null +++ b/en/latest/sandbox/Sandbox12/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox12 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox12/onprem-k8s.html b/en/latest/sandbox/Sandbox12/onprem-k8s.html new file mode 100644 index 0000000000..6142145119 --- /dev/null +++ b/en/latest/sandbox/Sandbox12/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-Sandbox12.netris.io:6443
+CoreDNS is running at https://api.k8s-Sandbox12.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-Sandbox12.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://Sandbox12.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.140   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.140:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.140
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.140   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.140   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.140   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.140   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.141   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.141
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.141
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.141
+
+SRV05-NYC
+curl 45.38.161.141
+
+SRV04-NYC
+curl 45.38.161.141
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1121
+  localIP: 45.38.161.121/30
+  remoteIP: 45.38.161.126/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 1122/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.121/30   45.38.161.126/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.121/30 45.38.161.126/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.121/30   45.38.161.126/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.121/30   45.38.161.126/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.140
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox12/sandbox-info.html b/en/latest/sandbox/Sandbox12/sandbox-info.html new file mode 100644 index 0000000000..81001ea5ac --- /dev/null +++ b/en/latest/sandbox/Sandbox12/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@sandbox12 -p 30061
+srv02-nyc: ssh demo@sandbox12 -p 30062
+srv03-nyc: ssh demo@sandbox12 -p 30063
+srv04-nyc: ssh demo@sandbox12 -p 30064
+srv05-nyc: ssh demo@sandbox12 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        216.172.128.212
+Local Address:                  45.38.161.142/30
+Remote Address:                 45.38.161.122/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 1122/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        216.172.128.212
+Local Address:                  2607:f358:11:ffcc::1/127
+Remote Address:                 2607:f358:11:ffc0::19/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 45.38.161.125/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1121
+Local Address:                  45.38.161.121/30
+Remote Address:                 45.38.161.126/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 1122/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      1122/28
+|___ PUBLIC LOOPBACK Subnet:   1122/30
+|___ NAT Subnet:               45.38.161.129/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.132/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.136/30
+
+| EXAMPLE IPv6 Allocation:     45.38.161.125/64
+|___ EXAMPLE IPv6 Subnet:      45.38.161.125/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox13/configurations.html b/en/latest/sandbox/Sandbox13/configurations.html new file mode 100644 index 0000000000..f0773a8570 --- /dev/null +++ b/en/latest/sandbox/Sandbox13/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox13.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffcd::1) from the “2607:f358:11:ffcd::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.144)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.213 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox13/creating-services.html b/en/latest/sandbox/Sandbox13/creating-services.html new file mode 100644 index 0000000000..41f7bf5d3a --- /dev/null +++ b/en/latest/sandbox/Sandbox13/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.213 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1132 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.166.

    6. +
    7. In the Remote IP field, type in 45.38.161.165.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.144/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.145)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.213 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.148/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.148/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.148/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.152/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.152”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.152” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.152) into the browser’s address bar or simply visit http://45.38.161.152/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.152 (name_45.38.161.152)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.152 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.213 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox13.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox13/index.html b/en/latest/sandbox/Sandbox13/index.html new file mode 100644 index 0000000000..c182239257 --- /dev/null +++ b/en/latest/sandbox/Sandbox13/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox13 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox13/onprem-k8s.html b/en/latest/sandbox/Sandbox13/onprem-k8s.html new file mode 100644 index 0000000000..38b79dcdfc --- /dev/null +++ b/en/latest/sandbox/Sandbox13/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox13.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox13.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox13.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox13.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.157   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.157:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.157
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.157   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.157   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.157   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.157   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.158   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.158
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.158
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.158
+
+SRV05-NYC
+curl 45.38.161.158
+
+SRV04-NYC
+curl 45.38.161.158
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1132
+  localIP: 45.38.161.166/30
+  remoteIP: 45.38.161.165/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.144/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.166/30   45.38.161.165/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.166/30 45.38.161.165/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.166/30   45.38.161.165/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.166/30   45.38.161.165/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.157
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox13/sandbox-info.html b/en/latest/sandbox/Sandbox13/sandbox-info.html new file mode 100644 index 0000000000..5beb60618f --- /dev/null +++ b/en/latest/sandbox/Sandbox13/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.213 -p 30061
+srv02-nyc: ssh demo@216.172.128.213 -p 30062
+srv03-nyc: ssh demo@216.172.128.213 -p 30063
+srv04-nyc: ssh demo@216.172.128.213 -p 30064
+srv05-nyc: ssh demo@216.172.128.213 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1131
+Local Address:                  45.38.161.162/30
+Remote Address:                 45.38.161.161/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.144/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1131
+Local Address:                  2607:f358:11:ffc0::1b/127
+Remote Address:                 2607:f358:11:ffc0::1a/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcd::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1132
+Local Address:                  45.38.161.166/30
+Remote Address:                 45.38.161.165/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.144/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.144/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.144/30
+|___ NAT Subnet:               45.38.161.148/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.152/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.156/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcd::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcd::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox14/configurations.html b/en/latest/sandbox/Sandbox14/configurations.html new file mode 100644 index 0000000000..2f9538dee4 --- /dev/null +++ b/en/latest/sandbox/Sandbox14/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox15.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffce::1) from the “2607:f358:11:ffce::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.176)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.214 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox14/creating-services.html b/en/latest/sandbox/Sandbox14/creating-services.html new file mode 100644 index 0000000000..944dcf42dd --- /dev/null +++ b/en/latest/sandbox/Sandbox14/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.214 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1142 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.174.

    6. +
    7. In the Remote IP field, type in 45.38.161.173.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.176/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.177)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.214 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.180/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.180/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.180/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.184/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.184”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.184” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.184) into the browser’s address bar or simply visit http://45.38.161.184/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.184 (name_45.38.161.184)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.184 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.214 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox14/index.html b/en/latest/sandbox/Sandbox14/index.html new file mode 100644 index 0000000000..bee6d211a1 --- /dev/null +++ b/en/latest/sandbox/Sandbox14/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox14 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox14/onprem-k8s.html b/en/latest/sandbox/Sandbox14/onprem-k8s.html new file mode 100644 index 0000000000..bec6ae30dd --- /dev/null +++ b/en/latest/sandbox/Sandbox14/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox15.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox15.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.189   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.189:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.189
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.189   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.189   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.189   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.189   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.190   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.190
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.190
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.190
+
+SRV05-NYC
+curl 45.38.161.190
+
+SRV04-NYC
+curl 45.38.161.190
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1142
+  localIP: 45.38.161.174/30
+  remoteIP: 45.38.161.173/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.176/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.174/30   45.38.161.173/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.174/30 45.38.161.173/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.174/30   45.38.161.173/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.174/30   45.38.161.173/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.189
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox14/sandbox-info.html b/en/latest/sandbox/Sandbox14/sandbox-info.html new file mode 100644 index 0000000000..fbd3f7e43a --- /dev/null +++ b/en/latest/sandbox/Sandbox14/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.214 -p 30061
+srv02-nyc: ssh demo@216.172.128.214 -p 30062
+srv03-nyc: ssh demo@216.172.128.214 -p 30063
+srv04-nyc: ssh demo@216.172.128.214 -p 30064
+srv05-nyc: ssh demo@216.172.128.214 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1141
+Local Address:                  45.38.161.170/30
+Remote Address:                 45.38.161.169/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.176/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1141
+Local Address:                  2607:f358:11:ffc0::1d/127
+Remote Address:                 2607:f358:11:ffc0::1c/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffce::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1142
+Local Address:                  45.38.161.174/30
+Remote Address:                 45.38.161.173/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.176/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.176/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.176/30
+|___ NAT Subnet:               45.38.161.180/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.184/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.188/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffce::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffce::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox15/configurations.html b/en/latest/sandbox/Sandbox15/configurations.html new file mode 100644 index 0000000000..2e493815e5 --- /dev/null +++ b/en/latest/sandbox/Sandbox15/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox15.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffcf::1) from the “2607:f358:11:ffcf::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.192)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.215 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox15/creating-services.html b/en/latest/sandbox/Sandbox15/creating-services.html new file mode 100644 index 0000000000..9129f9530c --- /dev/null +++ b/en/latest/sandbox/Sandbox15/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.215 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1152 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.214.

    6. +
    7. In the Remote IP field, type in 45.38.161.213.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.192/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.193)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.215 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.196/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.196/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.196/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.200/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.200”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.200” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.200) into the browser’s address bar or simply visit http://45.38.161.200/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.200 (name_45.38.161.200)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.200 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.215 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox15.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox15/index.html b/en/latest/sandbox/Sandbox15/index.html new file mode 100644 index 0000000000..407907788a --- /dev/null +++ b/en/latest/sandbox/Sandbox15/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox15 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox15/onprem-k8s.html b/en/latest/sandbox/Sandbox15/onprem-k8s.html new file mode 100644 index 0000000000..556ead971f --- /dev/null +++ b/en/latest/sandbox/Sandbox15/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox15.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox15.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox15.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.205   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.205:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.205
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.205   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.205   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.205   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.205   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.206   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.206
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.206
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.206
+
+SRV05-NYC
+curl 45.38.161.206
+
+SRV04-NYC
+curl 45.38.161.206
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1152
+  localIP: 45.38.161.214/30
+  remoteIP: 45.38.161.213/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.192/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.214/30   45.38.161.213/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.214/30 45.38.161.213/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.214/30   45.38.161.213/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.214/30   45.38.161.213/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.205
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox15/sandbox-info.html b/en/latest/sandbox/Sandbox15/sandbox-info.html new file mode 100644 index 0000000000..20756ab7e6 --- /dev/null +++ b/en/latest/sandbox/Sandbox15/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.215 -p 30061
+srv02-nyc: ssh demo@216.172.128.215 -p 30062
+srv03-nyc: ssh demo@216.172.128.215 -p 30063
+srv04-nyc: ssh demo@216.172.128.215 -p 30064
+srv05-nyc: ssh demo@216.172.128.215 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1151
+Local Address:                  45.38.161.210/30
+Remote Address:                 45.38.161.209/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.192/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1151
+Local Address:                  2607:f358:11:ffc0::1f/127
+Remote Address:                 2607:f358:11:ffc0::1e/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffcf::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1152
+Local Address:                  45.38.161.214/30
+Remote Address:                 45.38.161.213/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.192/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.192/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.192/30
+|___ NAT Subnet:               45.38.161.196/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.200/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.204/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffcf::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffcf::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox2/configurations.html b/en/latest/sandbox/Sandbox2/configurations.html new file mode 100644 index 0000000000..77cba1f9cc --- /dev/null +++ b/en/latest/sandbox/Sandbox2/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox2.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc2::1) from the “2607:f358:11:ffc2::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.32)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.202 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox2/creating-services.html b/en/latest/sandbox/Sandbox2/creating-services.html new file mode 100644 index 0000000000..5f4dcce099 --- /dev/null +++ b/en/latest/sandbox/Sandbox2/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.202 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1022 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.30.

    6. +
    7. In the Remote IP field, type in 45.38.161.29.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.32/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.33)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.202 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.36/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.36/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.36/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.40/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.40”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.40” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.40) into the browser’s address bar or simply visit http://45.38.161.40/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.40 (name_45.38.161.40)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.40 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.202 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox2.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox2/index.html b/en/latest/sandbox/Sandbox2/index.html new file mode 100644 index 0000000000..416d056095 --- /dev/null +++ b/en/latest/sandbox/Sandbox2/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox2 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox2/onprem-k8s.html b/en/latest/sandbox/Sandbox2/onprem-k8s.html new file mode 100644 index 0000000000..5bac756bc3 --- /dev/null +++ b/en/latest/sandbox/Sandbox2/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox2.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox2.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox2.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox2.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.45   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.45:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.45
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.45   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.45   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.45   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.45   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.46   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.46
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.46
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.46
+
+SRV05-NYC
+curl 45.38.161.46
+
+SRV04-NYC
+curl 45.38.161.46
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1022
+  localIP: 45.38.161.30/30
+  remoteIP: 45.38.161.29/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.32/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.30/30   45.38.161.29/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.30/30 45.38.161.29/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.30/30   45.38.161.29/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.30/30   45.38.161.29/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.45
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox2/sandbox-info.html b/en/latest/sandbox/Sandbox2/sandbox-info.html new file mode 100644 index 0000000000..a519d2f289 --- /dev/null +++ b/en/latest/sandbox/Sandbox2/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.202 -p 30061
+srv02-nyc: ssh demo@216.172.128.202 -p 30062
+srv03-nyc: ssh demo@216.172.128.202 -p 30063
+srv04-nyc: ssh demo@216.172.128.202 -p 30064
+srv05-nyc: ssh demo@216.172.128.202 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1021
+Local Address:                  45.38.161.26/30
+Remote Address:                 45.38.161.25/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.32/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1021
+Local Address:                  2607:f358:11:ffc0::5/127
+Remote Address:                 2607:f358:11:ffc0::4/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc2::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1022
+Local Address:                  45.38.161.30/30
+Remote Address:                 45.38.161.29/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.32/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.32/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.32/30
+|___ NAT Subnet:               45.38.161.36/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.40/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.44/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc2::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc2::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox3/configurations.html b/en/latest/sandbox/Sandbox3/configurations.html new file mode 100644 index 0000000000..6f01ec8b0b --- /dev/null +++ b/en/latest/sandbox/Sandbox3/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox3.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc3::1) from the “2607:f358:11:ffc3::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.48)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.203 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox3/creating-services.html b/en/latest/sandbox/Sandbox3/creating-services.html new file mode 100644 index 0000000000..1a729f60b7 --- /dev/null +++ b/en/latest/sandbox/Sandbox3/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.203 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1032 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.70.

    6. +
    7. In the Remote IP field, type in 45.38.161.69.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.48/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.49)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.203 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.52/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.52/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.52/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.56/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.56”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.56” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.56) into the browser’s address bar or simply visit http://45.38.161.56/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.56 (name_45.38.161.56)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.56 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.203 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox3.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox3/index.html b/en/latest/sandbox/Sandbox3/index.html new file mode 100644 index 0000000000..4c968b65d7 --- /dev/null +++ b/en/latest/sandbox/Sandbox3/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox3 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox3/onprem-k8s.html b/en/latest/sandbox/Sandbox3/onprem-k8s.html new file mode 100644 index 0000000000..1f606d8370 --- /dev/null +++ b/en/latest/sandbox/Sandbox3/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox3.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox3.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox3.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox3.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.61   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.61:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.61
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.61   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.61   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.61   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.61   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.62   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.62
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.62
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.62
+
+SRV05-NYC
+curl 45.38.161.62
+
+SRV04-NYC
+curl 45.38.161.62
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1032
+  localIP: 45.38.161.70/30
+  remoteIP: 45.38.161.69/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.48/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.70/30   45.38.161.69/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.70/30 45.38.161.69/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.70/30   45.38.161.69/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.70/30   45.38.161.69/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.61
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox3/sandbox-info.html b/en/latest/sandbox/Sandbox3/sandbox-info.html new file mode 100644 index 0000000000..7702e58d51 --- /dev/null +++ b/en/latest/sandbox/Sandbox3/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.203 -p 30061
+srv02-nyc: ssh demo@216.172.128.203 -p 30062
+srv03-nyc: ssh demo@216.172.128.203 -p 30063
+srv04-nyc: ssh demo@216.172.128.203 -p 30064
+srv05-nyc: ssh demo@216.172.128.203 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1031
+Local Address:                  45.38.161.66/30
+Remote Address:                 45.38.161.65/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.48/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1031
+Local Address:                  2607:f358:11:ffc0::7/127
+Remote Address:                 2607:f358:11:ffc0::6/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc3::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1032
+Local Address:                  45.38.161.70/30
+Remote Address:                 45.38.161.69/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.48/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.48/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.48/30
+|___ NAT Subnet:               45.38.161.52/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.56/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.60/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc3::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc3::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox4/configurations.html b/en/latest/sandbox/Sandbox4/configurations.html new file mode 100644 index 0000000000..142b3dd52d --- /dev/null +++ b/en/latest/sandbox/Sandbox4/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox4.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc4::1) from the “2607:f358:11:ffc4::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(45.38.161.80)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.204 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox4/creating-services.html b/en/latest/sandbox/Sandbox4/creating-services.html new file mode 100644 index 0000000000..0dffa371ad --- /dev/null +++ b/en/latest/sandbox/Sandbox4/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.204 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1042 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 45.38.161.78.

    6. +
    7. In the Remote IP field, type in 45.38.161.77.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 45.38.161.80/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(45.38.161.81)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.204 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “45.38.161.84/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “45.38.161.84/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 45.38.161.84/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “45.38.161.88/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “45.38.161.88”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “45.38.161.88” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (45.38.161.88) into the browser’s address bar or simply visit http://45.38.161.88/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “45.38.161.88 (name_45.38.161.88)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 45.38.161.88 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.204 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox4.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox4/index.html b/en/latest/sandbox/Sandbox4/index.html new file mode 100644 index 0000000000..314ec23e75 --- /dev/null +++ b/en/latest/sandbox/Sandbox4/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox4 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox4/onprem-k8s.html b/en/latest/sandbox/Sandbox4/onprem-k8s.html new file mode 100644 index 0000000000..eae9f1f576 --- /dev/null +++ b/en/latest/sandbox/Sandbox4/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox4.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox4.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox4.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox4.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   45.38.161.93   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 45.38.161.93:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 45.38.161.93
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.93   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.93   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   45.38.161.93   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   45.38.161.93   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   45.38.161.94   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 45.38.161.94
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 45.38.161.94
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 45.38.161.94
+
+SRV05-NYC
+curl 45.38.161.94
+
+SRV04-NYC
+curl 45.38.161.94
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1042
+  localIP: 45.38.161.78/30
+  remoteIP: 45.38.161.77/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 45.38.161.80/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         45.38.161.78/30   45.38.161.77/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 45.38.161.78/30 45.38.161.77/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         45.38.161.78/30   45.38.161.77/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         45.38.161.78/30   45.38.161.77/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 45.38.161.93
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox4/sandbox-info.html b/en/latest/sandbox/Sandbox4/sandbox-info.html new file mode 100644 index 0000000000..e053f5d28d --- /dev/null +++ b/en/latest/sandbox/Sandbox4/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.204 -p 30061
+srv02-nyc: ssh demo@216.172.128.204 -p 30062
+srv03-nyc: ssh demo@216.172.128.204 -p 30063
+srv04-nyc: ssh demo@216.172.128.204 -p 30064
+srv05-nyc: ssh demo@216.172.128.204 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1041
+Local Address:                  45.38.161.74/30
+Remote Address:                 45.38.161.73/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.80/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1041
+Local Address:                  2607:f358:11:ffc0::9/127
+Remote Address:                 2607:f358:11:ffc0::8/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc4::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1042
+Local Address:                  45.38.161.78/30
+Remote Address:                 45.38.161.77/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 45.38.161.80/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      45.38.161.80/28
+|___ PUBLIC LOOPBACK Subnet:   45.38.161.80/30
+|___ NAT Subnet:               45.38.161.84/30
+|___ L3 LOAD BALANCER Subnet:  45.38.161.88/30
+|___ L4 LOAD BALANCER Subnet:  45.38.161.92/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc4::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc4::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox5/configurations.html b/en/latest/sandbox/Sandbox5/configurations.html new file mode 100644 index 0000000000..7d7e66817d --- /dev/null +++ b/en/latest/sandbox/Sandbox5/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox5.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc5::1) from the “2607:f358:11:ffc5::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(50.117.59.128)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.205 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox5/creating-services.html b/en/latest/sandbox/Sandbox5/creating-services.html new file mode 100644 index 0000000000..e169068ddb --- /dev/null +++ b/en/latest/sandbox/Sandbox5/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.205 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1052 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.86.

    6. +
    7. In the Remote IP field, type in 50.117.59.85.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.128/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(50.117.59.129)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.205 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.132/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.132/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 50.117.59.132/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.136/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.136”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.136” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.136) into the browser’s address bar or simply visit http://50.117.59.136/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “50.117.59.136 (name_50.117.59.136)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.136 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.205 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox5.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox5/index.html b/en/latest/sandbox/Sandbox5/index.html new file mode 100644 index 0000000000..30d04bf47e --- /dev/null +++ b/en/latest/sandbox/Sandbox5/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox5 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox5/onprem-k8s.html b/en/latest/sandbox/Sandbox5/onprem-k8s.html new file mode 100644 index 0000000000..b302c62153 --- /dev/null +++ b/en/latest/sandbox/Sandbox5/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox5.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox5.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox5.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox5.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   50.117.59.141   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.141:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 50.117.59.141
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.141   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.141   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.141   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.141   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   50.117.59.142   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.142
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.142
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.142
+
+SRV05-NYC
+curl 50.117.59.142
+
+SRV04-NYC
+curl 50.117.59.142
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1052
+  localIP: 50.117.59.86/30
+  remoteIP: 50.117.59.85/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.128/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.86/30   50.117.59.85/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.86/30 50.117.59.85/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         50.117.59.86/30   50.117.59.85/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         50.117.59.86/30   50.117.59.85/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.141
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox5/sandbox-info.html b/en/latest/sandbox/Sandbox5/sandbox-info.html new file mode 100644 index 0000000000..1ea69a2901 --- /dev/null +++ b/en/latest/sandbox/Sandbox5/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.205 -p 30061
+srv02-nyc: ssh demo@216.172.128.205 -p 30062
+srv03-nyc: ssh demo@216.172.128.205 -p 30063
+srv04-nyc: ssh demo@216.172.128.205 -p 30064
+srv05-nyc: ssh demo@216.172.128.205 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1051
+Local Address:                  50.117.59.82/30
+Remote Address:                 50.117.59.81/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.128/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1051
+Local Address:                  2607:f358:11:ffc0::b/127
+Remote Address:                 2607:f358:11:ffc0::a/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc5::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1052
+Local Address:                  50.117.59.86/30
+Remote Address:                 50.117.59.85/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.128/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.128/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.128/30
+|___ NAT Subnet:               50.117.59.132/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.136/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.140/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc5::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc5::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox6/configurations.html b/en/latest/sandbox/Sandbox6/configurations.html new file mode 100644 index 0000000000..bc9d226594 --- /dev/null +++ b/en/latest/sandbox/Sandbox6/configurations.html @@ -0,0 +1,896 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+

Once you log into the Netris GUI, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

Once you log into the Netris GUI by visiting https://sandbox6.netris.io and navigating to Services > V-Net, you will find a V-Net service named “vnet-example” already configured for you as an example. You can examine the particular service settings by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “vnet-example” service.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Net > Looking Glass.

  2. +
  3. Select the sw01-nyc(10.254.46.1) switch from the Select device drop-down menu.

  4. +
  5. Select the Ping radio button from the row of available choices.

  6. +
  7. Type in 192.168.45.64 in the field labeled “IPv4 address”.

  8. +
  9. Click Submit.

  10. +
+

The result should look similar to the output below, indicating that the communication between the sw01-nyc switch and srv04-nyc server is working properly thanks to the configured V-Net service.

+
sw01-nyc# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=64 time=0.543 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=64 time=0.436 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=64 time=0.445 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=64 time=0.354 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=64 time=0.345 ms
+
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4074ms
+rtt min/avg/max/mdev = 0.345/0.424/0.543/0.075 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions in the “V-Net (Ethernet/Vlan/VXlan)” section in the “Learn by Creating Services” document.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Net > E-BGP. Here, aside from the system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and their respective adjacent spine switches ( sw01-nyc & sw02-nyc ), you will also find an E-BGP session named “iris-isp1-example” configured as example with IRIS ISP1. This ensures communication of the inside network with the Internet.

+

You can examine the particular session settings of the E-BGP connection by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “iris-isp1-example” connection. While viewing the settings, you may also expand the Advanced section located toward the bottom of the initial screen to able to see the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create a fault tolerant E-BGP session with IRIS ISP2 yourself, please refer to the step-by-step instructions in the “E-BGP (Exterior Border Gateway Protocol)” in the “Learn by Creating Services” document.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Net > NAT and you will find a NAT service named “NAT Example” configured as an example . The configured services ensures that there can be communication between the the private 192.168.45.0/24 network and the Internet.

+

You can examine the particular settings of the NAT service by clicking Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “NAT Example” service.

+

You may observe the functioning service by pinging the pubblic 1.1.1.1 IP address from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv04-nyc by typing ssh demo@216.172.128.206 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the public 1.1.1.1 IP address.

+

If you are interested in learning how to create a NAT services yourself, please refer to the step-by-step instructions in the “NAT (Network Address Translation)” section in the in the “Learn by Creating Services” document.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services > ACL and you will find an ACL services named “V-Net to WAN Example” set up as an example. This particular ACL ensures that the connectivity between the the private 192.168.45.0/24 network and the Internet is permitted through all protocols and ports, even in a scenario where the the Default Site Policy for the “US/NYC” site configured in the our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the “V-Net to WAN Example” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would allow or disallow particular connections. If you are interested in learning how to create a ACL policies yourself, please refer to the step-by-step instructions in the “ACL (Access Control List)” section in the in the “Learn by Creating Services” document.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox6/creating-services.html b/en/latest/sandbox/Sandbox6/creating-services.html new file mode 100644 index 0000000000..7caabdb64a --- /dev/null +++ b/en/latest/sandbox/Sandbox6/creating-services.html @@ -0,0 +1,979 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to the srv05-nyc server by typing ssh demo@216.172.128.206 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway by typing ping 192.168.46.1 and keep it running as an indicator for when the service becomes fully provisioned.

    8. +
    9. Until the service is provisioned, the received responses will indicate that the destination is not reachable in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Services > V-Net.

    2. +
    3. Click +Add to create a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select US/NYC.

    8. +
    9. Click +IPv4 Gateway, select subnet 192.168.46.0/24(CUSTOMER) and IP 192.168.46.1 to match the results of the ip route ls output on srv05-nyc.

    10. +
    11. Click +Port to define the port(s) to be included in the current V-Net service.

    12. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “srv05” in the Search field.

    • +
    +
      +
    1. Select the port named swp2(sw22-nyc-swp2 (srv05))@sw22-nyc (Admin), check the Untag check-box and click Add.

    2. +
    3. Click Save and the service will start provisioning.

    4. +
    +
  • +
+

Once fully provisioned, you will start seeing replies similar in form to “64 bytes from 192.168.46.1: icmp_seq=1 ttl=64 time=1.66 ms” to the ping previously started in the terminal window, indicating that now the gateway address is reachable.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Net > E-BGP.

    2. +
    3. Click +Add to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-customer).

    6. +
    7. From the Site drop-down menu, select US/NYC.

    8. +
    9. From the NFV Node drop-down menu, select SoftGate2.

    10. +
    11. In the Neighbor AS field, type in 65007.

    12. +
    13. In the Switch port field, define the port on the switch that is connected to the ISP2.

    14. +
    +
      +
    • For the purposes of this exercise, you can easily find the required port by typing “ISP2” in the Search field.

    • +
    +
      +
    1. Select the port named swp14(sw02-nyc-swp14 (ISP2))@sw02-nyc

    2. +
    3. For the VLAN ID field, unselect Untag and type in 1062.

    4. +
    5. In the Local IP field, type in 50.117.59.94

    6. +
    7. In the Remote IP field, type in 50.117.59.93.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.144/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Net > E-BGP page as well as on Telescope > Dashboard pages will turn green, indication a successfully established BGP session. .

+
+
+

NAT (Network Address Translation)

+

Now when we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@216.172.128.206 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running as an indicator for when the service starts to work.

    6. +
    +
  • +
+

Let’s configure a source NAT so our V-Net subnet 192.168.46.0/24 can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Net > NAT.

    2. +
    3. Click +Add to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Action drop-down menu, select SNAT.

    8. +
    9. From the Protocol drop-down menu, select ALL.

    10. +
    11. In the Source field, type in 192.168.46.0/24.

    12. +
    13. The Destination field can remain as 0.0.0.0/0.

    14. +
    15. From the Nat IP drop-down menu, select 50.117.59.150/32(US/NYC).

    16. +
    +
      +
    • This IP is from our sandbox address space and is indicated in the SoftGate configuration to be used as a global IP for NAT.

    • +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to srv05-nyc by typing ssh demo@216.172.128.206 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session by typing ping 1.1.1.1 and keep it running for the duration of this exercise.

    6. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris GUI by visiting https://sandbox6.netris.io and navigate to Net > Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots (⋮) on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from Permit to Deny.

    6. +
    7. Click Save.

    8. +
    +
  • +
  • Back in the terminal window:

  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command, indicating that the 1.1.1.1 IP address is no longer reachable.

+

Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser:

    +
      +
    1. Navigate to Services > ACL.

    2. +
    3. Click +Add to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net to WAN Customer).

    6. +
    7. From the Protocol drop-down menu, select ALL.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    15. Select Approve from the Actions menu indicated by three vertical dots (⋮) on the right side of the newly created “V-Net to WAN Example” ACL.

    16. +
    17. Click Approve one more time in the pop-up window.

    18. +
    +
  • +
  • Back in the terminal window again:

  • +
+

Once the Netris Controller has finished syncing the new ACL policy with all member devices, you can see that replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox6/index.html b/en/latest/sandbox/Sandbox6/index.html new file mode 100644 index 0000000000..a685d1663c --- /dev/null +++ b/en/latest/sandbox/Sandbox6/index.html @@ -0,0 +1,875 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox6 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox6/onprem-k8s.html b/en/latest/sandbox/Sandbox6/onprem-k8s.html new file mode 100644 index 0000000000..5c023070bc --- /dev/null +++ b/en/latest/sandbox/Sandbox6/onprem-k8s.html @@ -0,0 +1,1336 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris operations with Kubernetes

+ +
+

Intro

+

This Sandbox environment provides an existing Kubernetes cluster that has been deployed via Kubespray. For this scenario, we will be using the external LB option in Kubespray. A dedicated Netris L4LB service has been created in the Sandbox Controller to access the k8s apiservers from users and non-master nodes sides.

+../../_images/sandbox-l4lb-kubeapi.png +

To access the built-in Kubernetes cluster, put “Kubeconfig” file which you received by the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable export KUBECONFIG=~/Downloads/config on your local machine. After that try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

The output below means you’ve successfully connected to the sandbox cluster:

+
Kubernetes master is running at https://api.k8s-sandbox6.netris.io:6443
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/download/v0.4.7/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox6.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-576d5bf6bd-7z9jl   1/1     Running   0          49s
+pod/podinfo-576d5bf6bd-nhlmh   1/1     Running   0          33s
+
+NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
+service/podinfo      ClusterIP   172.21.65.106   <none>        9898/TCP,9999/TCP   50s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   <pending>     9898:32584/TCP,9999:30365/TCP   8m57s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+../../_images/sandbox-podinfo-prov.png +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
+podinfo      LoadBalancer   172.21.65.106   50.117.59.154   9898:32584/TCP,9999:30365/TCP   9m17s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.154:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+../../_images/sandbox-podinfo-ready.png +

Curl again, without specifying a port:

+
curl 50.117.59.154
+
+
+

The output is similar to this:

+
{
+ "hostname": "podinfo-576d5bf6bd-nhlmh",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK       33m
+podinfo-default-66d44feb-0278-412a-a32d-73afe011f2c6-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK       32m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-80     active   50.117.59.154   80/TCP     US/NYC   Admin    OK             2m17s
+podinfo-default-d07acd0f-51ea-429a-89dd-8e4c1d6d0a86-tcp-9999   active   50.117.59.154   9999/TCP   US/NYC   Admin    OK             3m47s
+srv04-5-nyc-http                                                active   50.117.59.155   80/TCP     US/NYC   Admin    Provisioning   6s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.155
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+../../_images/sandbox-l4lbs.png +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS         AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Admin   Provisioning   7s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.155
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV05-NYC
+curl 50.117.59.155
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+ name: vnet-customer
+ annotations:
+   resource.k8s.netris.ai/import: "true"
+spec:
+ ownerTenant: Admin
+ guestTenants: []
+ sites:
+   - name: US/NYC
+     gateways:
+       - 192.168.46.1/24
+     switchPorts:
+       - name: swp2@sw22-nyc
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   7s
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”. +Create a yaml file:

+
cat << EOF > isp2-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: isp2-customer
+spec:
+  site: US/NYC
+  softgate: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp14@sw02-nyc
+    vlanId: 1062
+  localIP: 50.117.59.94/30
+  remoteIP: 50.117.59.93/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.144/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f isp2-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME            STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled                            65007         50.117.59.94/30   50.117.59.93/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+
NAME            STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+isp2-customer   enabled   bgp: Established; prefix: 30; time: 00:00:51   UP           65007         50.117.59.94/30   50.117.59.93/30   2m3s
+
+
+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Net → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled, and asNumber is 64512 (it’s Calico default AS number):

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ asNumber: 64512
+ logSeverityScreen: Info
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:06:18   UP           65007         50.117.59.94/30   50.117.59.93/30    7m59s
+sandbox6-srv06-nyc-192.168.110.66   enabled                                                               4200070000    192.168.110.1/24   192.168.110.66/24   26s
+sandbox6-srv07-nyc-192.168.110.67   enabled                                                               4200070001    192.168.110.1/24   192.168.110.67/24   26s
+sandbox6-srv08-nyc-192.168.110.68   enabled                                                               4200070002    192.168.110.1/24   192.168.110.68/24   26s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                                STATE     BGP STATE                                      PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+isp2-customer                       enabled   bgp: Established; prefix: 28; time: 00:07:48   UP           65007         50.117.59.94/30   50.117.59.93/30    8m41s
+sandbox6-srv06-nyc-192.168.110.66   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070000    192.168.110.1/24   192.168.110.66/24   68s
+sandbox6-srv07-nyc-192.168.110.67   enabled   bgp: Established; prefix: 5; time: 00:00:19    N/A          4200070001    192.168.110.1/24   192.168.110.67/24   68s
+sandbox6-srv08-nyc-192.168.110.68   enabled   bgp: Established; prefix: 5; time: 00:00:44    N/A          4200070002    192.168.110.1/24   192.168.110.68/24   68s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: crd.projectcalico.org/v1
+kind: BGPConfiguration
+metadata:
+  annotations:
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  asNumber: 64512
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.154
+
+
+

Yes, it works:

+
{
+ "hostname": "podinfo-576d5bf6bd-mfpdt",
+ "version": "6.0.0",
+ "revision": "",
+ "color": "#34577c",
+ "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+ "message": "greetings from podinfo v6.0.0",
+ "goos": "linux",
+ "goarch": "amd64",
+ "runtime": "go1.16.5",
+ "num_goroutine": "8",
+ "num_cpu": "4"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations! You completed Milestone 2. Time to get yourself another iced coffee or even a beer depending on what time it is!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox6/sandbox-info.html b/en/latest/sandbox/Sandbox6/sandbox-info.html new file mode 100644 index 0000000000..738a5718ab --- /dev/null +++ b/en/latest/sandbox/Sandbox6/sandbox-info.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+

Netris sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you by email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+
+

Topology diagram

+../../_images/sandbox_topology.png +
+ +
+

Linux servers

+
+
Example pre-configured Netris services:
    +
  • srv01, srv02, srv03 & Netris Controller - are consuming a ROH (Routing On Host) Netris example service, see Services–>ROH.

  • +
  • srv01, srv02 - are behind Anycast L3 load balancer.

  • +
  • srv04, srv05 - are consuming a V-NET (routed VXLAN) Netris example service, see Services–>V-NET.

  • +
+
+
+

Accessing the Linux servers:

+
srv01: ssh demo@216.172.128.206 -p 30061
+srv02: ssh demo@216.172.128.206 -p 30062
+srv03: ssh demo@216.172.128.206 -p 30063
+srv04: ssh demo@216.172.128.206 -p 30064
+srv05: ssh demo@216.172.128.206 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing. +There is one pre-configured example under NET–>E-BGP service, which is advertising the public IP subnet to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured example)
+Name: iris-isp1-example
+Neighbor AS: 65007
+Vlan: 1061
+IP customer: 50.117.59.90/30
+IP Iris: 50.117.59.89/30
+
+Neighbor AS: 65007
+Vlan: 1062
+IP customer:  50.117.59.94/30
+IP Iris: 50.117.59.93/30
+
+
+
+
+

Networks used

+
Management subnet: 10.254.45.0/24
+Loopback subnet:   10.254.46.0/24
+Example subnet:    192.168.45.0/24
+Customer subnet:   192.168.46.0/24
+K8s subnet:        192.168.110.0/24
+Public subnet:     50.117.59.144/28
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox7/configurations.html b/en/latest/sandbox/Sandbox7/configurations.html new file mode 100644 index 0000000000..ee020d892a --- /dev/null +++ b/en/latest/sandbox/Sandbox7/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox7.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc7::1) from the “2607:f358:11:ffc7::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(50.117.59.160)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.207 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox7/creating-services.html b/en/latest/sandbox/Sandbox7/creating-services.html new file mode 100644 index 0000000000..74b82ff5dc --- /dev/null +++ b/en/latest/sandbox/Sandbox7/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.207 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1072 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.102.

    6. +
    7. In the Remote IP field, type in 50.117.59.101.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.160/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(50.117.59.161)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.207 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.164/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.164/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 50.117.59.164/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.168/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.168”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.168” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.168) into the browser’s address bar or simply visit http://50.117.59.168/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “50.117.59.168 (name_50.117.59.168)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.168 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.207 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox7.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox7/index.html b/en/latest/sandbox/Sandbox7/index.html new file mode 100644 index 0000000000..d219e3ef1d --- /dev/null +++ b/en/latest/sandbox/Sandbox7/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox7 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox7/onprem-k8s.html b/en/latest/sandbox/Sandbox7/onprem-k8s.html new file mode 100644 index 0000000000..b36ea3177b --- /dev/null +++ b/en/latest/sandbox/Sandbox7/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox7.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox7.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox7.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox7.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   50.117.59.173   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.173:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 50.117.59.173
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.173   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.173   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.173   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.173   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   50.117.59.174   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.174
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.174
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.174
+
+SRV05-NYC
+curl 50.117.59.174
+
+SRV04-NYC
+curl 50.117.59.174
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1072
+  localIP: 50.117.59.102/30
+  remoteIP: 50.117.59.101/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.160/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.102/30   50.117.59.101/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.102/30 50.117.59.101/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         50.117.59.102/30   50.117.59.101/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         50.117.59.102/30   50.117.59.101/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.173
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox7/sandbox-info.html b/en/latest/sandbox/Sandbox7/sandbox-info.html new file mode 100644 index 0000000000..ef08cf141e --- /dev/null +++ b/en/latest/sandbox/Sandbox7/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.207 -p 30061
+srv02-nyc: ssh demo@216.172.128.207 -p 30062
+srv03-nyc: ssh demo@216.172.128.207 -p 30063
+srv04-nyc: ssh demo@216.172.128.207 -p 30064
+srv05-nyc: ssh demo@216.172.128.207 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1071
+Local Address:                  50.117.59.98/30
+Remote Address:                 50.117.59.97/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.160/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1071
+Local Address:                  2607:f358:11:ffc0::f/127
+Remote Address:                 2607:f358:11:ffc0::e/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc7::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1072
+Local Address:                  50.117.59.102/30
+Remote Address:                 50.117.59.101/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.160/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.160/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.160/30
+|___ NAT Subnet:               50.117.59.164/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.168/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.172/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc7::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc7::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox8/configurations.html b/en/latest/sandbox/Sandbox8/configurations.html new file mode 100644 index 0000000000..21b06a8c67 --- /dev/null +++ b/en/latest/sandbox/Sandbox8/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox8.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc8::1) from the “2607:f358:11:ffc8::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(50.117.59.176)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.208 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox8/creating-services.html b/en/latest/sandbox/Sandbox8/creating-services.html new file mode 100644 index 0000000000..ff642aa75e --- /dev/null +++ b/en/latest/sandbox/Sandbox8/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.208 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1082 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.110.

    6. +
    7. In the Remote IP field, type in 50.117.59.109.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.176/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(50.117.59.177)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.208 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.180/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.180/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 50.117.59.180/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.184/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.184”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.184” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.184) into the browser’s address bar or simply visit http://50.117.59.184/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “50.117.59.184 (name_50.117.59.184)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.184 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.208 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox8.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox8/index.html b/en/latest/sandbox/Sandbox8/index.html new file mode 100644 index 0000000000..eea76943b1 --- /dev/null +++ b/en/latest/sandbox/Sandbox8/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox8 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox8/onprem-k8s.html b/en/latest/sandbox/Sandbox8/onprem-k8s.html new file mode 100644 index 0000000000..edb50d8074 --- /dev/null +++ b/en/latest/sandbox/Sandbox8/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox8.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox8.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox8.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox8.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   50.117.59.189   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.189:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 50.117.59.189
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.189   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.189   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.189   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.189   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   50.117.59.190   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.190
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.190
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.190
+
+SRV05-NYC
+curl 50.117.59.190
+
+SRV04-NYC
+curl 50.117.59.190
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1082
+  localIP: 50.117.59.110/30
+  remoteIP: 50.117.59.109/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.176/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.110/30   50.117.59.109/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.110/30 50.117.59.109/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         50.117.59.110/30   50.117.59.109/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         50.117.59.110/30   50.117.59.109/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.189
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox8/sandbox-info.html b/en/latest/sandbox/Sandbox8/sandbox-info.html new file mode 100644 index 0000000000..fcf2b4d1c7 --- /dev/null +++ b/en/latest/sandbox/Sandbox8/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.208 -p 30061
+srv02-nyc: ssh demo@216.172.128.208 -p 30062
+srv03-nyc: ssh demo@216.172.128.208 -p 30063
+srv04-nyc: ssh demo@216.172.128.208 -p 30064
+srv05-nyc: ssh demo@216.172.128.208 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1081
+Local Address:                  50.117.59.106/30
+Remote Address:                 50.117.59.105/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.176/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1081
+Local Address:                  2607:f358:11:ffc0::11/127
+Remote Address:                 2607:f358:11:ffc0::10/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc8::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1082
+Local Address:                  50.117.59.110/30
+Remote Address:                 50.117.59.109/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.176/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.176/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.176/30
+|___ NAT Subnet:               50.117.59.180/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.184/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.188/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc8::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc8::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox9/configurations.html b/en/latest/sandbox/Sandbox9/configurations.html new file mode 100644 index 0000000000..147b45af01 --- /dev/null +++ b/en/latest/sandbox/Sandbox9/configurations.html @@ -0,0 +1,913 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provided Example Configurations — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provided Example Configurations
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provided Example Configurations

+ +

Once you log into the Netris Controller, you will find that certain services have already been pre-configured for you to explore and interact with. You can also learn how to create some of these services yourself by following the step-by-step instructions in the “Learn by Creating Services” document.

+
+

V-Net (Ethernet/Vlan/VXlan) Example

+

To access the V-Net service example, first log into the Netris Controller by visiting https://sandbox9.netris.io and navigating to Services → V-Net, where you will find a pre-configured V-Net service named “vnet-example” available as an example.

+

To examine the service settings, select Edit from the Actions menu indicated by three vertical dots () on the right side of the “vnet-example” service, where you’ll see that the V-Net service is configured with VLAN ID 45 in order to enable EVPN Multihoming on the underlying switches.

+

You’ll also see that the V-Net service is configured with both an IPv4 gateway (192.168.45.1) from the “192.168.45.0/24 (EXAMPLE)” subnet and an IPv6 gateway (2607:f358:11:ffc9::1) from the “2607:f358:11:ffc9::/64 (EXAMPLE IPv6)” subnet.

+

Additionally, the V-Net service is configured to utilize network interfaces on both switches 21 and 22. Specifically, it is connected to swp4(swp4)@sw21-nyc (Admin) on switch 21 and swp4(swp4)@sw22-nyc (Admin) on switch 22.

+

You may also verify that the service is working properly from within the GUI: (*Fields not specified should remain unchanged and retain default values)

+
    +
  1. Navigate to Network → Looking Glass.

  2. +
  3. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  4. +
  5. Select “SoftGate1(50.117.59.192)” from the Hardware drop-down menu.

  6. +
  7. Leave the “Family: IPV4” as the selected choice from the Adrress Family drop-down menu.

  8. +
  9. Select “Ping” from the Command drop-down menu.

  10. +
  11. Leave the “Selecet IP address” as the selected choice from the Source drop-down menu.

  12. +
  13. Type 192.168.45.64 (the IP address configured on bond0.45 on srv04-nyc) in the field labeled IPv4 address.

  14. +
  15. Click Submit.

  16. +
+

The result should look similar to the output below, indicating that the communication between SoftGate SoftGate1 and server srv04-nyc is working properly thanks to the configured V-Net service.

+
SoftGate1# ping -c 5 192.168.45.64
+PING 192.168.45.64 (192.168.45.64) 56(84) bytes of data.
+64 bytes from 192.168.45.64: icmp_seq=1 ttl=61 time=6.29 ms
+64 bytes from 192.168.45.64: icmp_seq=2 ttl=61 time=5.10 ms
+64 bytes from 192.168.45.64: icmp_seq=3 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=4 ttl=61 time=4.82 ms
+64 bytes from 192.168.45.64: icmp_seq=5 ttl=61 time=4.79 ms
+--- 192.168.45.64 ping statistics ---
+5 packets transmitted, 5 received, 0% packet loss, time 4002ms
+rtt min/avg/max/mdev = 4.787/5.161/6.285/0.572 ms
+
+
+

If you are interested in learning how to create a V-Net service yourself, please refer to the step-by-step instructions found in the “V-Net (Ethernet/Vlan/VXlan)” section of the “Learn by Creating Services” document.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the V-Net” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol) Example

+

Navigate to Network → E-BGP. Here, aside from the required system generated IPv4/IPv6 E-BGP peer connections between the two border routers ( SoftGate1 & SoftGate2 ) and the rest of the switching fabric (which can be toggled on/off using the Show System Generated toggle at the top of the page), you will also find two E-BGP sessions named “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” configured as examples with IRIS ISP1. This ensures communication between the internal network and the Internet.

+

You may examine the particular session configurations of the E-BGP connections by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of either the “iris-isp1-ipv4-example” and “iris-isp1-ipv6-example” connections. You may also expand the Advanced section located toward the bottom of the Edit window to be able to access the more advanced settings available while configuring an E-BGP session.

+

If you are interested in learning how to create an additional E-BGP session with IRIS ISP2 in order to make the Sandbox upstream connections fault tolerant yourself, please refer to the step-by-step instructions found in the “E-BGP (Exterior Border Gateway Protocol)” section of the “Learn by Creating Services” document.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation) Example

+

Navigate to Network → NAT and you will find a NAT rule named “NAT Example” configured as an example for you. The configured “SNAT” rule ensures that there can be communication between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet.

+

You can examine the particular settings of the NAT rule by clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “NAT Example” service.

+

You may also observe the functioning NAT rule in action by pinging any public IP address (e.g. 1.1.1.1) from the srv04-nyc server.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv04-nyc: ssh demo@216.172.128.209 -p 30064.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping4 1.1.1.1

    6. +
    +
  • +
+

You will see replies in the form of “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.10 ms” indicating proper communication with the 1.1.1.1 public IP address.

+

If you are interested in learning how to create a NAT rule yourself, please refer to the step-by-step instructions found in the “NAT (Network Address Translation)” section of the “Learn by Creating Services” document.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

ACL (Access Control List) Example

+

Navigate to Services → ACL and you will find an ACL services named “V-Net Example to WAN” set up as an example for you. This particular ACL ensures that the connectivity between the the private “192.168.45.0/24 (EXAMPLE)” subnet and the Internet is permitted through all protocols and ports, even in a scenario where the the “ACL Default Policy” for the “US/NYC” site configured under Network → Sites in our Sandbox is changed from Permit to Deny.

+

You can examine the particular settings of this ACL policy by selecting Edit from the Actions menu indicated by three vertical dots () on the right side of the “V-Net Example to WAN” ACL policy.

+

By utilizing ACLs, you can impose granular controls and implement policies that would permit or deny particular connections of any complexity. If you are interested in learning how to create ACL policies yourself, please refer to the step-by-step instructions found in the “ACL (Access Control List)” section of the “Learn by Creating Services” document.

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox9/creating-services.html b/en/latest/sandbox/Sandbox9/creating-services.html new file mode 100644 index 0000000000..fabbd778f3 --- /dev/null +++ b/en/latest/sandbox/Sandbox9/creating-services.html @@ -0,0 +1,1062 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn by Creating Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn by Creating Services
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn by Creating Services

+ +

Following these short exercises we will be able to demonstrate how the Netris Controller, in conjunction with the Netris Agents deployed on the switches and SoftGates, is able to intelligently and automagically deploy the necessary configurations across the network fabric to provision the desired services within a matter of seconds.

+
+

V-Net (Ethernet/Vlan/VXlan)

+

Let’s create a V-Net service to give server srv05-nyc the ability to reach its gateway address.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.209 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Type ip route ls and we can see 192.168.46.1 is configured as the default gateway, indicated by the “default via 192.168.46.1 dev eth1 proto kernel onlink” line in the output.

    6. +
    7. Start a ping session towards the default gateway: ping 192.168.46.1

    8. +
    9. Keep the ping running as an indicator for when the service becomes fully provisioned.

    10. +
    11. Until the service is provisioned, we can see that the destination is not reachable judging by the outputs in the form of “From 192.168.46.64 icmp_seq=1 Destination Host Unreachable”.

    12. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Services → V-Net.

    2. +
    3. Click the + Add button in the top right corner of the page to get started with creating a new V-Net service.

    4. +
    5. Define a name in the Name field (e.g. vnet-customer).

    6. +
    7. From the Sites drop-down menu, select “US/NYC”.

    8. +
    9. From the VLAN ID drop-down menu, select “Enter manually” and type in “46” in the field to the right.

    10. +
    11. From the Owner drop-down menu, select “Demo”.

    12. +
    13. From the IPv4 Gateway drop-down menu, select the “192.168.46.0/24(CUSTOMER)” subnet.

    14. +
    15. The first available IP address “192.168.46.1” is automatically selected in the second drop-down menu of the list of IP addresses. This matches the results of the ip route ls command output on srv05-nyc we observed earlier.

    16. +
    17. From the Add Network Interface drop-down menu put a check mark next to both network interfaces “swp5(swp5 | srv05-nyc)@sw12-nyc (Demo)” and “swp5(swp5 | srv05-nyc)@sw21-nyc (Demo)”, which we can see are the interfaces where srv05-nyc is wired into when we reference the “Sandbox Topology diagram”.

    18. +
    +
    +
      +
    • The drop-down menu only contains these two network interfaces as they are the only interfaces that have been assigned to the Demo tenant.

    • +
    +
    +
      +
    1. Click the Add button.

    2. +
    3. Click the Add button at the bottom right of the “Add new V-Net” window and the service will start provisioning.

    4. +
    +
  • +
+

After just a few seconds, once fully provisioned, you will start seeing successful ping replies, similar in form to “64 bytes from 192.168.46.1: icmp_seq=55 ttl=64 time=1.66 ms”, to the ping that was previously started in the terminal window, indicating that now the gateway address is accessible from host srv05-nyc.

+

More details about V-Net (Ethernet/Vlan/VXlan) can be found on the the “V-Network” page.

+
+
+

E-BGP (Exterior Border Gateway Protocol)

+

Our internal network is already connected with the outside world so that our servers can communicate with the Internet through the E-BGP session with IRIS ISP1 named “iris-isp1-example”.

+

Optionally you can configure an E-BGP session to IRIS ISP2 for fault tolerance.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Network → E-BGP.

    2. +
    3. Click the + Add button in the top right corner of the page to configure a new E-BGP session.

    4. +
    5. Define a name in the Name field (e.g. iris-isp2-ipv4-customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the BGP Router drop-down menu, select “SoftGate2”.

    10. +
    11. From the Switch Port drop-down menu, select port “swp16(swp16 | ISP2)@sw02-nyc (Admin)” on the switch that is connected to the ISP2.

    12. +
    +
    +
      +
    • For the purposes of this exercise, the required switch port can easily be found by typing ISP2 in the Search field.

    • +
    +
    +
      +
    1. For the VLAN ID field, type in 1092 while leaving the Untagged check-box unchecked.

    2. +
    3. In the Neighbor AS field, type in 65007.

    4. +
    5. In the Local IP field, type in 50.117.59.118.

    6. +
    7. In the Remote IP field, type in 50.117.59.117.

    8. +
    9. Expand the Advanced section

    10. +
    11. In the Prefix List Outbound field, type in permit 50.117.59.192/28 le 32

    12. +
    13. And finally click Add

    14. +
    +
  • +
+

Allow up to 1 minute for both sides of the BGP sessions to come up and then the BGP state on Network → E-BGP page as well as on Telescope → Dashboard pages will turn green, indication a successfully established BGP session. We can glean further insight into the BGP session details by navigating to Net → Looking Glass.

+
+
    +
  1. Make sure “vpc-1:Default” is selected from the VPC drop-down menu.

  2. +
  3. Select “SoftGate2(50.117.59.193)” (the border router where our newly created BGP session is terminated on) from the Hardware drop-down menu.

  4. +
  5. Leaving the Address Family drop-down menu on “Family: IPV4” and the Command drop-down menu on “Command: BGP Summary”, click on the Submit button.

  6. +
+
+

We are presented with the summary of the BGP sessions terminated on SoftGate2. You can also click on each BGP neighbor name to further see the “Advertised routes” and “Routes” received to/from that BGP neighbor.

+

More details about E-BGP (Exterior Border Gateway Protocol) can be found on the the “BGP” page.

+
+
+

NAT (Network Address Translation)

+

Now that we have both internal and external facing services, we can aim for our srv05-nyc server to be able to communicate with the Internet.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.209 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session towards any public IP address (e.g. ping 1.1.1.1).

    6. +
    7. Keep the ping running as an indicator for when the service starts to work.

    8. +
    +
  • +
+

Let’s configure a Source NAT so our Customer subnet 192.168.46.0/24, which is used in the V-Net services called vnet-customer, can communicate with the Internet.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Network → NAT.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new NAT rule.

    4. +
    5. Define a name in the Name field (e.g. NAT Customer).

    6. +
    7. From the Site drop-down menu, select “US/NYC”.

    8. +
    9. From the Action drop-down menu, select “SNAT”.

    10. +
    11. Leave ALL selected in the Protocol drop-down menu.

    12. +
    13. In the Source Address field, type in 192.168.46.0/24.

    14. +
    15. In the Destination Address field, leave the default value of 0.0.0.0/0.

    16. +
    17. Toggle the switch from SNAT to Pool to SNAT to IP.

    18. +
    19. From the Select subnet drop-down menu, select the “50.117.59.196/30 (NAT)” subnet.

    20. +
    21. From the Select IP drop-down menu, select the “50.117.59.196/32” IP address.

    22. +
    +
    +
      +
    • This public IP address is part of 50.117.59.196/30 (NAT) subnet which is configured in the Network → IPAM section with the purpose of NAT and indicated in the SoftGate configurations to be used as a global IP for NAT by the “Netris SoftGate Agent”.

    • +
    +
    +
      +
    1. Click Add

    2. +
    +
  • +
+

Soon you will start seeing replies similar in form to “64 bytes from 1.1.1.1: icmp_seq=1 ttl=62 time=1.23 ms” to the ping previously started in the terminal window, indicating that now the Internet is reachable from srv05-nyc.

+

More details about NAT (Network Address Translation) can be found on the “NAT” page.

+
+
+

L3LB (Anycast L3 Load Balancer)

+

In this exercise we will quickly configure an Anycast IP address in the Netris Controller for two of our “ROH (Routing on the Host)” servers (srv01-nyc & srv02-nyc) which both have a running Web Server configured to display a simple HTML webpage and observe ECMP load balancing it in action.

+
    +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Services → ROH.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv01-nyc” server.

    4. +
    5. From the IPv4 drop-down menu, select the “50.117.59.200/30 (L3 LOAD BALANCER)” subnet.

    6. +
    7. From the second drop-down menu that appears to the right, select the first available IP “50.117.59.200”.

    8. +
    9. Check the Anycast check-box next to the previously selected IP and click the Save button.

    10. +
    11. Repeat steps 3 through 4 for “srv02-nyc” by first clicking Edit from the Actions menu indicated by three vertical dots () on the right side of the “srv02-nyc” server.

    12. +
    +
    +
      +
    • While editing “srv02-nyc”, after selecting the “50.117.59.200” IP address , the Anycast check-box will already be automatically checked as we had designated the IP address as such in step 5.

    • +
    +
    +
  • +
  • In a new web browser window/tab:

    +
      +
    1. Type in the Anycast IP address we just configured (50.117.59.200) into the browser’s address bar or simply visit http://50.117.59.200/.

    2. +
    3. Based on the unique hash calculated from factors such as source IP/Protocol/Port, the L3LB will use ECMP to load balance the traffic from your browser to either srv01-nyc or srv02-nyc, with the text on the website indicating where the traffic ended up.

    4. +
    +
    +
      +
    • It should be noted that the TCP session will continue to exist between the given end-user and server pair for the lifetime of the session. In our case we have landed on srv01-nyc.

    • +
    +
    +
  • +
+SRV01 L3LB +

In order to trigger the L3 load balancer to switch directing the traffic towards the other backend server (in this case from srv01-nyc to srv02-nyc, which based on the unique hash in your situation could be the other way around), we can simulate the unavailability of the backend server we ended up on by putting it in Maintenance mode.

+
    +
  • Back in the Netris Controller, navigate to Services → L3 Load Balancer.

    +
      +
    1. Expand the LB Vip that was created when we defined the Anycast IP address earlier by clicking on the > button to the left of “50.117.59.200 (name_50.117.59.200)”.

    2. +
    3. Click Action v to the right of the server you originally ended up on (in this case srv01-nyc).

    4. +
    5. Click Maintenance on.

    6. +
    7. Click Maintenance one more time in the pop-up window.

    8. +
    +
  • +
  • Back in the browser window/tab directed at the 50.117.59.200 Anycast IP address.

    +
      +
    1. After just a few seconds, we can observe that now the website indicates that the traffic is routed to srv02-nyc (once more, your case could be opposite for you based on the original hash).

    2. +
    +
  • +
+SRV02 L3LB +

More details about AL3LB (Anycast L3 load balancer) can be found on the “L3 Load Balancer (Anycast LB)” page.

+
+
+

ACL (Access Control List)

+

Now that srv05-nyc can communicate with both internal and external hosts, let’s check Access Policy and Control options.

+
    +
  • In a terminal window:

    +
      +
    1. SSH to server srv05-nyc: ssh demo@216.172.128.209 -p 30065.

    2. +
    3. Enter the password provided in the introductory e-mail.

    4. +
    5. Start a ping session: ping 1.1.1.1.

    6. +
    7. If the previous steps were followed, you should see successful ping replies in the form of “64 bytes from 1.1.1.1: icmp_seq=55 ttl=62 time=1.23 ms”.

    8. +
    9. Keep the ping running as an indicator for when the service starts to work.

    10. +
    +
  • +
  • In a web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Log into the Netris Controller by visiting https://sandbox9.netris.io and navigate to Network → Sites.

    2. +
    3. Click Edit from the Actions menu indicated by three vertical dots () on the right side of the UC/NYC site.

    4. +
    5. From the ACL Default Policy drop-down menu, change the value from “Permit” to “Deny”.

    6. +
    7. Click Save.

    8. +
    +
  • +
+

Soon you will notice that there are no new replies to our previously started ping 1.1.1.1 command in the terminal window, indicating that the 1.1.1.1 IP address is no longer reachable. Now that the Default ACL Policy is set to Deny, we need to configure an ACL entry that will allow the srv05-nyc server to communicate with the Internet.

+
    +
  • Back in the web browser: (*Fields not specified should remain unchanged and retain default values)

    +
      +
    1. Navigate to Services → ACL.

    2. +
    3. Click the + Add button in the top right corner of the page to define a new ACL.

    4. +
    5. Define a name in the Name field (e.g. V-Net Customer to WAN).

    6. +
    7. From the Protocol drop-down menu, select “ALL”.

    8. +
    9. In the Source field, type in 192.168.46.0/24.

    10. +
    11. In the Destination field, type in 0.0.0.0/0.

    12. +
    13. Click Add.

    14. +
    +
  • +
+

You can observer the status of the syncing process by clicking on the Syncing yellow label at the top right of the ACL windown. Once the Netris Controller has finished syncing the new ACL policy with all relevant member devices, the label will turn green and read as Synced. Back in the terminal window we can observer that the replies to our ping 1.1.1.1 command have resumed, indicating that the srv05-nyc server can communicate with the Internet once again..

+

More details about ACL (Access Control List) can be found on the “ACL” page.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox9/index.html b/en/latest/sandbox/Sandbox9/index.html new file mode 100644 index 0000000000..1eaaf3d8e9 --- /dev/null +++ b/en/latest/sandbox/Sandbox9/index.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Sandbox9 — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox9/onprem-k8s.html b/en/latest/sandbox/Sandbox9/onprem-k8s.html new file mode 100644 index 0000000000..116352831a --- /dev/null +++ b/en/latest/sandbox/Sandbox9/onprem-k8s.html @@ -0,0 +1,1350 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Learn Netris Operations with Kubernetes — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Learn Netris Operations with Kubernetes
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Learn Netris Operations with Kubernetes

+ +
+

Intro

+

The Sandbox environment offers a pre-existing 3 node Kubernetes cluster deployed through K3S in HA Mode. To enable user access to the Kubernetes API, a dedicated Netris L4LB service has been created in the Sandbox Controller. Furthermore, this L4LB address serves as K3S_URL environment variable for all nodes within the cluster.

+Sandbox L4LB KubeAPI +

To access the built-in Kubernetes cluster, put the “Kubeconfig” file which you received via the introductory email into your ~/.kube/config or set “KUBECONFIG” environment variable using export KUBECONFIG=~/Downloads/config on your local machine. Afterwards, try to connect to the k8s cluster:

+
kubectl cluster-info
+
+
+

If your output matches the one below, that means you’ve successfully connected to the Sandbox cluster:

+
Kubernetes control plane is running at https://api.k8s-sandbox9.netris.io:6443
+CoreDNS is running at https://api.k8s-sandbox9.netris.io:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
+Metrics-server is running at https://api.k8s-sandbox9.netris.io:6443/api/v1/namespaces/kube-system/services/https:metrics-server:https/proxy
+
+To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+
+
+
+
+

Install Netris Operator

+

The first step to integrate the Netris Controller with the Kubernetes API is to install the Netris Operator. Installation can be accomplished by installing a regular manifests or a helm chart. For this example we will use the Kubernetes regular manifests:

+
    +
  1. Install the latest Netris Operator:

  2. +
+
kubectl apply -f https://github.com/netrisai/netris-operator/releases/latest/download/netris-operator.yaml
+
+
+
    +
  1. Create credentials secret for Netris Operator:

  2. +
+
kubectl -nnetris-operator create secret generic netris-creds \
+--from-literal=host='https://sandbox9.netris.io' \
+--from-literal=login='demo' --from-literal=password='Your Demo user pass'
+
+
+
    +
  1. Inspect the pod logs and make sure the operator is connected to Netris Controller:

  2. +
+
kubectl -nnetris-operator logs -l netris-operator=controller-manager --all-containers -f
+
+
+

Example output demonstrating the successful operation of Netris Operator:

+
{"level":"info","ts":1629994653.6441543,"logger":"controller","msg":"Starting workers","reconcilerGroup":"k8s.netris.ai","reconcilerKind":"L4LB","controller":"l4lb","worker count":1}
+
+
+
+

Note

+

After installing the Netris Operator, your Kubernetes cluster and physical network control planes are connected.

+
+
+
+

Deploy an Application with an On-Demand Netris Load Balancer

+

In this scenario we will be installing a simple application that requires a network load balancer:

+

Install the application “Podinfo”:

+
kubectl apply -k github.com/stefanprodan/podinfo/kustomize
+
+
+

Get the list of pods and services in the default namespace:

+
kubectl get po,svc
+
+
+

As you can see, the service type is “ClusterIP”:

+
NAME                           READY   STATUS    RESTARTS   AGE
+pod/podinfo-7cf557d9d7-6gfwx   1/1     Running   0          34s
+pod/podinfo-7cf557d9d7-nb2t7   1/1     Running   0          18s
+
+NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)             AGE
+service/kubernetes   ClusterIP   10.43.0.1      <none>        443/TCP             33m
+service/podinfo      ClusterIP   10.43.68.103   <none>        9898/TCP,9999/TCP   35s
+
+
+

In order to request access from outside, change the type to “LoadBalancer”:

+
kubectl patch svc podinfo -p '{"spec":{"type":"LoadBalancer"}}'
+
+
+

Check the services again:

+
kubectl get svc
+
+
+

Now we can see that the service type has changed to LoadBalancer, and “EXTERNAL-IP” switched to pending state:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         37m
+podinfo      LoadBalancer   10.43.68.103   <pending>       9898:32486/TCP,9999:30455/TCP   3m45s
+
+
+

Going into the Netris Controller web interface, navigate to Services → L4 Load Balancer, and you may see L4LBs provisioning in real-time. If you do not see the provisioning process it is likely because it already completed. Look for the service with the name “podinfo-xxxxxxxx”

+Sandbox PodInfo Provisioning +

After provisioning has finished, let’s one more time look at service in k8s:

+
kubectl get svc
+
+
+

You can see that “EXTERNAL-IP” has been injected into Kubernetes:

+
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)                         AGE
+kubernetes   ClusterIP      10.43.0.1      <none>          443/TCP                         29m
+podinfo      LoadBalancer   10.43.42.190   50.117.59.205   9898:30771/TCP,9999:30510/TCP   5m14s
+
+
+

Let’s try to curl it (remember to replace the IP below with the IP that has been assigned in the previous command):

+
curl 50.117.59.205:9898
+
+
+

The application is now accessible directly on the internet:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

As seen, “PodInfo” developers decided to expose 9898 port for HTTP, let’s switch it to 80:

+
kubectl patch svc podinfo --type='json' -p='[{"op": "replace", "path": "/spec/ports/0/port", "value":80}]'
+
+
+

Wait a few seconds, you can see the provisioning process on the controller:

+Sandbox PodInfo Ready +

Curl again, without specifying a port:

+
curl 50.117.59.205
+
+
+

The output is similar to this:

+
{
+  "hostname": "podinfo-7cf557d9d7-6gfwx",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+

You can also verify the application is reachable by putting this IP address directly into your browser.

+
+

Milestone 1

+

Congratulations! You successfully deployed a network load balancer and exposed an application from your cloud to the internet. Time to get yourself an iced coffee.

+
+
+
+

Using Netris Custom Resources

+
+

Introduction to Netris Custom Resources

+

In addition to provisioning on-demand network load balancers, Netris Operator can also provide automatic creation of network services based on Kubernetes CRD objects. Let’s take a look at a few common examples:

+
+
+

L4LB Custom Resource

+

In the previous section, when we changed the service type from “ClusterIP” to “LoadBalancer”, Netris Operator detected a new request for a network load balancer, then it created L4LB custom resources. Let’s see them:

+
kubectl get l4lb
+
+
+

As you can see, there are two L4LB resources, one for each podinfo’s service port:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS   AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.205   80/TCP     US/NYC   Admin    OK       7m21s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.205   9999/TCP   US/NYC   Admin    OK       15m
+
+
+

You can’t edit/delete them, because Netris Operator will recreate them based on what was originally deployed in the service specifications.

+

Instead, let’s create a new load balancer using the CRD method. This method allows us to create L4 load balancers for services outside of what is being created natively with the Kubernetes service schema. Our new L4LB’s backends will be “srv04-nyc” & “srv05-nyc” on TCP port 80. These servers are already running the Nginx web server, with the hostname present in the index.html file.

+

Create a yaml file:

+
cat << EOF > srv04-5-nyc-http.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: L4LB
+metadata:
+ name: srv04-5-nyc-http
+spec:
+ ownerTenant: Admin
+ site: US/NYC
+ state: active
+ protocol: tcp
+ frontend:
+   port: 80
+ backend:
+   - 192.168.45.64:80
+   - 192.168.46.65:80
+ check:
+   type: tcp
+   timeout: 3000
+EOF
+
+
+

And apply it:

+
kubectl apply -f srv04-5-nyc-http.yaml
+
+
+

Inspect the new L4LB resources via kubectl:

+
kubectl get l4lb
+
+
+

As you can see, provisioning started:

+
NAME                                                            STATE    FRONTEND        PORT       SITE     TENANT   STATUS         AGE
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-80     active   50.117.59.205   80/TCP     US/NYC   Admin    OK             9m56s
+podinfo-default-5bdf0a53-027d-449f-8896-547e06028c6b-tcp-9999   active   50.117.59.205   9999/TCP   US/NYC   Admin    OK             17m
+srv04-5-nyc-http                                                active   50.117.59.206   80/TCP     US/NYC   Admin    Provisioning   5s
+
+
+

When provisioning is finished, you should be able to connect to L4LB. Try to curl, using the L4LB frontend address displayed in the above command output:

+
curl 50.117.59.206
+
+
+

You will see the servers’ hostname in curl output:

+
SRV04-NYC
+
+
+

You can also inspect the L4LB in the Netris Controller web interface:

+Sandbox L4LBs +
+
+

V-Net Custom Resource

+

If one of the backend health-checks is marked as unhealthy like in the screenshot above, it means you didn’t create “vnet-customer” V-Net as described in the “Learn by Creating Services” manual. If that’s the case, let’s create it from Kubernetes using the V-Net custom resource.

+

Let’s create our V-Net manifest:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

And apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

Let’s check our V-Net resources in Kubernetes:

+
kubectl get vnet
+
+
+

As you can see, provisioning for our new V-Net has started:

+
NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   10s
+
+
+

After provisioning has completed, the L4LB’s checks should work for both backend servers, and incoming requests should be balanced between them.

+

Let’s curl several times to see that:

+
curl 50.117.59.206
+
+
+

As we can see, the curl request shows the behavior of “round robin” between the backends:

+
SRV05-NYC
+curl 50.117.59.206
+
+SRV05-NYC
+curl 50.117.59.206
+
+SRV04-NYC
+curl 50.117.59.206
+
+SRV04-NYC
+
+
+
+

Note

+

If intermittently the result of the curl command is “Connection timed out”, it is likely that the request went to the srv05-nyc backend, and the “Default ACL Policy” is set to “Deny”. To remedy this, configure an ACL entry that will allow the srv05-nyc server to communicate with external addresses. For step-by-step instruction review the ACL documentation.

+
+

BTW, if you already created “vnet-customer” V-Net as described in the “Learn by Creating Services”, you may import that to k8s, by adding resource.k8s.netris.ai/import: "true" annotation in V-Net manifest, the manifest should look like this:

+
cat << EOF > vnet-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: VNet
+metadata:
+  name: vnet-customer
+  annotations:
+    resource.k8s.netris.ai/import: "true"
+spec:
+  ownerTenant: Demo
+  guestTenants: []
+  vlanId: "46"
+  sites:
+    - name: US/NYC
+      gateways:
+        - prefix: 192.168.46.1/24
+      switchPorts:
+        - name: swp5@sw12-nyc
+          untagged: "no"
+        - name: swp5@sw21-nyc
+          untagged: "no"
+EOF
+
+
+

Apply it:

+
kubectl apply -f vnet-customer.yaml
+
+
+

After applying the manifest containing “import” annotation, the V-Net, created from the Netris Controller web interface, will appear in k8s and you will be able to manage it from Kubernetes.

+
kubectl get vnet
+
+NAME            STATE    GATEWAYS          SITES    OWNER   STATUS   AGE
+vnet-customer   active   192.168.46.1/24   US/NYC   Demo    Active   2m
+
+
+
+
+

BGP Custom Resource

+

Let’s create a new BGP peer, that is listed in the “Learn by Creating Services”.

+

Create a yaml file:

+
cat << EOF > iris-isp2-ipv4-customer.yaml
+apiVersion: k8s.netris.ai/v1alpha1
+kind: BGP
+metadata:
+  name: iris-isp2-ipv4-customer
+spec:
+  site: US/NYC
+  hardware: SoftGate2
+  neighborAs: 65007
+  transport:
+    name: swp16@sw02-nyc
+    vlanId: 1092
+  localIP: 50.117.59.118/30
+  remoteIP: 50.117.59.117/30
+  description: Example BGP to ISP2
+  prefixListOutbound:
+    - permit 50.117.59.192/28 le 32
+EOF
+
+
+

And apply it:

+
kubectl apply -f iris-isp2-ipv4-customer.yaml
+
+
+

Check created BGP:

+
kubectl get bgp
+
+
+

Allow up to 1 minute for both sides of the BGP sessions to come up:

+
NAME                      STATE     BGP STATE   PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS     AGE
+iris-isp2-ipv4-customer   enabled               Link Up      65007         50.117.59.118/30   50.117.59.117/30   15s
+
+
+

Then check the state again:

+
kubectl get bgp
+
+
+

The output is similar to this:

+

+
+
+

NAME STATE BGP STATE PORT STATE NEIGHBOR AS LOCAL ADDRESS REMOTE ADDRESS AGE +iris-isp2-ipv4-customer enabled bgp: Established; prefix: 957240; time: 00:04:02 65007 50.117.59.118/30 50.117.59.117/30 2m3s

+

Feel free to use the import annotation for this BGP if you created it from the Netris Controller web interface previously.

+

Return to the Netris Controller and navigate to Network → Topology to see the new BGP neighbor you created.

+
+
+

Importing Existing Resources from Netris Controller to Kubernetes

+
+

You can import any custom resources already created from the Netris Controller to k8s by adding the following annotation:

+
+
resource.k8s.netris.ai/import: "true"
+
+
+

Otherwise, if try to apply them without the “import” annotation, the Netris Operator will complain that the resource with such name or specs already exists.

+

After importing resources to k8s, they will belong to the Netris Operator, and you won’t be able to edit/delete them directly from the Netris Controller web interface, because the Netris Operator will put everything back, as declared in the custom resources.

+
+
+

Reclaim Policy

+

There is also one useful annotation. So suppose you want to remove some custom resource from k8s, and want to prevent its deletion from the Netris Controller, for that you can use “reclaimPolicy” annotation:

+
resource.k8s.netris.ai/reclaimPolicy: "retain"
+
+
+

Just add this annotation in any custom resource while creating it. Or if the custom resource has already been created, change the "delete" value to "retain" for key resource.k8s.netris.ai/reclaimPolicy in the resource annotation. After that, you’ll be able to delete any Netris Custom Resource from Kubernetes, and it won’t be deleted from the Netris Controller.

+
+

See also

+

See all options and examples for Netris Custom Resources here.

+
+
+
+
+

Netris Calico CNI Integration

+

Netris Operator can integrate with Calico CNI, in your Sandbox k8s cluster, Calico has already been configured as the CNI, so you can try this integration. It will automatically create BGP peering between cluster nodes and the leaf/TOR switch for each node, then to clean up it will disable Calico Node-to-Node mesh. To understand why you need to configure peering between Kubernetes nodes and the leaf/TOR switch, and why you should disable Node-to-Node mesh, review the Calico docs.

+

Integration is very simple, you just need to add the annotation in calico’s bgpconfigurations custom resource. Before doing that, let’s see the current state of bgpconfigurations:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

As we can see, nodeToNodeMeshEnabled is enabled:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+ annotations:
+   ...
+ name: default
+ ...
+spec:
+ nodeToNodeMeshEnabled: true
+
+
+

Let’s enable the “netris-calico” integration:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico='true'
+
+
+

Let’s check our BGP resources in k8s:

+
kubectl get bgp
+
+
+

Here are our freshly created BGPs, one for each k8s node:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957241; time: 00:15:03                65007         50.117.59.118/30   50.117.59.117/30    16m
+sandbox-srv06-192.168.110.66   enabled                                                                   4230000000    192.168.110.1/24   192.168.110.66/24   37s
+sandbox-srv07-192.168.110.67   enabled                                                                   4230000001    192.168.110.1/24   192.168.110.67/24   37s
+sandbox-srv08-192.168.110.68   enabled                                                                   4230000002    192.168.110.1/24   192.168.110.68/24   37s
+
+
+

You might notice that peering neighbor AS is different from Calico’s default 64512. The is because the Netris Operator is setting a particular AS number for each node.

+

Allow up to 1 minute for the BGP sessions to come up, then check BGP resources again:

+
kubectl get bgp
+
+
+

As we can see, our BGP peers have become established:

+
NAME                           STATE     BGP STATE                                          PORT STATE   NEIGHBOR AS   LOCAL ADDRESS      REMOTE ADDRESS      AGE
+iris-isp2-ipv4-customer        enabled   bgp: Established; prefix: 957194; time: 00:18:24                65007         50.117.59.118/30   50.117.59.117/30    19m
+sandbox-srv06-192.168.110.66   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000000    192.168.110.1/24   192.168.110.66/24   2m7s
+sandbox-srv07-192.168.110.67   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000001    192.168.110.1/24   192.168.110.67/24   2m7s
+sandbox-srv08-192.168.110.68   enabled   bgp: Established; prefix: 1; time: 00:01:26        N/A          4230000002    192.168.110.1/24   192.168.110.68/24   2m7s
+
+
+

Now let’s check if nodeToNodeMeshEnabled is still enabled:

+
kubectl get bgpconfigurations default -o yaml
+
+
+

It is disabled, which means the “netris-calico” integration process is finished:

+
apiVersion: projectcalico.org/v3
+kind: BGPConfiguration
+metadata:
+  annotations:
+    ...
+    manage.k8s.netris.ai/calico: "true"
+    ...
+  name: default
+  ...
+spec:
+  nodeToNodeMeshEnabled: false
+
+
+
+

Note

+

Netris Operator won’t disable Node-to-Node mesh until all BGP peers of all the nodes in the k8s cluster become established.

+
+

Finally, let’s check if our earlier deployed “Podinfo” application is still working when Calico Node-to-Node mesh is disabled:

+
curl 50.117.59.205
+
+
+

Yes, it works:

+
{
+  "hostname": "podinfo-7cf557d9d7-nb2t7",
+  "version": "6.6.0",
+  "revision": "357009a86331a987811fefc11be1350058da33fc",
+  "color": "#34577c",
+  "logo": "https://raw.githubusercontent.com/stefanprodan/podinfo/gh-pages/cuddle_clap.gif",
+  "message": "greetings from podinfo v6.6.0",
+  "goos": "linux",
+  "goarch": "amd64",
+  "runtime": "go1.21.7",
+  "num_goroutine": "8",
+  "num_cpu": "2"
+}
+
+
+
+

Disabling Netris-Calico Integration

+

To disable “Netris-Calico” integration, delete the annotation from Calico’s bgpconfigurations resource:

+
kubectl annotate bgpconfigurations default manage.k8s.netris.ai/calico-
+
+
+

or change its value to "false".

+
+

Milestone 2

+

Congratulations on completing Milestone 2!

+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/sandbox/Sandbox9/sandbox-info.html b/en/latest/sandbox/Sandbox9/sandbox-info.html new file mode 100644 index 0000000000..bccee28fad --- /dev/null +++ b/en/latest/sandbox/Sandbox9/sandbox-info.html @@ -0,0 +1,963 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Welcome to Netris Sandbox — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Welcome to Netris Sandbox
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Welcome to Netris Sandbox

+ +

Netris Sandbox is a ready-to-use environment for testing Netris automatic NetOps. +We have pre-created some example services for you, details of which can be found in the “Provided Example Configurations” document. Feel free to view, edit, delete, and create new services. In case of any questions, reach out to us on Slack.

+

The credentials for the sandbox have been provided to you via email in response to your Sandbox request.

+

The Sandbox environment includes:

+
    +
  • Netris Controller: A cloud-hosted Netris Controller, loaded with examples.

  • +
  • Switching fabric: Two spine switches and four leaf switches, all operated by Netris.

  • +
  • SoftGates: Two SoftGate gateway nodes for border routing, L4 Load Balancing, site-to-site VPN, and NAT. Both operated by Netris.

  • +
  • Linux servers: Five Linux servers, with root access where you can run any applications for your tests.

  • +
  • Kubernetes cluster: A 3 node Kubernetes cluster, user integratable with Netris controller, feel free to deploy any applications for your tests.

  • +
  • ISP: Internet upstream with IRIS ISP, providing the sandbox Internet connectivity with real-world routable public IP addresses.

  • +
+ + +
+

Linux servers

+
+
Example pre-configured Netris services:
+
+
+

Accessing the Linux servers:

+
srv01-nyc: ssh demo@216.172.128.209 -p 30061
+srv02-nyc: ssh demo@216.172.128.209 -p 30062
+srv03-nyc: ssh demo@216.172.128.209 -p 30063
+srv04-nyc: ssh demo@216.172.128.209 -p 30064
+srv05-nyc: ssh demo@216.172.128.209 -p 30065
+
+
+
+
+

Kubernetes cluster

+

This Sandbox provides an up and running 3 node Kubernetes cluster. You can integrate it with the Netris Controller by installing the netris-operator. Step-by-step instructions are included in the “Learn Netris operations with Kubernetes” document.

+
+
+

Upstream ISP

+

This Sandbox also provides an upstream ISP service with real-world Internet routing configured through “BGP”. +There are two pre-configured examples under Network → E-BGP , one using IPv4 and the other using IPv6, which are advertising the public IP subnets belonging to the Sandbox to the upstream ISP IRIS.

+

ISP settings:

+
(pre-configured examples)
+Name:                           iris-isp1-ipv4-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1091
+Local Address:                  50.117.59.114/30
+Remote Address:                 50.117.59.113/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.192/28 le 32
+
+Name:                           iris-isp1-ipv6-example
+BGP Router:                     Softage1
+Switch Port:                    swp16@sw01-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1091
+Local Address:                  2607:f358:11:ffc0::13/127
+Remote Address:                 2607:f358:11:ffc0::12/127
+Prefix List Inbound:            permit ::/0
+Prefix List Outbound:           permit 2607:f358:11:ffc9::/64
+
+(configurable by you)
+BGP Router:                     Softage2
+Switch Port:                    swp16@sw02-nyc
+Neighbor AS:                    65007
+VLAN ID:                        1092
+Local Address:                  50.117.59.118/30
+Remote Address:                 50.117.59.117/30
+Prefix List Inbound:            permit 0.0.0.0/0
+Prefix List Outbound:           permit 50.117.59.192/28 le 32
+
+
+
+
+

Networks Used

+

Allocations and subnets defined under “IPAM”, see Network → IPAM.

+
| MANAGEMENT Allocation:       10.254.45.0/24
+|___ MANAGEMENT Subnet:        10.254.45.0/24
+
+| LOOPBACK Allocation:         10.254.46.0/24
+|___ LOOPBACK Subnet:          10.254.46.0/24
+
+| ROH Allocation:              192.168.44.0/24
+|___ ROH Subnet:               192.168.44.0/24
+
+| EXAMPLE Allocation:          192.168.45.0/24
+|___ EXAMPLE Subnet:           192.168.45.0/24
+
+| CUSTOMER Allocation:         192.168.46.0/24
+|___ CUSTOMER Subnet:          192.168.46.0/24
+
+| K8s Allocation:              192.168.110.0/24
+|___ K8s Subnet:               192.168.110.0/24
+
+| PUBLIC IPv4 Allocation:      50.117.59.192/28
+|___ PUBLIC LOOPBACK Subnet:   50.117.59.192/30
+|___ NAT Subnet:               50.117.59.196/30
+|___ L3 LOAD BALANCER Subnet:  50.117.59.200/30
+|___ L4 LOAD BALANCER Subnet:  50.117.59.204/30
+
+| EXAMPLE IPv6 Allocation:     2607:f358:11:ffc9::/64
+|___ EXAMPLE IPv6 Subnet:      2607:f358:11:ffc9::/64
+
+
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/search.html b/en/latest/search.html new file mode 100644 index 0000000000..a987568d20 --- /dev/null +++ b/en/latest/search.html @@ -0,0 +1,851 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Search — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Search
  • + + +
  • + +
  • + +
+ + +
+
+
+
+ + + + +
+ +
+ +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/searchindex.js b/en/latest/searchindex.js new file mode 100644 index 0000000000..431e2723be --- /dev/null +++ b/en/latest/searchindex.js @@ -0,0 +1 @@ +Search.setIndex({"docnames": ["Controller-local-repository", "Dell-SONiC-Switch-initial-setup", "EdgeCore-SONiC-Switch-initial-setup", "Nvidia-Cumulus-v3.7-Switch-initial-setup", "Nvidia-Cumulus-v5-Switch-initial-setup", "Nvidia-Cumulus-v5.9+-Switch-initial-setup", "SoftGate-PRO-installation", "SoftGate-installation", "Ubuntu-SwitchDev-Switch-initial-setup", "accounts", "acls", "controller-k3s-installation", "controller-k8s-installation", "controller-k8s-quickstart", "controller-vm-installation", "definitions", "index", "installation", "installing-netris-controller", "introduction", "inventory-profiles", "ipam", "kubernetes-integration", "l3-load-balancer", "l4-load-balancer", "lag", "maintenance-mode", "netris-architecture", "network-policies", "reference-designs", "release-notes", "roh", "sandbox/Sandbox1/configurations", "sandbox/Sandbox1/creating-services", "sandbox/Sandbox1/index", "sandbox/Sandbox1/onprem-k8s", "sandbox/Sandbox1/sandbox-info", "sandbox/Sandbox10/configurations", "sandbox/Sandbox10/creating-services", "sandbox/Sandbox10/index", "sandbox/Sandbox10/onprem-k8s", "sandbox/Sandbox10/sandbox-info", "sandbox/Sandbox11/configurations", "sandbox/Sandbox11/creating-services", "sandbox/Sandbox11/index", "sandbox/Sandbox11/onprem-k8s", "sandbox/Sandbox11/sandbox-info", "sandbox/Sandbox12/configurations", "sandbox/Sandbox12/creating-services", "sandbox/Sandbox12/index", "sandbox/Sandbox12/onprem-k8s", "sandbox/Sandbox12/sandbox-info", "sandbox/Sandbox13/configurations", "sandbox/Sandbox13/creating-services", "sandbox/Sandbox13/index", "sandbox/Sandbox13/onprem-k8s", "sandbox/Sandbox13/sandbox-info", "sandbox/Sandbox14/configurations", "sandbox/Sandbox14/creating-services", "sandbox/Sandbox14/index", "sandbox/Sandbox14/onprem-k8s", "sandbox/Sandbox14/sandbox-info", "sandbox/Sandbox15/configurations", "sandbox/Sandbox15/creating-services", "sandbox/Sandbox15/index", "sandbox/Sandbox15/onprem-k8s", "sandbox/Sandbox15/sandbox-info", "sandbox/Sandbox2/configurations", "sandbox/Sandbox2/creating-services", "sandbox/Sandbox2/index", "sandbox/Sandbox2/onprem-k8s", "sandbox/Sandbox2/sandbox-info", "sandbox/Sandbox3/configurations", "sandbox/Sandbox3/creating-services", "sandbox/Sandbox3/index", "sandbox/Sandbox3/onprem-k8s", "sandbox/Sandbox3/sandbox-info", "sandbox/Sandbox4/configurations", "sandbox/Sandbox4/creating-services", "sandbox/Sandbox4/index", "sandbox/Sandbox4/onprem-k8s", "sandbox/Sandbox4/sandbox-info", "sandbox/Sandbox5/configurations", "sandbox/Sandbox5/creating-services", "sandbox/Sandbox5/index", "sandbox/Sandbox5/onprem-k8s", "sandbox/Sandbox5/sandbox-info", "sandbox/Sandbox6/configurations", "sandbox/Sandbox6/creating-services", "sandbox/Sandbox6/index", "sandbox/Sandbox6/onprem-k8s", "sandbox/Sandbox6/sandbox-info", "sandbox/Sandbox7/configurations", "sandbox/Sandbox7/creating-services", "sandbox/Sandbox7/index", "sandbox/Sandbox7/onprem-k8s", "sandbox/Sandbox7/sandbox-info", "sandbox/Sandbox8/configurations", "sandbox/Sandbox8/creating-services", "sandbox/Sandbox8/index", "sandbox/Sandbox8/onprem-k8s", "sandbox/Sandbox8/sandbox-info", "sandbox/Sandbox9/configurations", "sandbox/Sandbox9/creating-services", "sandbox/Sandbox9/index", "sandbox/Sandbox9/onprem-k8s", "sandbox/Sandbox9/sandbox-info", "softgate-performance", "supported-networks", "supported-platform-matrix", "supported-switch-hardware", "switch-agent-installation", "switch-ports", "terraform-integration", "topology-management", "try-learn/index", "try-learn/netris-cloudsim", "try-learn/nvidia-spectrum-x-scenario", "try-learn/using-netris-cloudsim", "tutorials/activating-bgp-on-equinix-metal-project", "tutorials/adding-netris-softgate-nodes-in-equinix-metal", "tutorials/aws-concept", "tutorials/aws-deploy-softgate", "tutorials/connecting-fabric-to-isp", "tutorials/connecting-servers-fabric", "tutorials/create-interconnection-to-fabric", "tutorials/enable-services-on-equinix-metal-project", "tutorials/enabling-load-balancing-services", "tutorials/enabling-nat-services", "tutorials/equinix-metal-api-integration-enablement", "tutorials/gcp-concept", "tutorials/gcp-deploy-softgate", "tutorials/getting-started-for-equinix-metal", "tutorials/index", "tutorials/index-equinix", "tutorials/index-vpc", "tutorials/install-netris-controller-in-equinix-metal", "tutorials/installing-netris-controller", "tutorials/inventory-setup", "tutorials/ipam-setup", "tutorials/more-features", "tutorials/netris-controller-installation", "tutorials/netris-managed-fabric-overview", "tutorials/netris-switch-agent-installation", "tutorials/netris-vpc-for-aws", "tutorials/netris-vpc-for-equinix-metal", "tutorials/netris-vpc-for-gcp", "tutorials/netris-vpc-for-phoenixnap-bmc", "tutorials/new-site-setup", "tutorials/phoenixnap-bmc-adding-netris-softgate-nodes", "tutorials/phoenixnap-bmc-api-integration-enablement", "tutorials/phoenixnap-bmc-concept", "tutorials/phoenixnap-bmc-install-netris-controller", "tutorials/phoenixnap-bmc-ipam-setup", "tutorials/phoenixnap-bmc-link-to-installation", "tutorials/phoenixnap-bmc-using-l4lb", "tutorials/phoenixnap-bmc-using-nat", "tutorials/phoenixnap-bmc-using-vnet", "tutorials/softgate-software-provisioning", "tutorials/topology-setup", "tutorials/upgrading-netris", "tutorials/upgrading-sonic-os", "tutorials/using-l4-load-balancer", "tutorials/using-vnet-in-equinix-metal-project", "tutorials/vpc-anywhere", "tutorials/vpc-anywhere-check-default-site", "tutorials/vpc-anywhere-concept", "tutorials/vpc-anywhere-controller-installation", "tutorials/vpc-anywhere-ipam-setup", "tutorials/vpc-anywhere-softgate-installation", "tutorials/vpc-anywhere-upstream-peering", "tutorials/vpc-anywhere-using-l4lb", "tutorials/vpc-anywhere-using-multi-interface-softgate", "tutorials/vpc-anywhere-using-nat", "tutorials/vpc-anywhere-using-vnet", "tutorials/vpc-gateways-with-managed-fabric", "tutorials/vpc-setup", "visibility", "vnet", "vpc"], "filenames": ["Controller-local-repository.rst", "Dell-SONiC-Switch-initial-setup.rst", "EdgeCore-SONiC-Switch-initial-setup.rst", "Nvidia-Cumulus-v3.7-Switch-initial-setup.rst", "Nvidia-Cumulus-v5-Switch-initial-setup.rst", "Nvidia-Cumulus-v5.9+-Switch-initial-setup.rst", "SoftGate-PRO-installation.rst", "SoftGate-installation.rst", "Ubuntu-SwitchDev-Switch-initial-setup.rst", "accounts.rst", "acls.rst", "controller-k3s-installation.rst", "controller-k8s-installation.rst", "controller-k8s-quickstart.rst", "controller-vm-installation.rst", "definitions.rst", "index.rst", "installation.rst", "installing-netris-controller.rst", "introduction.rst", "inventory-profiles.rst", "ipam.rst", "kubernetes-integration.rst", "l3-load-balancer.rst", "l4-load-balancer.rst", "lag.rst", "maintenance-mode.rst", "netris-architecture.rst", "network-policies.rst", "reference-designs.rst", "release-notes.rst", "roh.rst", "sandbox/Sandbox1/configurations.rst", "sandbox/Sandbox1/creating-services.rst", "sandbox/Sandbox1/index.rst", "sandbox/Sandbox1/onprem-k8s.rst", "sandbox/Sandbox1/sandbox-info.rst", "sandbox/Sandbox10/configurations.rst", "sandbox/Sandbox10/creating-services.rst", "sandbox/Sandbox10/index.rst", "sandbox/Sandbox10/onprem-k8s.rst", "sandbox/Sandbox10/sandbox-info.rst", "sandbox/Sandbox11/configurations.rst", "sandbox/Sandbox11/creating-services.rst", "sandbox/Sandbox11/index.rst", "sandbox/Sandbox11/onprem-k8s.rst", "sandbox/Sandbox11/sandbox-info.rst", "sandbox/Sandbox12/configurations.rst", "sandbox/Sandbox12/creating-services.rst", "sandbox/Sandbox12/index.rst", "sandbox/Sandbox12/onprem-k8s.rst", "sandbox/Sandbox12/sandbox-info.rst", "sandbox/Sandbox13/configurations.rst", "sandbox/Sandbox13/creating-services.rst", "sandbox/Sandbox13/index.rst", "sandbox/Sandbox13/onprem-k8s.rst", "sandbox/Sandbox13/sandbox-info.rst", "sandbox/Sandbox14/configurations.rst", "sandbox/Sandbox14/creating-services.rst", "sandbox/Sandbox14/index.rst", "sandbox/Sandbox14/onprem-k8s.rst", "sandbox/Sandbox14/sandbox-info.rst", "sandbox/Sandbox15/configurations.rst", "sandbox/Sandbox15/creating-services.rst", "sandbox/Sandbox15/index.rst", "sandbox/Sandbox15/onprem-k8s.rst", "sandbox/Sandbox15/sandbox-info.rst", "sandbox/Sandbox2/configurations.rst", "sandbox/Sandbox2/creating-services.rst", "sandbox/Sandbox2/index.rst", "sandbox/Sandbox2/onprem-k8s.rst", "sandbox/Sandbox2/sandbox-info.rst", "sandbox/Sandbox3/configurations.rst", "sandbox/Sandbox3/creating-services.rst", "sandbox/Sandbox3/index.rst", "sandbox/Sandbox3/onprem-k8s.rst", "sandbox/Sandbox3/sandbox-info.rst", "sandbox/Sandbox4/configurations.rst", "sandbox/Sandbox4/creating-services.rst", "sandbox/Sandbox4/index.rst", "sandbox/Sandbox4/onprem-k8s.rst", "sandbox/Sandbox4/sandbox-info.rst", "sandbox/Sandbox5/configurations.rst", "sandbox/Sandbox5/creating-services.rst", "sandbox/Sandbox5/index.rst", "sandbox/Sandbox5/onprem-k8s.rst", "sandbox/Sandbox5/sandbox-info.rst", "sandbox/Sandbox6/configurations.rst", "sandbox/Sandbox6/creating-services.rst", "sandbox/Sandbox6/index.rst", "sandbox/Sandbox6/onprem-k8s.rst", "sandbox/Sandbox6/sandbox-info.rst", "sandbox/Sandbox7/configurations.rst", "sandbox/Sandbox7/creating-services.rst", "sandbox/Sandbox7/index.rst", "sandbox/Sandbox7/onprem-k8s.rst", "sandbox/Sandbox7/sandbox-info.rst", "sandbox/Sandbox8/configurations.rst", "sandbox/Sandbox8/creating-services.rst", "sandbox/Sandbox8/index.rst", "sandbox/Sandbox8/onprem-k8s.rst", "sandbox/Sandbox8/sandbox-info.rst", "sandbox/Sandbox9/configurations.rst", "sandbox/Sandbox9/creating-services.rst", "sandbox/Sandbox9/index.rst", "sandbox/Sandbox9/onprem-k8s.rst", "sandbox/Sandbox9/sandbox-info.rst", "softgate-performance.rst", "supported-networks.rst", "supported-platform-matrix.rst", "supported-switch-hardware.rst", "switch-agent-installation.rst", "switch-ports.rst", "terraform-integration.rst", "topology-management.rst", "try-learn/index.rst", "try-learn/netris-cloudsim.rst", "try-learn/nvidia-spectrum-x-scenario.rst", "try-learn/using-netris-cloudsim.rst", "tutorials/activating-bgp-on-equinix-metal-project.rst", "tutorials/adding-netris-softgate-nodes-in-equinix-metal.rst", "tutorials/aws-concept.rst", "tutorials/aws-deploy-softgate.rst", "tutorials/connecting-fabric-to-isp.rst", "tutorials/connecting-servers-fabric.rst", "tutorials/create-interconnection-to-fabric.rst", "tutorials/enable-services-on-equinix-metal-project.rst", "tutorials/enabling-load-balancing-services.rst", "tutorials/enabling-nat-services.rst", "tutorials/equinix-metal-api-integration-enablement.rst", "tutorials/gcp-concept.rst", "tutorials/gcp-deploy-softgate.rst", "tutorials/getting-started-for-equinix-metal.rst", "tutorials/index.rst", "tutorials/index-equinix.rst", "tutorials/index-vpc.rst", "tutorials/install-netris-controller-in-equinix-metal.rst", "tutorials/installing-netris-controller.rst", "tutorials/inventory-setup.rst", "tutorials/ipam-setup.rst", "tutorials/more-features.rst", "tutorials/netris-controller-installation.rst", "tutorials/netris-managed-fabric-overview.rst", "tutorials/netris-switch-agent-installation.rst", "tutorials/netris-vpc-for-aws.rst", "tutorials/netris-vpc-for-equinix-metal.rst", "tutorials/netris-vpc-for-gcp.rst", "tutorials/netris-vpc-for-phoenixnap-bmc.rst", "tutorials/new-site-setup.rst", "tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.rst", "tutorials/phoenixnap-bmc-api-integration-enablement.rst", "tutorials/phoenixnap-bmc-concept.rst", "tutorials/phoenixnap-bmc-install-netris-controller.rst", "tutorials/phoenixnap-bmc-ipam-setup.rst", "tutorials/phoenixnap-bmc-link-to-installation.rst", "tutorials/phoenixnap-bmc-using-l4lb.rst", "tutorials/phoenixnap-bmc-using-nat.rst", "tutorials/phoenixnap-bmc-using-vnet.rst", "tutorials/softgate-software-provisioning.rst", "tutorials/topology-setup.rst", "tutorials/upgrading-netris.rst", "tutorials/upgrading-sonic-os.rst", "tutorials/using-l4-load-balancer.rst", "tutorials/using-vnet-in-equinix-metal-project.rst", "tutorials/vpc-anywhere.rst", "tutorials/vpc-anywhere-check-default-site.rst", "tutorials/vpc-anywhere-concept.rst", "tutorials/vpc-anywhere-controller-installation.rst", "tutorials/vpc-anywhere-ipam-setup.rst", "tutorials/vpc-anywhere-softgate-installation.rst", "tutorials/vpc-anywhere-upstream-peering.rst", "tutorials/vpc-anywhere-using-l4lb.rst", "tutorials/vpc-anywhere-using-multi-interface-softgate.rst", "tutorials/vpc-anywhere-using-nat.rst", "tutorials/vpc-anywhere-using-vnet.rst", "tutorials/vpc-gateways-with-managed-fabric.rst", "tutorials/vpc-setup.rst", "visibility.rst", "vnet.rst", "vpc.rst"], "titles": ["Netris Local Repository Setup", "Dell SONiC Switch Initial Setup", "EdgeCore SONiC Switch Initial Setup", "Nvidia Cumulus v3.7 Switch Initial Setup", "Nvidia Cumulus v5 Switch Initial Setup", "Nvidia Cumulus v5.9+ Switch Initial Setup", "SoftGate PRO Installation", "SoftGate Installation", "Ubuntu SwitchDev Switch Initial Setup", "Accounts", "Access Control Lists (ACL)", "Netris Controller installation on a generic Linux host", "Helm Chart Installation", "Quickstart Installation", "Virtual Machine Installation", "Definitions", "Welcome to Netris Documentation", "Controller Installation", "Installing a Netris Controller", "Introduction to Netris", "Inventory Profiles", "IP Address Management (IPAM)", "Kubernetes Integration", "L3 Load Balancer (Anycast LB)", "L4 Load Balancer (L4LB)", "Link Aggregation (LAG)", "Maintenance Mode", "Netris Architecture", "VPC", "Network Reference Designs", "Release notes", "ROH (Routing on the Host)", "Provided Example Configurations", "Learn by Creating Services", "Sandbox1", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox10", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox11", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox12", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox13", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox14", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox15", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox2", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox3", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox4", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox5", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox6", "Learn Netris operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox7", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox8", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "Provided Example Configurations", "Learn by Creating Services", "Sandbox9", "Learn Netris Operations with Kubernetes", "Welcome to Netris Sandbox", "SoftGate Performance", "Reference Network Architectures", "Netris Supported Functionality & Platforms Matrix", "Hardware Requirements", "Network Switch Initial Setup", "Switch Ports", "Terraform: Netris provider", "Inventory", "Netris Try & Learn", "Netris Infrastructure Simulation Platform", "Lab Scenario: GPU-as-a-Service network with NVIDIA Spectrum-X architecture", "Netris Test Controller & Infrastructure Simulation", "Activating BGP on Equinix Metal Project", "Provisioning Netris SoftGate nodes in Equinix Metal Project", "Site Mesh with AWS Overview", "Deploy a Softgate in AWS", "Connecting Netris managed fabric to an ISP", "Connecting servers to the Netris managed fabric", "Creating an Interconnection to Equinix Fabric", "Enabling services (NAT, V-Net, Load Balancer, IP pools)", "Enabling Load-balancing services", "Enabling NAT services", "Enable Equinix Metal API integration", "Site Mesh with GCP Overview", "Deploy a Softgate in GCP", "Equinix Metal API integration enablement", "Netris Generic Tutorials", "Netris VPC for Equinix Metal Getting Started Guide", "VPC for Anywhere Getting Started Guide", "Installing a Netris Controller on Equinix Metal on-demand server", "Installing a Netris Controller", "Inventory setup", "IPAM setup", "More features", "Install a Netris Controller", "Netris managed fabric Overview", "Netris Switch Agent Installation", "Enabling Site Mesh with AWS", "Getting Started with Equinix Metal VPC", "Enabling Site Mesh with GCP", "Getting Started with PhoenixNAP VPC", "New Site setup", "Provisioning Netris SoftGate nodes in phoenixNAP BMC", "Enable phoenixNAP BMC API integration", "Netris VPC for phoenixNAP BMC Overview", "Installing a Netris Controller on phoenixNAP BMC server", "IPAM Setup for Services", "Installing Netris on phoenixNAP BMC", "Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC", "Using NAT services in phoenixNAP BMC", "Using V-Net (isolated virtual network) services in phoenixNAP BMC", "SoftGate software provisioning", "Topology setup", "Upgrade and Rollback", "Upgrading SONiC Operating System", "Using on-demand (elastic) L4 Load Balancer service", "Using V-Net (isolated virtual network) services in Equinix Metal Project", "VPC Anywhere Getting Started Guide", "Check Default Site Settings", "VPC Anywhere Overview", "Install a Netris Controller", "IPAM Setup for Services", "SoftGate Installation", "Connecting VPC to upstream networks (use one of two options)", "Using on-demand (elastic) L4 Load Balancer service", "Using Multi-interface SoftGate", "Using NAT services", "Using V-Net (isolated virtual network) services", "Getting Started with Switch-Fabric Manager & VPC", "VPC setup", "Visibility (Telescope)", "V-Net", "Netris VPC"], "terms": {"A": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179], "variabl": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179], "The": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 22, 23, 24, 25, 26, 27, 28, 29, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 107, 112, 113, 114, 117, 122, 131, 136, 137, 138, 141, 142, 143, 152, 156, 157, 158, 159, 161, 163, 165, 167, 168, 169, 173, 174, 176, 179], "i": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 112, 113, 114, 116, 117, 118, 120, 121, 122, 123, 124, 125, 126, 127, 130, 131, 132, 136, 137, 138, 141, 142, 143, 149, 151, 152, 153, 155, 156, 157, 158, 159, 160, 161, 162, 163, 165, 166, 167, 168, 169, 170, 171, 173, 174, 176, 177, 178, 179], "essenti": [0, 122, 131, 156, 168, 173], "environ": [0, 11, 15, 16, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 109, 113, 117, 118, 121, 130, 173, 174], "where": [0, 9, 10, 11, 18, 24, 26, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 114, 117, 125, 131, 137, 156, 157, 163, 167, 173, 178], "switch": [0, 9, 10, 11, 13, 15, 19, 20, 21, 22, 23, 25, 28, 29, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 113, 115, 122, 123, 124, 129, 131, 132, 138, 142, 148, 150, 159, 161, 166, 177, 178], "softgat": [0, 13, 16, 20, 21, 22, 24, 28, 30, 32, 33, 36, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 88, 90, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 113, 117, 119, 121, 125, 126, 128, 130, 134, 135, 138, 142, 144, 145, 146, 151, 153, 156, 157, 163, 164, 166, 170, 173, 174, 175], "other": [0, 3, 4, 6, 7, 8, 9, 10, 11, 15, 21, 22, 26, 27, 28, 30, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 93, 96, 98, 101, 103, 106, 109, 110, 112, 117, 119, 121, 122, 124, 125, 126, 128, 130, 131, 139, 151, 153, 156, 157, 158, 160, 163, 165, 166, 169, 173, 174, 178, 179], "infrastructur": [0, 12, 16, 109, 115, 117, 157, 163], "devic": [0, 11, 20, 26, 27, 28, 33, 37, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 87, 88, 93, 98, 103, 109, 112, 114, 122, 125, 131, 138, 139, 159, 160], "do": [0, 6, 11, 13, 14, 21, 22, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 122, 123, 125, 129, 132, 139, 150, 158], "have": [0, 6, 11, 13, 15, 18, 21, 24, 25, 27, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 112, 113, 114, 117, 122, 125, 126, 131, 137, 139, 148, 151, 156, 157, 159, 160, 161, 163, 165, 167, 173, 174, 179], "direct": [0, 26, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 122, 131, 179], "access": [0, 1, 2, 3, 4, 5, 8, 9, 11, 15, 16, 18, 22, 24, 27, 28, 29, 30, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 86, 89, 90, 91, 94, 95, 96, 99, 100, 101, 104, 105, 106, 109, 117, 118, 121, 122, 123, 126, 128, 130, 131, 136, 137, 140, 141, 142, 152, 153, 156, 157, 161, 163, 167, 168, 173, 174, 177], "internet": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 14, 18, 27, 28, 30, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 113, 119, 123, 126, 128, 137, 153, 156, 157, 158, 163, 167, 170, 173, 174, 179], "By": [0, 14, 20, 31, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 113, 114, 122, 138, 142, 172, 177], "set": [0, 1, 2, 3, 4, 5, 6, 8, 9, 11, 12, 14, 16, 18, 22, 24, 25, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 112, 113, 114, 117, 121, 125, 126, 129, 130, 132, 136, 137, 141, 151, 152, 153, 157, 158, 160, 163, 164, 167, 168, 169, 170, 174, 177, 178], "up": [0, 1, 2, 3, 4, 5, 6, 7, 8, 18, 22, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 110, 112, 113, 117, 118, 119, 121, 125, 130, 136, 141, 152, 157, 158, 160, 167, 169, 177], "you": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 12, 13, 15, 16, 18, 19, 20, 22, 23, 24, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 113, 114, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 141, 142, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 160, 161, 162, 163, 165, 167, 168, 169, 170, 171, 172, 173, 174, 176, 177, 178, 179], "ensur": [0, 1, 4, 7, 18, 25, 26, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 122, 126, 131, 136, 137, 141, 152, 158, 160, 165, 167, 169, 174], "can": [0, 4, 7, 9, 10, 11, 15, 17, 18, 20, 21, 22, 23, 24, 25, 27, 28, 29, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 110, 112, 113, 114, 117, 118, 119, 120, 121, 122, 125, 126, 127, 130, 131, 132, 136, 137, 138, 139, 140, 141, 148, 149, 151, 152, 153, 155, 156, 157, 159, 160, 162, 163, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 176, 177, 178, 179], "still": [0, 22, 25, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 160], "download": [0, 8, 14, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 156, 161, 173], "necessari": [0, 7, 19, 25, 27, 33, 37, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 122, 124, 131, 151, 169, 170, 174], "packag": [0, 118, 156, 173], "updat": [0, 11, 12, 30, 117, 156, 160, 161, 173], "through": [0, 1, 2, 3, 4, 5, 10, 15, 18, 24, 27, 28, 32, 33, 35, 36, 37, 38, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 117, 118, 121, 126, 127, 130, 134, 136, 137, 139, 141, 152, 154, 155, 156, 157, 158, 160, 162, 163, 166, 167, 168, 171, 173, 174], "apt": [0, 14, 117], "thi": [0, 1, 9, 10, 11, 13, 18, 20, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 112, 113, 114, 117, 118, 119, 120, 121, 122, 123, 125, 126, 128, 130, 131, 132, 134, 136, 137, 138, 139, 141, 143, 149, 152, 153, 154, 156, 157, 158, 159, 160, 161, 163, 165, 167, 170, 172, 173, 174, 178, 179], "particularli": 0, "air": 0, "gap": 0, "restrict": [0, 18, 137, 141, 167], "network": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 18, 19, 21, 22, 25, 26, 27, 28, 30, 31, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 104, 105, 110, 113, 114, 115, 119, 121, 122, 123, 124, 125, 126, 128, 130, 131, 134, 137, 138, 139, 140, 141, 142, 143, 145, 147, 148, 151, 153, 156, 158, 159, 160, 164, 165, 166, 167, 169, 173, 176, 177, 178, 179], "reli": 0, "extern": [0, 15, 16, 22, 26, 27, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 122, 131, 157, 177], "an": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 16, 18, 20, 24, 25, 27, 30, 31, 32, 33, 34, 36, 37, 38, 39, 41, 42, 43, 44, 46, 47, 48, 49, 51, 52, 53, 54, 56, 57, 58, 59, 61, 62, 63, 64, 66, 67, 68, 69, 71, 72, 73, 74, 76, 77, 78, 79, 81, 82, 83, 84, 86, 87, 88, 89, 91, 92, 93, 94, 96, 97, 98, 99, 101, 102, 103, 104, 106, 109, 110, 114, 117, 118, 121, 124, 127, 129, 130, 131, 132, 136, 137, 138, 139, 141, 144, 150, 151, 152, 153, 155, 160, 161, 162, 163, 164, 166, 167, 169, 170, 171, 172, 174, 175, 178, 179], "option": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 13, 16, 17, 22, 23, 25, 27, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 112, 113, 117, 122, 124, 131, 147, 157, 160, 163, 164, 169, 177], "In": [0, 11, 12, 18, 20, 22, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 117, 119, 120, 122, 125, 126, 131, 132, 136, 137, 139, 141, 149, 152, 153, 156, 157, 160, 161, 163, 167, 168, 170, 172, 173, 174], "addit": [0, 6, 11, 16, 22, 23, 28, 30, 31, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 117, 123, 131, 133, 134, 135, 139, 158, 168, 172, 176, 179], "serv": [0, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 157, 163, 168, 174], "also": [0, 9, 10, 18, 22, 25, 26, 27, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 110, 112, 113, 114, 116, 117, 120, 122, 126, 132, 138, 139, 149, 151, 157, 163, 168, 172, 177, 179], "host": [0, 6, 7, 10, 12, 14, 16, 17, 18, 21, 22, 24, 26, 28, 30, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 109, 115, 118, 119, 128, 136, 137, 140, 141, 152, 157, 158, 160, 163, 167, 174, 178], "custom": [0, 8, 9, 10, 16, 20, 27, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 88, 89, 91, 93, 94, 96, 98, 99, 101, 103, 104, 106, 109, 114, 116, 117, 138, 177], "iso": 0, "file": [0, 3, 4, 6, 8, 11, 14, 16, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 118, 156, 158, 160, 173], "featur": [0, 10, 16, 22, 28, 30, 114, 122, 123, 128, 139, 157, 163, 168, 175, 179], "allow": [0, 20, 21, 22, 28, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 87, 88, 90, 93, 95, 98, 100, 103, 105, 109, 114, 117, 121, 122, 128, 130, 131, 138, 156, 157, 166, 173, 178, 179], "store": [0, 11, 27, 113, 118], "manag": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 14, 15, 18, 22, 26, 27, 30, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 112, 113, 117, 122, 125, 131, 137, 138, 139, 141, 157, 158, 159, 160, 163, 166, 167, 169, 174, 176, 177, 178, 179], "specif": [0, 9, 10, 16, 21, 22, 28, 30, 32, 35, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 118, 122, 131, 139, 142, 156, 161, 173, 179], "case": [0, 23, 25, 28, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 109, 110, 117, 118, 126, 153, 160, 178], "deploi": [0, 16, 17, 18, 21, 22, 28, 33, 34, 36, 38, 39, 41, 43, 44, 46, 48, 49, 51, 53, 54, 56, 58, 59, 61, 63, 64, 66, 68, 69, 71, 73, 74, 76, 78, 79, 81, 83, 84, 86, 89, 91, 93, 94, 96, 98, 99, 101, 103, 104, 106, 110, 113, 117, 136, 137, 141, 144, 146, 151, 152, 154, 167], "oper": [0, 1, 2, 3, 4, 5, 7, 8, 15, 19, 25, 26, 27, 28, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 114, 117, 122, 126, 131, 136, 142, 143, 152, 153, 169, 174, 176, 179], "system": [0, 1, 2, 3, 4, 5, 8, 9, 10, 11, 14, 15, 20, 26, 27, 28, 32, 35, 37, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 92, 95, 97, 100, 102, 105, 109, 114, 117, 122, 131, 139, 142, 143, 148, 160, 176, 177, 179], "firmwar": 0, "your": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 13, 14, 16, 18, 19, 20, 22, 25, 26, 28, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 109, 114, 117, 118, 119, 121, 122, 123, 124, 125, 126, 128, 129, 130, 131, 132, 136, 137, 138, 139, 141, 142, 148, 150, 152, 153, 156, 157, 158, 159, 161, 163, 165, 166, 167, 168, 169, 170, 173, 174, 176, 179], "server": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 13, 14, 15, 16, 20, 22, 23, 25, 27, 31, 32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 62, 63, 64, 65, 67, 68, 69, 70, 72, 73, 74, 75, 77, 78, 79, 80, 82, 83, 84, 85, 87, 88, 89, 90, 92, 93, 94, 95, 97, 98, 99, 100, 102, 103, 104, 105, 109, 110, 113, 114, 115, 118, 120, 122, 126, 131, 132, 134, 138, 139, 142, 145, 149, 151, 153, 156, 158, 161, 162, 163, 166, 169, 173, 174, 175, 179], "while": [0, 10, 15, 18, 22, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 136, 137, 139, 141, 152, 160, 167, 176, 179], "provid": [0, 5, 9, 10, 13, 14, 15, 16, 17, 18, 20, 21, 22, 24, 25, 28, 30, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96, 98, 99, 100, 101, 103, 104, 105, 106, 109, 112, 114, 117, 118, 120, 121, 122, 123, 124, 126, 130, 131, 132, 141, 149, 153, 154, 156, 157, 163, 168, 173, 176, 177, 178, 179], "flexibl": [0, 15, 157, 163, 179], "over": [0, 9, 15, 18, 27, 28, 30, 31, 124, 125, 126, 137, 141, 157, 163, 167, 170, 174, 178, 179], "resourc": [0, 9, 10, 11, 15, 16, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104, 109, 112, 122, 126, 131, 148, 153, 157, 168, 176, 179], "especi": [0, 170], "work": [0, 9, 11, 16, 22, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 117, 122], "isol": [0, 16, 109, 134, 145, 147, 151, 164, 166], "util": [0, 10, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 109, 110, 117, 122, 125, 128, 131, 139, 166, 177], "alwai": [0, 21, 28, 126, 131, 139, 153, 156, 160, 173], "even": [0, 31, 32, 37, 40, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 90, 92, 97, 102, 151, 157, 163, 166, 177, 179], "connect": [0, 1, 2, 3, 4, 5, 6, 7, 8, 11, 13, 14, 16, 22, 23, 24, 25, 26, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 113, 114, 115, 118, 122, 125, 128, 131, 141, 156, 157, 158, 159, 160, 164, 166, 169, 173, 174, 175, 178], "limit": [0, 10, 25, 28, 109, 117, 179], "unavail": [0, 27, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 160], "To": [0, 1, 2, 3, 4, 5, 8, 9, 10, 11, 12, 18, 22, 23, 25, 26, 28, 31, 32, 35, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 114, 117, 118, 122, 123, 124, 125, 126, 131, 153, 154, 161, 168, 170, 172, 174, 176, 177], "instal": [0, 1, 2, 3, 4, 5, 8, 30, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 118, 120, 121, 122, 130, 131, 132, 133, 134, 135, 142, 145, 147, 149, 151, 156, 158, 160, 161, 164, 173, 175], "run": [0, 6, 11, 13, 18, 22, 27, 28, 30, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 110, 113, 117, 121, 130, 160], "follow": [0, 1, 4, 6, 11, 12, 13, 15, 21, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 107, 112, 113, 114, 122, 125, 131, 154, 160, 161, 179], "command": [0, 1, 2, 3, 4, 5, 7, 8, 11, 13, 14, 18, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 112, 113, 120, 122, 131, 132, 149, 160, 169], "note": [0, 7, 11, 25, 26, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 112, 122, 126, 131, 139, 147, 157, 160, 163, 168, 169, 174, 179], "must": [0, 5, 11, 12, 21, 28, 112, 113, 122, 125, 131, 136, 137, 139, 141, 152, 167, 168], "ctl": [0, 11, 18, 117, 118, 136, 137, 141, 152, 160, 167], "hostnam": [0, 11, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 136, 137, 141, 152, 167], "argument": [0, 11, 113, 118], "otherwis": [0, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 169, 178], "script": [0, 11, 117, 169], "exit": [0, 1, 28, 117], "error": [0, 109, 113, 117, 177], "curl": [0, 11, 13, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 136, 137, 141, 152, 160, 167], "ss": [0, 11], "http": [0, 1, 2, 3, 4, 5, 8, 11, 12, 13, 14, 18, 22, 24, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 113, 116, 117, 136, 137, 141, 152, 160, 161, 167], "get": [0, 1, 2, 3, 4, 5, 8, 10, 11, 13, 14, 16, 17, 18, 22, 24, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 113, 117, 122, 123, 126, 133, 136, 137, 141, 152, 153, 157, 160, 163, 167], "io": [0, 11, 12, 13, 14, 18, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 116, 136, 137, 141, 152, 160, 167], "sh": [0, 11, 13, 18, 117, 136, 137, 141, 152, 160, 167], "mai": [0, 1, 18, 22, 25, 31, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 117, 118, 125, 126, 128, 137, 141, 148, 151, 153, 160, 165, 166, 167, 168, 179], "take": [0, 1, 11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 122, 124, 131, 158, 160, 161, 168, 174, 179], "depend": [0, 40, 90, 136, 137, 141, 152, 167, 170, 178], "": [0, 1, 2, 3, 4, 5, 7, 8, 9, 10, 16, 18, 19, 22, 25, 27, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 110, 112, 113, 117, 118, 121, 122, 126, 130, 136, 137, 139, 141, 142, 149, 151, 152, 153, 156, 157, 158, 160, 163, 167, 169, 172, 173, 174, 178], "avail": [0, 1, 2, 3, 4, 5, 6, 11, 13, 22, 24, 25, 28, 30, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103, 109, 110, 112, 113, 116, 117, 124, 142, 161, 172, 174, 177, 178, 179], "output": [0, 4, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 160], "look": [0, 16, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 117, 122, 124, 125, 131, 170, 179], "similar": [0, 22, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 131, 142, 156, 160, 166, 173], "ubuntu": [0, 6, 7, 13, 14, 16, 30, 109, 110, 111, 117, 118, 120, 122, 131, 132, 149, 157, 158, 161, 169], "check": [0, 10, 16, 22, 23, 24, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 109, 113, 115, 123, 127, 155, 157, 158, 162, 164, 170, 171, 177], "fqdn": [0, 11, 30, 118], "creat": [0, 6, 7, 9, 10, 11, 12, 13, 16, 18, 22, 25, 28, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59, 60, 61, 62, 64, 65, 66, 67, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 99, 100, 101, 102, 104, 105, 106, 109, 112, 114, 116, 117, 118, 120, 121, 123, 126, 128, 129, 130, 132, 136, 137, 138, 139, 141, 144, 146, 147, 149, 150, 151, 152, 153, 156, 159, 160, 163, 164, 167, 169, 173, 176, 177, 178, 179], "servic": [0, 1, 2, 3, 4, 5, 8, 9, 10, 14, 15, 19, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 32, 34, 35, 36, 37, 39, 40, 41, 42, 44, 45, 46, 47, 49, 50, 51, 52, 54, 55, 56, 57, 59, 60, 61, 62, 64, 65, 66, 67, 69, 70, 71, 72, 74, 75, 76, 77, 79, 80, 81, 82, 84, 85, 86, 87, 89, 90, 91, 92, 94, 95, 96, 97, 99, 100, 101, 102, 104, 105, 106, 107, 109, 114, 115, 119, 122, 124, 125, 131, 134, 136, 137, 138, 139, 141, 142, 145, 147, 151, 152, 164, 167, 175, 177, 178], "under": [0, 7, 10, 15, 23, 24, 28, 31, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 91, 92, 96, 97, 101, 102, 106, 117, 119, 120, 122, 125, 126, 129, 131, 132, 139, 140, 149, 150, 153, 156, 157, 160, 163, 165, 169, 170, 173, 174, 177, 178], "url": 0, "exampl": [0, 1, 2, 3, 4, 5, 6, 9, 10, 11, 13, 14, 18, 20, 22, 23, 24, 28, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96, 98, 99, 100, 101, 103, 104, 105, 106, 109, 112, 113, 114, 117, 118, 119, 122, 125, 126, 131, 136, 137, 141, 152, 153, 156, 157, 158, 160, 161, 163, 167, 168, 172, 173, 174, 177, 178], "com": [0, 11, 12, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 136, 137, 141, 167], "wait": [0, 1, 10, 26, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 120, 122, 131, 132, 149], "pod": [0, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 160], "readi": [0, 1, 2, 3, 4, 5, 8, 22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 125, 157, 160, 163], "No": [0, 6, 113, 117, 158], "found": [0, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 140, 148, 160], "creation": [0, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 122, 131, 138, 151, 157], "current": [0, 1, 2, 3, 4, 5, 8, 11, 16, 21, 22, 25, 26, 28, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 88, 90, 95, 100, 105, 114, 117, 122, 131, 159, 170, 177, 178], "statu": [0, 5, 22, 28, 33, 35, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 112, 114, 117, 122, 123, 125, 131, 160], "pend": [0, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "now": [0, 13, 22, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 113, 117, 119, 125, 126, 129, 132, 150, 153, 157, 177], "path": [0, 11, 24, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "upload": 0, "var": [0, 14, 161], "lib": [0, 4, 14], "rancher": 0, "k3": [0, 11, 13, 18, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 136, 137, 141, 152, 167], "storag": [0, 117], "pvc": 0, "ea0dd3ef": 0, "ded1": 0, "49d3": 0, "bbd8": 0, "b797c91d76b5_netri": 0, "controller_netri": 0, "directori": [0, 11, 16, 118, 160], "copi": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 27, 117, 120, 122, 129, 131, 132, 149, 150, 158, 160, 169], "past": [0, 1, 3, 7, 117, 120, 122, 129, 131, 132, 149, 150, 160, 169], "web": [0, 1, 2, 3, 4, 5, 13, 18, 22, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 109, 115, 117, 119, 120, 122, 126, 127, 129, 131, 132, 136, 137, 141, 149, 150, 152, 153, 155, 156, 157, 160, 161, 162, 167, 168, 171, 172, 173, 179], "ui": [0, 13, 22, 28, 31, 112, 113], "section": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 15, 22, 24, 26, 28, 30, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 112, 114, 117, 118, 122, 126, 127, 131, 138, 153, 155, 157, 160, 162, 163, 168, 169, 170, 171, 172], "shown": [0, 112], "screenshot": [0, 10, 23, 24, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 119, 120, 126, 127, 132, 149, 153, 155, 162, 171, 177], "below": [0, 6, 7, 10, 14, 17, 18, 21, 24, 25, 28, 31, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 110, 114, 116, 117, 119, 120, 122, 125, 126, 127, 129, 131, 132, 134, 136, 137, 141, 149, 150, 152, 153, 155, 156, 158, 160, 162, 167, 169, 171, 173, 179], "addition": [0, 32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "which": [0, 1, 2, 3, 4, 5, 7, 9, 18, 21, 22, 27, 28, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 113, 117, 139, 141, 151, 161, 169, 177], "want": [0, 11, 13, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 122, 125, 126, 148, 153, 156, 165, 168, 172, 173, 177], "onc": [0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 18, 26, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 112, 114, 117, 118, 121, 122, 125, 126, 130, 131, 136, 137, 138, 141, 151, 152, 153, 156, 157, 158, 159, 160, 163, 167, 168, 169, 173], "function": [0, 6, 7, 16, 17, 27, 28, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 142, 151, 166, 169, 177, 179], "agent": [0, 1, 2, 3, 4, 5, 7, 8, 10, 13, 15, 16, 25, 30, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 114, 117, 120, 122, 131, 132, 137, 141, 142, 149, 151, 158, 159, 161, 167, 169, 175], "onelin": 0, "automat": [0, 1, 2, 3, 4, 5, 7, 8, 16, 19, 22, 24, 26, 27, 28, 30, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 109, 112, 113, 114, 117, 119, 122, 124, 126, 131, 142, 151, 153, 157, 159, 166, 169, 170, 177, 178, 179], "point": [0, 18, 28, 109, 113, 117, 120, 125, 126, 132, 149, 157, 163], "includ": [0, 20, 21, 24, 28, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 88, 91, 96, 101, 106, 109, 114, 117, 138, 139, 157, 160, 161, 163, 168, 173, 174], "all": [0, 6, 7, 9, 10, 11, 15, 17, 20, 22, 25, 26, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 112, 113, 114, 117, 121, 122, 130, 131, 138, 148, 151, 157, 160, 161, 163, 166, 169, 174, 177, 179], "nvue": 0, "cumulu": [0, 2, 16, 25, 109, 110, 111, 112, 117], "5": [0, 4, 5, 6, 10, 22, 25, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 71, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 110, 117, 129, 132, 150, 153, 160, 161], "9": [0, 16, 81, 109, 111, 117], "higher": [0, 25, 30, 31], "h": [0, 11, 16, 109, 157], "24": [0, 10, 13, 22, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 110, 113, 117, 157, 163, 173, 174], "04": [0, 6, 7, 8, 14, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 109, 110, 117, 120, 122, 131, 132, 158, 169], "restart": [0, 11, 14, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 112, 160], "deploy": [0, 11, 13, 15, 18, 30, 137, 141, 148, 167, 169, 179], "Then": [0, 1, 2, 3, 4, 5, 7, 8, 10, 11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 120, 123, 125, 126, 129, 132, 149, 150, 153, 157, 163, 169], "verifi": [0, 6, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 131, 160, 161], "rollout": [0, 11], "succe": 0, "kubectl": [0, 11, 12, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 160], "nnetri": [0, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "nginx": [0, 14, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "expect": [0, 11, 31, 161, 169], "app": [0, 117, 160], "finish": [0, 18, 22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 120, 122, 131, 132, 136, 137, 141, 152, 160, 167], "1": [0, 1, 4, 5, 8, 12, 13, 14, 22, 28, 30, 32, 33, 37, 38, 42, 43, 47, 48, 51, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 107, 109, 110, 113, 117, 134, 137, 141, 145, 147, 149, 157, 160, 161, 163, 167, 174, 176, 179], "old": [0, 11, 28, 31, 161, 178], "replica": 0, "ar": [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 13, 15, 16, 18, 21, 22, 23, 24, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 107, 110, 112, 113, 114, 117, 118, 119, 120, 121, 122, 124, 125, 126, 130, 131, 132, 133, 134, 135, 136, 137, 139, 149, 152, 153, 156, 157, 158, 160, 163, 165, 167, 168, 169, 170, 173, 178, 179], "termin": [0, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 109, 113, 122, 131], "successfulli": [0, 11, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 122, 125, 131, 160, 168], "roll": [0, 28, 157], "out": [0, 1, 2, 3, 4, 5, 6, 8, 11, 13, 21, 22, 26, 28, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 113, 114, 117, 123, 141, 158, 160, 169, 179], "further": [1, 2, 3, 4, 5, 6, 7, 8, 27, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 119, 158, 169], "requir": [1, 2, 3, 4, 5, 8, 10, 16, 17, 18, 19, 21, 22, 24, 27, 28, 30, 32, 33, 35, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 113, 117, 122, 123, 125, 131, 135, 136, 137, 139, 142, 149, 152, 156, 158, 164, 168, 173, 175, 178, 179], "consol": [1, 2, 3, 4, 5, 8, 14, 18, 109, 115, 117, 119, 120, 122, 125, 126, 127, 129, 131, 132, 136, 137, 141, 149, 150, 152, 153, 155, 157, 162, 163, 167, 168, 171, 172], "via": [1, 2, 3, 4, 5, 6, 7, 8, 15, 21, 22, 28, 33, 35, 36, 38, 40, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 93, 95, 96, 98, 100, 101, 103, 105, 106, 112, 113, 122, 131, 151, 157, 158, 160, 161, 163, 169], "port": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16, 22, 23, 24, 25, 26, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 110, 113, 114, 117, 122, 123, 124, 125, 131, 137, 141, 156, 158, 159, 166, 167, 169, 173, 177, 178], "If": [1, 2, 3, 4, 5, 6, 8, 11, 13, 14, 22, 25, 26, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 117, 118, 123, 124, 128, 139, 141, 148, 158, 160, 168, 172, 178], "ha": [1, 2, 3, 4, 5, 8, 9, 14, 18, 22, 26, 27, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 109, 110, 113, 117, 119, 131, 136, 137, 139, 141, 151, 152, 153, 157, 160, 163, 167, 168], "pre": [1, 2, 3, 4, 5, 7, 8, 32, 35, 36, 37, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 91, 92, 95, 96, 97, 100, 101, 102, 105, 106, 109, 169], "NOS": [1, 2, 3, 4, 5, 8, 16, 27, 110, 114, 142], "need": [1, 2, 3, 4, 5, 7, 8, 10, 11, 18, 22, 23, 25, 28, 30, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 107, 109, 112, 114, 117, 118, 119, 120, 121, 122, 126, 128, 129, 130, 131, 132, 137, 138, 141, 149, 150, 151, 153, 157, 160, 161, 163, 166, 167, 168, 169, 170, 176, 177, 178, 179], "uninstal": [1, 2, 3, 4, 5, 8, 17], "first": [1, 2, 3, 4, 5, 8, 11, 26, 28, 32, 33, 35, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 121, 125, 130, 139, 153, 161, 168, 177], "oni": [1, 2, 3, 4, 5, 8], "from": [1, 2, 3, 4, 5, 6, 7, 8, 10, 11, 13, 14, 16, 20, 21, 23, 24, 25, 26, 27, 28, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 109, 112, 114, 117, 120, 122, 123, 125, 126, 128, 129, 131, 132, 133, 134, 135, 138, 139, 149, 150, 151, 153, 156, 157, 158, 160, 161, 163, 166, 168, 169, 172, 173, 174, 177, 178], "grub": [1, 2, 3, 4, 5, 8], "menu": [1, 2, 3, 4, 5, 8, 9, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 112, 117, 120, 122, 126, 129, 131, 132, 149, 150, 153, 160, 168, 177], "select": [1, 2, 3, 4, 5, 6, 8, 13, 17, 20, 21, 23, 24, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 112, 114, 117, 122, 125, 126, 128, 129, 131, 132, 148, 150, 152, 153, 157, 159, 168, 172, 177], "o": [1, 2, 3, 4, 5, 6, 7, 8, 11, 18, 22, 25, 27, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 107, 109, 110, 120, 132, 136, 137, 141, 149, 152, 157, 158, 161, 167, 169], "done": [1, 2, 3, 4, 5, 8, 113, 117, 120, 121, 122, 126, 130, 131, 132, 149, 151], "reboot": [1, 2, 3, 4, 5, 6, 7, 8, 14, 120, 122, 131, 132, 149, 158, 161, 169], "dhcp": [1, 2, 3, 4, 5, 8, 14, 27, 109, 142, 157, 174], "stop": [1, 2, 3, 4, 5, 8, 16, 115, 117, 122], "discoveri": [1, 2, 3, 4, 5, 8, 117], "configur": [1, 2, 3, 4, 5, 7, 8, 13, 14, 15, 16, 17, 19, 22, 23, 25, 27, 28, 30, 31, 33, 34, 35, 36, 38, 39, 40, 41, 43, 44, 45, 46, 48, 49, 50, 51, 53, 54, 55, 56, 58, 59, 60, 61, 63, 64, 65, 66, 68, 69, 70, 71, 73, 74, 75, 76, 78, 79, 80, 81, 83, 84, 85, 86, 88, 89, 90, 91, 93, 94, 95, 96, 98, 99, 100, 101, 103, 104, 105, 106, 109, 112, 114, 117, 118, 119, 121, 123, 124, 125, 126, 130, 142, 144, 146, 151, 157, 158, 159, 169, 174, 177, 178], "ip": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23, 24, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 114, 117, 119, 120, 121, 122, 125, 128, 130, 131, 132, 134, 136, 137, 139, 141, 145, 149, 151, 152, 153, 156, 158, 161, 163, 164, 167, 168, 169, 173, 174, 176, 179], "address": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 18, 20, 22, 23, 24, 27, 30, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 86, 89, 90, 91, 94, 95, 96, 99, 100, 101, 104, 105, 106, 109, 113, 114, 117, 119, 120, 122, 128, 131, 132, 134, 136, 137, 139, 141, 142, 145, 149, 151, 152, 153, 156, 157, 158, 161, 163, 164, 167, 169, 173, 174, 177, 178], "default": [1, 2, 3, 4, 5, 6, 7, 8, 9, 12, 14, 15, 16, 18, 20, 22, 28, 30, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 112, 113, 114, 117, 122, 124, 129, 131, 132, 136, 137, 138, 141, 148, 150, 151, 152, 157, 158, 163, 164, 166, 167, 168, 169, 170, 172, 174, 176, 178, 179], "gatewai": [1, 2, 3, 4, 5, 6, 7, 8, 14, 22, 30, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 86, 89, 90, 91, 94, 95, 96, 99, 100, 101, 104, 105, 106, 109, 113, 114, 117, 122, 124, 126, 131, 142, 151, 153, 157, 158, 163, 166, 169, 174, 178], "manual": [1, 2, 3, 4, 5, 7, 8, 24, 25, 33, 35, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 112, 117, 169], "addr": [1, 2, 3, 4, 5, 8], "add": [1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 16, 22, 23, 24, 25, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 112, 117, 120, 123, 124, 125, 127, 128, 129, 132, 136, 137, 138, 141, 148, 150, 152, 153, 157, 158, 163, 167, 168, 169, 176, 177, 178], "prefix": [1, 2, 3, 4, 5, 6, 7, 8, 14, 16, 21, 22, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 93, 95, 96, 98, 100, 101, 103, 105, 106, 113, 122, 131, 139, 153, 158, 168, 169, 170], "dev": [1, 2, 3, 4, 5, 8, 18, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 118, 136, 137, 141, 152, 167], "eth0": [1, 2, 3, 4, 5, 8, 14, 122], "rout": [1, 2, 3, 4, 5, 6, 7, 8, 16, 19, 26, 27, 30, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101, 103, 106, 107, 117, 119, 121, 124, 125, 130, 140, 142, 146, 158, 169, 170, 178, 179], "echo": [1, 2, 3, 4, 5, 8], "nameserv": [1, 2, 3, 4, 5, 8, 14], "dn": [1, 2, 3, 4, 5, 8, 14, 18, 20, 30, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 109, 114, 136, 137, 138, 141, 152, 167], "etc": [1, 2, 3, 4, 5, 6, 7, 8, 14, 21, 27, 28, 109, 110, 112, 113, 117, 139, 148, 158, 169, 177], "resolv": [1, 2, 3, 4, 5, 8, 18, 136, 137, 141, 152, 167], "conf": [1, 2, 3, 4, 5, 8], "imag": [1, 2, 3, 4, 5, 8, 14, 117, 161], "should": [1, 2, 3, 4, 5, 6, 7, 11, 13, 14, 15, 22, 23, 24, 25, 26, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 117, 119, 120, 122, 124, 125, 126, 131, 132, 139, 142, 148, 149, 151, 153, 157, 158, 161, 163, 169, 170, 174, 178, 179], "local": [1, 2, 3, 4, 5, 11, 16, 17, 22, 26, 27, 28, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 93, 95, 96, 98, 100, 101, 103, 105, 106, 113, 117, 125, 156, 170, 173], "nos": [1, 2, 3, 4, 5, 8, 113], "192": [1, 2, 3, 4, 5, 13, 22, 28, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 113, 117, 139, 168], "168": [1, 2, 3, 4, 5, 13, 22, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 117, 139, 168], "100": [1, 2, 3, 4, 5, 13, 27, 28, 43, 46, 109, 110, 113], "10": [1, 2, 3, 4, 5, 6, 10, 28, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 91, 92, 95, 96, 97, 100, 101, 102, 105, 106, 107, 110, 117, 126, 131, 139, 156, 157, 160, 161, 163, 168], "enterprise_sonic_os_4": 1, "1_enterprise_premium": 1, "bin": [1, 2, 3, 4, 5, 8, 11, 117, 161], "after": [1, 2, 3, 4, 5, 7, 10, 11, 14, 21, 22, 24, 28, 33, 35, 37, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 112, 113, 119, 122, 131, 160, 161, 169], "complet": [1, 2, 3, 4, 5, 6, 11, 13, 22, 26, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 122, 125, 131, 158, 160, 161], "login": [1, 2, 3, 4, 5, 13, 14, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 161], "us": [1, 2, 3, 4, 5, 7, 8, 9, 10, 11, 13, 14, 15, 16, 17, 18, 20, 21, 23, 24, 25, 26, 27, 28, 31, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 88, 89, 92, 93, 94, 97, 98, 99, 102, 103, 104, 109, 110, 112, 113, 114, 116, 117, 118, 119, 120, 122, 123, 124, 125, 126, 127, 128, 131, 132, 136, 137, 139, 141, 142, 148, 151, 152, 153, 158, 159, 160, 161, 165, 166, 167, 168, 169, 176, 178, 179], "usernam": [1, 2, 3, 4, 5, 8, 9, 15, 22, 117, 118], "password": [1, 2, 3, 4, 5, 8, 9, 14, 15, 18, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 117, 118, 136, 137, 141, 152, 161, 167], "admin": [1, 2, 9, 10, 15, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 112, 113, 114, 117, 126, 136, 137, 141, 152, 153, 157, 161, 167, 168], "yourpassword": [1, 2], "band": [1, 2, 3, 4, 5, 8, 21, 28, 114, 117, 141], "oob": [1, 2, 3, 4, 5, 6, 8, 110, 117, 158], "upon": [1, 4, 5, 158], "boot": [1, 6, 7, 117, 158, 169], "newli": [1, 6, 18, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 117, 120, 126, 132, 136, 137, 141, 149, 152, 153, 157, 158, 163, 167], "pleas": [1, 4, 6, 7, 18, 25, 26, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 116, 117, 118, 124, 126, 127, 136, 153, 154, 155, 158, 160, 161, 162, 169, 171, 179], "until": [1, 10, 22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 120, 132, 149], "messag": [1, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 161], "appear": [1, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 112, 131], "typic": [1, 9, 15, 28, 114, 117, 151, 166, 170], "resembl": 1, "03": [1, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 157, 163], "15": [1, 3, 22, 35, 40, 41, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 107, 117], "02": [1, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 157, 163], "35": [1, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 117], "430469": 1, "displai": [1, 22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 113, 122, 131, 158], "proce": [1, 11, 119, 122, 131, 168], "disabl": [1, 2, 4, 5, 6, 10, 16, 25, 26, 28, 112, 122, 178], "ztp": [1, 2, 4, 5], "cli": 1, "config": [1, 5, 10, 14, 30, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "enabl": [1, 6, 16, 17, 22, 25, 26, 27, 32, 35, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 109, 117, 119, 121, 130, 134, 138, 139, 142, 145, 151, 156, 157, 163, 164, 172, 173, 174, 175, 178, 179], "more": [1, 7, 16, 23, 25, 26, 27, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 110, 113, 117, 126, 148, 153, 156, 157, 163, 169, 173, 175, 179], "reappear": 1, "approxim": 1, "2": [1, 6, 21, 27, 28, 30, 31, 32, 36, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 109, 110, 113, 117, 119, 120, 122, 124, 125, 131, 132, 134, 137, 141, 145, 147, 149, 157, 161, 163, 167, 178, 179], "minut": [1, 11, 22, 26, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 117, 119, 120, 132, 149, 160], "isn": 1, "t": [1, 18, 21, 22, 28, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 117, 118, 126, 137, 151, 153, 156, 157, 163, 167, 173, 178, 179], "ll": [1, 22, 31, 32, 35, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 113, 117, 121, 130, 178], "management0": 1, "interfac": [1, 3, 4, 5, 6, 7, 8, 9, 14, 16, 18, 22, 25, 27, 28, 31, 32, 33, 35, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 114, 117, 122, 131, 137, 141, 158, 160, 164, 167, 169, 179], "0": [1, 4, 7, 10, 11, 13, 14, 22, 25, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 107, 109, 113, 117, 122, 126, 131, 139, 156, 157, 158, 160, 161, 163, 168, 169, 170, 173, 174, 179], "mgmt": [1, 4, 5, 14, 113, 117], "ipv4": [1, 10, 15, 16, 20, 23, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 114, 117, 122, 125, 131, 178], "subnet": [1, 13, 15, 16, 20, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 88, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 109, 113, 114, 117, 121, 126, 130, 138, 139, 147, 156, 163, 168, 170, 173, 174, 179], "mask": [1, 5], "gwaddr": 1, "name": [1, 6, 7, 9, 10, 18, 20, 21, 22, 24, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 112, 113, 114, 117, 120, 122, 125, 129, 131, 132, 136, 137, 138, 141, 148, 150, 152, 153, 158, 160, 167, 168, 169, 177, 178], "server1": [1, 5], "server2": [1, 5], "end": [1, 10, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 109, 125, 178], "write": [1, 129, 132], "memori": [1, 6, 117], "netri": [1, 2, 3, 4, 5, 8, 9, 10, 12, 13, 15, 17, 20, 21, 24, 25, 28, 29, 30, 31, 32, 33, 34, 37, 38, 39, 42, 43, 44, 47, 48, 49, 52, 53, 54, 57, 58, 59, 62, 63, 64, 67, 68, 69, 72, 73, 74, 77, 78, 79, 82, 83, 84, 87, 88, 89, 92, 93, 94, 97, 98, 99, 102, 103, 104, 112, 114, 119, 121, 125, 126, 127, 128, 129, 130, 135, 138, 139, 144, 145, 146, 147, 148, 150, 153, 155, 156, 157, 158, 159, 161, 162, 163, 164, 165, 166, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178], "navig": [1, 2, 3, 4, 5, 6, 7, 8, 13, 21, 22, 23, 26, 28, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 114, 117, 118, 122, 124, 127, 128, 131, 158, 160, 168, 169, 172, 177], "net": [1, 2, 3, 4, 5, 6, 7, 8, 10, 13, 16, 21, 24, 25, 28, 30, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 112, 117, 119, 120, 122, 124, 125, 129, 131, 132, 134, 139, 145, 147, 149, 150, 151, 153, 156, 164, 165, 168, 169, 170, 173, 177], "inventori": [1, 2, 3, 4, 5, 6, 7, 8, 10, 15, 16, 26, 30, 112, 113, 117, 118, 120, 122, 131, 132, 149, 158, 160, 169, 172, 175], "click": [1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 21, 23, 24, 28, 30, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 114, 117, 119, 120, 122, 125, 126, 129, 131, 132, 149, 150, 153, 159, 168, 169, 172, 177], "three": [1, 2, 3, 4, 5, 6, 7, 8, 17, 28, 30, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 114, 122, 131, 138, 157, 163, 169, 172], "vertic": [1, 2, 3, 4, 5, 6, 7, 8, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 122, 131, 169, 172], "dot": [1, 2, 3, 4, 5, 6, 7, 8, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 120, 122, 126, 131, 132, 149, 153, 160, 169, 172], "right": [1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 18, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 114, 117, 118, 122, 131, 136, 137, 152, 153, 159, 160, 167, 168, 169, 172, 177], "side": [1, 2, 3, 4, 5, 6, 7, 8, 9, 22, 25, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 119, 122, 125, 131, 151, 153, 169, 172, 174, 177], "provis": [1, 2, 3, 4, 5, 8, 16, 22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 109, 113, 117, 126, 132, 134, 135, 145, 153, 157, 163, 164, 175], "one": [1, 2, 3, 4, 5, 7, 8, 10, 16, 18, 20, 21, 22, 24, 25, 27, 28, 29, 30, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 114, 117, 120, 122, 126, 131, 132, 139, 143, 148, 149, 153, 158, 160, 161, 164, 169, 177, 179], "line": [1, 2, 3, 4, 5, 6, 7, 8, 10, 20, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 114, 122, 131, 158, 169], "clipboard": [1, 2, 3, 4, 5, 6, 7, 8, 13, 122, 131, 169], "sudo": [1, 2, 3, 4, 5, 6, 7, 8, 14, 117, 158, 160, 161, 169], "ec": [2, 110], "sonic_20211125_074752_ec202012_227": 2, "y": [2, 5, 7, 11, 158, 161, 169], "establish": [2, 10, 22, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 113, 117, 119, 121, 125, 130, 156, 168, 173, 179], "linux": [3, 4, 5, 6, 7, 13, 14, 16, 17, 18, 22, 24, 27, 31, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 104, 105, 109, 110, 117, 136, 137, 141, 151, 152, 158, 166, 167, 169], "3": [3, 10, 12, 25, 27, 30, 31, 32, 33, 35, 36, 37, 38, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 107, 109, 110, 113, 117, 120, 124, 132, 134, 137, 145, 149, 153, 160, 163, 178, 179], "mlx": [3, 4, 5], "amd64": [3, 4, 5, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 161], "cumuluslinux": 3, "open": [3, 4, 8, 117, 122, 131, 157, 163, 172, 177], "detail": [3, 4, 6, 8, 12, 13, 19, 23, 28, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 113, 117, 131, 136, 137, 141, 148, 152, 160, 167, 177], "vim": [3, 4, 6, 7, 8, 14, 158, 169], "loopback": [3, 4, 6, 7, 8, 14, 21, 23, 28, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 113, 114, 122, 131, 139, 158, 169], "auto": [3, 4, 6, 7, 8, 14, 31, 113, 117, 158, 169, 178], "lo": [3, 4, 6, 7, 8, 14, 158, 169], "ifac": [3, 4, 6, 7, 8, 14, 158, 169], "inet": [3, 4, 6, 7, 8, 14, 158, 169], "primari": [3, 4, 8, 14, 125, 139], "static": [3, 4, 6, 7, 8, 14, 16, 109, 117, 140, 158, 169, 179], "length": [3, 4, 6, 7, 8, 14, 28, 122, 131, 158, 169], "sourc": [3, 4, 6, 7, 8, 10, 14, 28, 32, 33, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 88, 92, 93, 97, 98, 102, 103, 113, 131, 156, 158, 169, 173, 177], "d": [3, 4, 5, 6, 7, 8, 11, 14, 28, 116, 158, 169], "tee": [3, 4], "ifreload": [3, 4, 8, 14], "licens": [3, 118], "cl": [3, 117], "string": [3, 28], "press": [3, 113, 157], "ctrl": 3, "4": [4, 11, 14, 18, 22, 24, 25, 27, 28, 32, 33, 36, 37, 38, 40, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 71, 72, 73, 77, 78, 82, 83, 87, 90, 92, 93, 97, 98, 102, 103, 109, 110, 117, 119, 122, 131, 134, 136, 137, 141, 142, 145, 152, 160, 161, 167, 179], "virtual": [4, 6, 15, 16, 18, 30, 109, 112, 122, 124, 125, 126, 131, 134, 139, 145, 147, 151, 153, 160, 164, 176, 178, 179], "forward": [4, 10, 28, 107, 109, 110, 122, 126, 131, 153, 156, 173], "vrf": [4, 15, 28, 37, 109, 176, 179], "refer": [4, 6, 18, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103, 118, 136, 137, 141, 152, 167, 179], "instruct": [4, 11, 13, 18, 22, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 102, 105, 106, 117, 136, 137, 141, 152, 167], "exec": [4, 11, 37, 160], "bash": [4, 11, 117, 160], "remov": [4, 6, 22, 23, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 158, 161, 169, 176, 178, 179], "might": [4, 11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "see": [4, 6, 7, 9, 10, 12, 14, 22, 24, 28, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 113, 114, 117, 119, 120, 126, 132, 136, 149, 153, 158, 160, 169, 177], "time": [4, 10, 13, 20, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 114, 117, 160, 161, 177], "warn": [4, 117], "ignor": 4, "cmd": 4, "usr": [4, 11], "helper": 4, "delet": [4, 6, 7, 12, 16, 22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 114, 117, 138, 158, 169], "1001": [4, 125], "fail": [4, 25], "return": [4, 10, 22, 27, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117], "cgroup": [4, 117], "later": [5, 117, 126, 157, 177], "inform": [5, 19, 26, 27, 28, 113, 114, 117, 123, 125, 172], "nv": 5, "unset": 5, "appli": [5, 10, 11, 19, 22, 25, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 113, 117, 118, 122, 131, 141, 156, 173, 174], "save": [5, 6, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 113, 114, 117, 118, 122, 131, 158, 172, 177], "control": [5, 6, 7, 9, 12, 13, 15, 16, 20, 25, 28, 30, 31, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 91, 94, 99, 104, 109, 112, 113, 114, 115, 120, 123, 124, 125, 126, 129, 132, 133, 134, 135, 138, 139, 142, 144, 145, 146, 149, 150, 151, 153, 156, 157, 158, 159, 163, 164, 166, 168, 169, 170, 173, 174, 175, 176, 177, 178, 179], "back": [5, 22, 26, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 117, 125, 143, 157], "heartbeat": [5, 6, 7, 114, 117, 120, 132, 149, 158, 169], "ok": [5, 6, 7, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 117, 120, 132, 136, 137, 149, 152, 158, 167, 169], "x": [6, 7, 16, 27, 109, 110, 115, 158, 169], "intel": [6, 27, 109, 110], "xeon": [6, 107, 109], "silver": 6, "processor": 6, "physic": [6, 7, 14, 15, 19, 22, 25, 30, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 112, 114, 117, 126, 142, 153, 158, 159, 160, 166, 169, 170, 174, 178, 179], "core": [6, 7, 11, 14, 18, 27, 29, 107, 109, 110, 136, 137, 141, 152, 167, 169], "per": [6, 9, 10, 15, 20, 25, 28, 31, 114, 116, 117, 120, 132, 149, 157, 163, 177, 178], "socket": 6, "20": [6, 109, 117, 131, 158, 174], "total": [6, 119], "128": [6, 32, 33, 36, 37, 38, 41, 42, 43, 46, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 85, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 109, 110, 117, 131, 157], "gb": [6, 7, 11, 14, 18, 110, 122, 131, 136, 137, 141, 152, 167, 169], "64": [6, 11, 18, 22, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 110, 117, 136, 137, 141, 152, 167], "ram": [6, 7, 11, 14, 18, 109, 110, 117, 122, 131, 136, 137, 141, 152, 167, 169, 177], "multichannel": 6, "300": [6, 7, 110, 169], "hdd": [6, 7, 169], "nvidia": [6, 16, 109, 111, 115], "mellanox": 6, "6": [6, 22, 32, 35, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 76, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 107, 109, 110, 113, 117], "smartnic": [6, 27, 142, 158], "card": [6, 27, 142, 158], "some": [6, 11, 22, 31, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 102, 105, 106, 112, 117, 125, 152, 161], "recommend": [6, 7, 11, 14, 27, 28, 109, 110, 114, 122, 125, 136, 137, 141, 152, 157, 166, 167, 169, 170, 174], "differ": [6, 9, 15, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 113, 121, 130, 136, 137, 141, 147, 152, 167, 177, 178, 179], "vendor": [6, 117, 161], "so": [6, 18, 22, 27, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 117, 119, 122, 126, 141, 157, 163, 167, 169, 177], "mainli": 6, "befor": [6, 18, 22, 26, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 136, 137, 139, 141, 152, 157, 160, 167], "start": [6, 10, 14, 16, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 115, 120, 122, 132, 149, 151, 157, 163, 177], "consid": 6, "reset": 6, "power": [6, 109, 177], "perform": [6, 10, 16, 19, 21, 27, 28, 30, 31, 109, 117, 120, 132, 139, 142, 149, 156, 158, 160, 161, 170, 173], "tune": [6, 28], "cpu": [6, 7, 11, 14, 18, 27, 107, 109, 110, 117, 122, 131, 136, 137, 141, 152, 167, 169, 177], "p": [6, 11, 22, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 157, 160], "state": [6, 10, 22, 24, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 110, 113, 117, 122, 123, 131, 157, 177, 178], "c3": [6, 120, 136], "report": [6, 27], "c6": 6, "polici": [6, 9, 15, 19, 27, 28, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 140, 142, 148], "turbo": 6, "boost": 6, "frequenc": 6, "highest": [6, 179], "number": [6, 9, 10, 11, 15, 22, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 107, 109, 114, 117, 148, 160, 161, 170, 177, 178], "NOT": 6, "when": [6, 9, 10, 11, 13, 15, 17, 22, 25, 26, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 109, 112, 113, 114, 117, 120, 122, 125, 126, 131, 132, 149, 151, 153, 157, 163], "test": [6, 16, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 107, 113, 115, 116, 117], "nic": [6, 14, 25, 27, 109, 110], "turn": [6, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 125, 151, 157, 163], "off": [6, 26, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "vt": 6, "hyper": 6, "thread": 6, "freshli": [6, 7, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 169], "18": [6, 8, 14, 22, 35, 36, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 117], "lt": [6, 7, 117, 158, 169], "netplan": [6, 7, 158, 169], "topologi": [6, 7, 13, 16, 22, 26, 28, 30, 33, 34, 35, 38, 39, 40, 43, 44, 45, 48, 49, 50, 53, 54, 55, 58, 59, 60, 63, 64, 65, 68, 69, 70, 73, 74, 75, 78, 79, 80, 83, 84, 85, 89, 90, 93, 94, 95, 98, 99, 100, 103, 104, 105, 109, 113, 115, 118, 138, 169, 175], "document": [6, 13, 21, 28, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 102, 105, 106, 113, 117, 118, 125, 127, 155, 162, 171], "here": [6, 7, 11, 13, 22, 27, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 113, 122, 123, 124, 125, 129, 131, 132, 136, 137, 141, 143, 148, 152, 158, 160, 167, 169], "ad": [6, 7, 9, 16, 22, 23, 24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 112, 113, 138, 147, 169, 178], "ordinari": [6, 7, 21, 28, 169], "user": [6, 7, 10, 13, 15, 16, 18, 19, 20, 21, 22, 23, 24, 27, 28, 30, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 112, 114, 117, 120, 121, 126, 130, 132, 136, 137, 138, 139, 141, 142, 149, 152, 153, 157, 158, 161, 167, 168, 169, 177], "review": [6, 10, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 118, 131], "ifupdown": [6, 7, 158, 169], "present": [6, 7, 22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 158, 160, 169], "correspond": [6, 25, 120, 132, 149, 157, 160, 163, 174], "what": [6, 16, 18, 21, 22, 25, 27, 28, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 118, 129, 132, 136, 137, 141, 150, 151, 152, 153, 158, 165, 166, 167], "dure": [6, 27, 117, 158, 160, 169], "gener": [6, 10, 17, 18, 20, 22, 24, 27, 28, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 117, 118, 125, 129, 132, 136, 137, 138, 141, 142, 152, 158, 160, 161, 167], "base": [6, 9, 10, 15, 19, 21, 22, 23, 28, 30, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 109, 117, 118, 126, 142, 151, 153, 157, 158, 163, 168, 179], "initi": [6, 16, 26, 30, 87, 109, 113, 114, 115, 116, 118, 122, 131, 138, 157, 158], "same": [6, 7, 17, 18, 23, 117, 122, 131, 136, 137, 141, 152, 157, 158, 163, 167, 169, 174], "ensz": [6, 7, 158, 169], "comment": [6, 7, 10, 28, 158, 169], "locat": [6, 7, 11, 23, 28, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 117, 120, 122, 125, 129, 131, 132, 141, 148, 150, 151, 152, 157, 158, 161, 169, 170, 179], "node": [6, 7, 11, 16, 18, 22, 24, 28, 30, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 88, 90, 91, 95, 96, 100, 101, 105, 106, 114, 117, 119, 134, 136, 137, 138, 141, 145, 151, 152, 157, 158, 159, 163, 166, 167, 169, 170, 172, 174], "everyth": [6, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 123, 157], "seem": 6, "ani": [6, 7, 9, 10, 18, 21, 22, 24, 25, 28, 29, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 113, 117, 122, 125, 126, 127, 131, 136, 137, 141, 151, 152, 153, 155, 157, 158, 160, 162, 163, 165, 166, 167, 171, 174, 178, 179], "than": [6, 25, 27, 117, 158, 160], "those": [6, 21, 28, 117, 121, 122, 130, 139, 149, 157, 158, 168, 179], "describ": [6, 7, 10, 14, 21, 24, 25, 28, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 114, 117, 118, 126, 134, 158, 159, 160, 169], "abov": [6, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 122, 131, 158], "its": [6, 7, 22, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 113, 117, 118, 122, 131, 139, 157, 158, 168, 169], "go": [6, 7, 9, 18, 21, 22, 25, 27, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 118, 119, 120, 122, 123, 125, 126, 129, 131, 132, 136, 137, 149, 150, 152, 153, 158, 167, 169, 176], "critic": [6, 7, 11, 109, 117, 141, 158, 169, 172, 177], "telescop": [6, 7, 16, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 137, 141, 160, 167, 169], "dashboard": [6, 7, 16, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 115, 169], "color": [6, 7, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 169], "reflect": [6, 7, 117, 160, 169], "health": [6, 7, 19, 23, 24, 27, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 114, 117, 169, 177], "8": [7, 11, 14, 18, 22, 27, 33, 35, 36, 40, 45, 50, 55, 60, 65, 70, 75, 80, 81, 85, 90, 95, 100, 105, 107, 110, 113, 117, 126, 136, 137, 139, 141, 152, 156, 167, 168, 169], "16": [7, 14, 22, 28, 40, 46, 90, 109, 110, 113, 117, 122, 126, 139, 161, 168, 169, 173, 174], "22": [7, 32, 33, 35, 36, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102, 109, 117, 120, 122, 131, 132, 137, 141, 156, 158, 167, 169, 173], "ship": [7, 169], "two": [7, 9, 11, 16, 22, 25, 26, 28, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 110, 113, 117, 120, 125, 126, 132, 139, 141, 149, 151, 153, 164, 166, 169, 172, 177, 179], "defin": [7, 9, 10, 14, 15, 16, 19, 20, 21, 24, 27, 30, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 93, 96, 98, 101, 103, 106, 109, 113, 114, 117, 122, 129, 131, 132, 138, 139, 142, 148, 150, 151, 159, 165, 168, 169, 170, 172, 174, 179], "site": [7, 10, 15, 16, 22, 24, 27, 28, 30, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 113, 114, 117, 120, 125, 129, 132, 136, 140, 141, 142, 150, 151, 152, 153, 157, 163, 164, 168, 169, 170, 172, 174, 175, 178], "softgate1": [7, 28, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 120, 132, 149, 169], "softgate2": [7, 22, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 169], "we": [7, 22, 27, 28, 30, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 110, 113, 117, 119, 120, 122, 125, 126, 131, 132, 136, 141, 149, 151, 152, 153, 157, 161, 163, 166, 167, 169], "new": [7, 11, 13, 16, 22, 24, 25, 30, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 112, 113, 114, 117, 119, 122, 125, 129, 131, 132, 134, 136, 137, 138, 139, 141, 145, 150, 151, 152, 153, 160, 161, 163, 167, 168, 169, 172, 174, 175, 177, 178], "altern": [7, 24, 28, 109, 168, 169, 170, 174], "learn": [7, 28, 32, 34, 36, 37, 39, 41, 42, 44, 46, 47, 49, 51, 52, 54, 56, 57, 59, 61, 62, 64, 66, 67, 69, 71, 72, 74, 76, 77, 79, 81, 82, 84, 86, 87, 89, 91, 92, 94, 96, 97, 99, 101, 102, 104, 106, 113, 117, 118, 169], "how": [7, 14, 16, 17, 24, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103, 113, 117, 119, 126, 169], "keep": [7, 10, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 169], "mind": [7, 10, 169], "uniqu": [7, 9, 10, 11, 21, 23, 24, 28, 31, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 157, 160, 163, 169, 174, 178], "each": [7, 10, 15, 18, 21, 22, 23, 25, 27, 28, 30, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 110, 112, 113, 116, 117, 118, 120, 129, 131, 132, 137, 138, 139, 143, 147, 148, 150, 151, 157, 160, 163, 167, 169, 170, 172, 174, 177, 178, 179], "replac": [7, 11, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 156, 158, 169, 173], "regular": [7, 16, 27, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 142, 149, 151, 158, 169], "attempt": [7, 161, 169], "migrat": [7, 28, 169], "prior": [7, 11, 26, 122, 131], "handoff": [7, 109, 169], "bond0": [7, 32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102, 117, 122, 131, 158, 169, 172], "subinterfac": [7, 157, 163, 169, 172, 174], "xyz": [7, 169], "But": [7, 169], "bind": [7, 14, 117, 160, 169], "onli": [7, 9, 10, 18, 21, 25, 27, 28, 31, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 112, 113, 114, 117, 119, 126, 129, 132, 150, 151, 153, 169, 177, 178, 179], "make": [7, 11, 14, 22, 26, 27, 32, 33, 35, 37, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 117, 118, 126, 141, 151, 153, 157, 160, 161, 163, 169, 178], "chang": [7, 10, 14, 18, 22, 27, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 113, 117, 118, 120, 122, 126, 131, 132, 136, 137, 141, 151, 152, 153, 160, 165, 167, 169, 170, 172, 178], "remain": [7, 25, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 169], "trunk": [7, 122, 131, 158, 166, 169], "en": [7, 158, 169], "bond": [7, 25, 117, 158, 169], "ensx": [7, 158, 169], "actual": [7, 28, 31, 114, 117, 158, 169, 177], "slave": [7, 158, 169], "adjust": [7, 20, 114, 169], "mode": [7, 16, 25, 28, 31, 33, 35, 38, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 93, 95, 98, 100, 103, 105, 109, 112, 169, 178], "accord": [7, 27, 114, 117, 159, 169], "desir": [7, 13, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 114, 122, 131, 151, 153, 159, 168, 169], "activ": [7, 10, 16, 22, 25, 26, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 110, 113, 117, 125, 134, 145, 156, 168, 169, 173, 178], "backup": [7, 16, 17, 27, 125, 141, 169], "maintain": [7, 15, 28, 169, 176, 179], "ai": [8, 16, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "newnet0p": [8, 14, 15, 18, 136, 137, 141, 152, 167], "descript": [9, 10, 20, 22, 28, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 112, 113, 114, 117, 120, 122, 129, 131, 132, 149, 150, 153, 168, 172, 177], "field": [9, 10, 13, 16, 24, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 113, 122, 126, 129, 131, 132, 149, 150, 153, 157, 168, 172, 177], "full": [9, 27, 28, 31, 109, 170], "e": [9, 11, 15, 28, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 119, 123, 125, 160, 161, 163, 170, 172, 177, 179], "mail": [9, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "email": [9, 10, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106], "notif": [9, 10], "retriev": [9, 122, 131], "cc": 9, "send": [9, 27, 125], "phone": 9, "compani": [9, 31], "usual": [9, 28, 114, 179], "multi": [9, 16, 109, 117, 141, 164, 178], "posit": [9, 114], "within": [9, 28, 32, 33, 35, 37, 38, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 119, 122, 123, 131, 141, 142, 148, 156, 168, 173, 176, 179], "object": [9, 10, 16, 22, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 160], "rbac": [9, 10, 15, 109], "deactiv": [9, 25, 26], "view": [9, 16, 21, 31, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 87, 91, 96, 101, 106, 109, 117, 122, 131, 138, 139, 159, 170, 177, 178], "edit": [9, 10, 14, 22, 23, 26, 28, 31, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 109, 112, 114, 117, 122, 126, 131, 138, 153, 157, 172, 176, 178, 179], "part": [9, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 113, 117, 151, 157, 163, 166, 177], "assign": [9, 15, 24, 28, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 109, 112, 114, 117, 122, 131, 139, 151, 153, 157, 174, 177, 178, 179], "variou": [9, 15, 28, 30, 107, 117, 139, 151, 176, 177, 179], "read": [9, 33, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 126, 129, 132], "form": [9, 25, 27, 28, 30, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 170], "list": [9, 15, 16, 20, 21, 22, 23, 24, 28, 31, 34, 35, 36, 39, 40, 41, 44, 45, 46, 49, 50, 51, 54, 55, 56, 59, 60, 61, 64, 65, 66, 69, 70, 71, 74, 75, 76, 79, 80, 81, 84, 85, 86, 89, 90, 94, 95, 96, 99, 100, 101, 104, 105, 106, 109, 112, 113, 114, 117, 120, 131, 132, 137, 140, 141, 142, 143, 149, 157, 163, 167, 177, 178], "own": [9, 15, 28, 118, 122, 131, 138, 139, 157], "concept": [9, 117, 135, 141, 144, 146, 147, 164, 175, 179], "share": [9, 15, 28, 31, 109, 110, 117, 179], "deleg": [9, 15, 109, 126, 153, 168], "team": [9, 10, 15, 25, 116, 117, 178], "grant": [9, 15], "request": [9, 10, 15, 22, 24, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 116, 119, 120, 125, 127, 132, 134, 136, 137, 141, 145, 149, 152, 153, 155, 157, 162, 167, 171], "self": [9, 11, 15], "portal": [9, 15, 109, 127, 129, 132, 150, 155, 162, 171], "programmat": [9, 15], "kubernet": [9, 12, 13, 15, 16, 17, 21, 24, 27, 28, 30, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104, 109, 110, 126, 127, 153, 155, 162, 168, 171, 179], "crd": [9, 15, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 127, 155, 162, 171], "terraform": [9, 15, 16, 27, 109, 117, 118, 126, 127, 153, 155, 157, 162, 163, 168, 171, 179], "devop": [9, 15], "netop": [9, 15, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "pipelin": [9, 15], "just": [9, 11, 22, 27, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 117, 126, 160], "basi": [9, 10, 15, 178], "attach": [9, 15, 20, 31, 114, 138, 151, 178], "individu": [9, 15, 18, 23, 28, 31, 118, 129, 132, 137, 148, 150, 167, 170, 178, 179], "everi": [9, 23, 28, 30, 31, 114, 117, 157, 163], "attribut": [9, 15, 26, 28, 30, 170, 177], "particular": [9, 22, 24, 26, 27, 28, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 120, 132, 163, 177, 178], "either": [9, 10, 11, 19, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 92, 93, 97, 98, 102, 103, 117, 126, 138, 153, 156, 157, 160, 168, 173], "link": [9, 15, 16, 26, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 109, 113, 116, 117, 124, 125, 159, 164, 166, 177, 179], "directli": [9, 22, 27, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 112, 117], "support": [10, 11, 12, 14, 16, 25, 27, 28, 29, 30, 31, 112, 113, 117, 122, 126, 128, 131, 133, 134, 135, 153, 156, 161, 168, 173, 177, 178], "acl2": 10, "destin": [10, 20, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 114, 121, 130, 131, 138, 156, 173], "format": [10, 28, 178], "orient": [10, 26], "wai": [10, 11, 18, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 121, 130, 136, 137, 152, 157, 163, 167], "both": [10, 22, 25, 26, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 112, 117, 119, 122, 125, 126, 153], "tenant": [10, 15, 16, 21, 22, 24, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 109, 112, 113, 114, 117, 122, 126, 131, 153, 157, 168, 177, 178], "execut": [10, 11, 28, 113, 117, 118, 158, 160, 161], "hardwar": [10, 16, 21, 25, 28, 31, 32, 33, 35, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 92, 93, 95, 97, 98, 100, 102, 103, 105, 107, 113, 114, 117, 122, 131, 135, 152, 158, 164, 177], "rate": [10, 23, 117], "secur": [10, 15, 16, 19, 20, 27, 30, 114, 117, 121, 122, 130, 131, 133, 134, 135, 138, 145, 148, 164, 175, 176, 179], "enforc": 10, "It": [10, 11, 18, 21, 22, 25, 26, 27, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 113, 117, 122, 126, 131, 137, 139, 149, 153, 156, 167, 168, 173], "import": [10, 15, 16, 18, 117, 122, 126, 131, 136, 137, 141, 152, 153, 167], "size": [10, 107], "tcam": 10, "seen": [10, 22, 25, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "sever": [10, 25, 26, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 143, 157, 163], "optim": [10, 31, 109, 117, 142, 151, 166], "algorithm": 10, "minim": [10, 14, 26, 126, 136], "usag": [10, 21, 27, 28, 113, 117, 139, 151], "achiev": [10, 122, 131, 172], "permit": [10, 22, 23, 28, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 113], "commun": [10, 16, 27, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 114, 121, 130, 133, 134, 135, 151, 159, 166, 172, 179], "deni": [10, 20, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 114, 138, 148], "given": [10, 23, 24, 28, 31, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 157, 178], "drop": [10, 11, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 160], "traffic": [10, 20, 23, 25, 26, 27, 28, 31, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 110, 114, 117, 119, 121, 122, 130, 131, 138, 142, 151, 156, 157, 160, 163, 166, 173, 174, 177, 178, 179], "unless": [10, 178], "sitedefault": 10, "entri": [10, 15, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 122, 131], "protocol": [10, 22, 23, 24, 25, 27, 28, 31, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 104, 105, 131, 151], "match": [10, 28, 33, 35, 38, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 93, 95, 98, 100, 103, 105, 122, 131], "tcp": [10, 22, 23, 24, 26, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 127, 137, 141, 155, 162, 167, 171], "udp": [10, 24, 28, 114, 127, 155, 162, 171], "icmp": [10, 28], "code": [10, 109], "icmpv6": 10, "ipv6": [10, 15, 16, 20, 32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 92, 96, 97, 101, 102, 106, 114, 178], "date": [10, 11, 160, 161], "action": [10, 19, 26, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 112, 122, 131, 172], "packet": [10, 27, 28, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 109, 126, 151, 153, 166, 177], "revers": 10, "For": [10, 13, 24, 28, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 112, 113, 117, 120, 122, 126, 129, 131, 132, 148, 149, 150, 157, 160, 163, 172, 179], "except": [10, 20, 114, 138], "syn": 10, "flag": 10, "non": [10, 25, 28, 40, 90, 109, 117, 158], "swap": [10, 117], "type": [10, 11, 16, 21, 24, 28, 29, 30, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 107, 113, 114, 117, 122, 126, 129, 131, 132, 139, 148, 149, 150, 152, 153, 156, 168, 172, 173, 177], "rang": [10, 15, 21, 28, 109, 112, 119, 131, 139, 148, 151, 157, 165, 166, 172, 174, 176, 179], "window": [10, 21, 27, 28, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 157, 158, 163, 172], "group": [10, 15, 16, 109, 110, 117, 122, 174], "commonli": [10, 31, 109], "button": [10, 21, 23, 28, 31, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 87, 93, 98, 103, 114, 122, 131, 153, 157, 163, 168, 177], "anoth": [10, 27, 28, 40, 90, 117, 139, 157, 163], "alreadi": [10, 13, 14, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 150], "ssh": [10, 20, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 109, 114, 115, 120, 122, 131, 132, 137, 138, 141, 149, 156, 157, 160, 161, 167, 173], "show": [10, 28, 30, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 117, 122, 125, 131, 161, 177, 179], "broader": [10, 28], "respons": [10, 27, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 88, 91, 96, 101, 106, 117, 142, 151], "repres": [10, 27], "qa_ten": 10, "belong": [10, 22, 24, 28, 31, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 95, 96, 100, 101, 105, 106, 112, 114, 117], "stai": 10, "receiv": [10, 27, 28, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 118, 122, 170], "gui": [10, 14, 15, 24, 27, 28, 30, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 88, 89, 92, 97, 102], "reject": 10, "soon": [10, 20, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 114], "push": [10, 109], "appropri": [10, 28, 109, 117, 118, 122, 126, 131, 153, 157, 160, 163, 168], "throughout": [10, 117, 124], "fabric": [10, 23, 26, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 113, 117, 122, 129, 131, 132, 138, 148, 150, 178], "rest": [10, 21, 28, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102, 117, 119, 124, 129, 132, 150, 179], "disk": [11, 14, 18, 109, 110, 136, 137, 141, 152, 167, 177], "50gb": [11, 18, 136, 137, 141, 152, 167], "bit": [11, 18, 136, 137, 141, 152, 167, 177], "most": [11, 31, 118, 126, 136, 152, 153, 156, 168, 170, 173], "modern": [11, 29, 30, 31, 110], "oss": 11, "raspbian": 11, "buster": 11, "step": [11, 12, 14, 18, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 113, 114, 117, 119, 120, 121, 125, 130, 132, 135, 137, 138, 145, 149, 154, 157, 160, 161, 164, 175], "legaci": [11, 19, 29, 31], "iptabl": 11, "alpin": 11, "setup": [11, 16, 17, 20, 114, 124, 151, 158, 169, 175], "red": [11, 117, 160], "hat": 11, "cento": 11, "enterpris": [11, 109], "sfl": [11, 13, 18, 136, 137, 141, 152, 160, 167], "abl": [11, 18, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 125, 126, 136, 137, 141, 152, 153, 157, 158, 167], "log": [11, 16, 22, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 117, 122, 131], "doe": [11, 13, 25, 26, 117, 165, 167], "cert": [11, 14], "helm": [11, 13, 16, 17, 18, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 136, 137, 141, 152, 160, 167], "chart": [11, 13, 16, 17, 18, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 136, 137, 141, 152, 167, 177], "order": [11, 12, 16, 22, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 122, 131, 157, 163], "ingress": [11, 12, 131, 178], "sign": 11, "certif": [11, 18, 136, 137, 141, 152, 167], "box": [11, 19, 21, 23, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 142], "pass": [11, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 151], "cluster": [11, 13, 18, 22, 30, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 104, 105, 109, 110, 115, 136, 137, 141, 152, 167], "valu": [11, 12, 21, 22, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 117, 122, 129, 131, 132, 150, 165], "clusterissu": 11, "selfsign": 11, "letsencrypt": [11, 136, 137, 141, 152, 167], "acm": 11, "http01": 11, "challeng": 11, "valid": [11, 28, 109, 113, 157], "cname": 11, "record": [11, 18, 136, 137, 141, 152, 167], "domain": [11, 12, 18, 20, 25, 29, 114, 136, 137, 141, 152, 167], "subdomain": 11, "exist": [11, 13, 16, 23, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 121, 130, 131, 151, 163, 166, 168, 172], "simplest": 11, "issu": [11, 117], "behind": [11, 16, 28, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 119, 124, 178], "public": [11, 18, 28, 30, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 109, 119, 122, 131, 134, 136, 137, 141, 145, 148, 149, 151, 152, 153, 156, 157, 164, 167, 173, 177], "common": [11, 20, 21, 22, 28, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 113, 114, 126, 139, 168], "approach": [11, 19, 141, 142], "privat": [11, 14, 21, 28, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 109, 114, 125, 126, 128, 139, 148, 151, 153, 156, 157, 168, 170, 173], "dns01": 11, "familiar": [11, 113], "specifi": [11, 21, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 113, 122, 131, 157, 178], "yaml": [11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "suitabl": 11, "f": [11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 96, 100, 105], "my": [11, 12, 18, 113, 136, 137, 141, 152, 167], "rerun": 11, "latest": [11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "version": [11, 16, 22, 27, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 158, 161], "simpli": [11, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 124, 155, 157, 162, 171], "newer": 11, "few": [11, 22, 26, 28, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 117, 119, 120, 126, 132, 149, 157], "data": [11, 15, 16, 27, 28, 29, 30, 31, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 113, 114, 117, 118, 148, 161, 177], "mariadb": [11, 160], "highli": [11, 27, 110, 151, 166], "cronjob": 11, "mysqldump": [11, 160], "databas": [11, 16, 27, 156, 157, 163, 173], "snapshot": [11, 160], "n": [11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 160], "c": [11, 31, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 160], "u": [11, 22, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 117, 122, 160], "mariadb_us": [11, 160], "mariadb_password": [11, 160], "mariadb_databas": [11, 160], "db": [11, 160], "m": [11, 18, 24, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 126, 153, 156, 173], "sql": [11, 160], "find": [11, 23, 27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 88, 92, 97, 102, 117, 118, 126, 153, 157], "yyyi": 11, "mm": 11, "dd": [11, 110], "hh": 11, "re": [11, 26, 113, 148], "move": [11, 17, 18, 28, 114, 119, 125, 136, 137, 141, 151, 152, 166, 167], "reinstal": [11, 117], "sens": [11, 120, 126, 132, 149, 151, 153], "purpos": [11, 21, 24, 28, 30, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 113, 114, 116, 117, 122, 126, 128, 131, 139, 147, 156, 168, 173], "futur": [11, 21, 28, 114, 117, 120, 126, 128, 132, 141, 149, 153, 157, 176, 178, 179], "overwis": 11, "reiniti": 11, "grpc": [11, 27, 137, 141, 160, 167], "jsonpath": 11, "mysql": [11, 160], "root": [11, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 117, 120, 132, 160, 161], "mariadb_root_password": [11, 160], "contain": [11, 13, 22, 27, 31, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 109, 117, 131, 160], "cp": [11, 160], "opt": [11, 117, 157, 160], "too": [11, 120, 132, 149], "place": [11, 30, 114, 159, 161], "old_secret": 11, "taken": [11, 122, 131, 160], "export": [11, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "vudodffsakjcu2lfvva4t1c0cnpuumdimkqxem85y2dns3pkajlnsg": 11, "patch": [11, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "json": [11, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117], "op": [11, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "microservic": 11, "12": [12, 36, 106, 109, 110, 117, 139, 161, 168], "pv": 12, "provision": 12, "underli": [12, 32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102, 114, 159], "repositori": [12, 14, 16, 17], "netrisai": [12, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "github": [12, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "namespac": [12, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "releas": [12, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 128, 178], "readm": 12, "about": [12, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 92, 93, 97, 98, 102, 103, 113, 114, 117, 126, 151, 153], "paramet": [12, 15, 28, 113, 117, 129, 131, 132, 150, 178], "offer": [13, 25, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 107, 176, 179], "simplifi": 13, "model": [13, 25, 28, 110, 148], "who": [13, 109, 114, 117, 168, 178], "quickli": [13, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "shortest": 13, "amount": 13, "streamlin": 13, "wish": [13, 131], "instead": [13, 19, 22, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 122, 131, 142, 156, 172, 173], "w": [13, 22, 109], "credenti": [13, 14, 18, 22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 113, 118, 136, 137, 141, 150, 152, 167], "ipam": [13, 16, 24, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 93, 96, 98, 101, 103, 106, 109, 113, 114, 117, 118, 122, 126, 131, 147, 151, 164, 170, 175, 178], "plan": [13, 16, 28, 124, 139, 141, 179], "upper": [13, 151], "fill": [13, 21, 28, 31, 114, 125], "proper": [13, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 113, 117, 157, 174], "selector": 13, "congratul": [13, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "100gb": 14, "qemu": 14, "virt": 14, "contact": [14, 118], "permiss": [14, 15, 16, 117], "cd": [14, 117, 118], "libvirt": 14, "wget": 14, "img": 14, "controller3": 14, "qcow2": 14, "definit": [14, 16, 114], "xml": 14, "virsh": 14, "br": 14, "correct": [14, 123, 141, 160], "bridg": [14, 178], "autostart": 14, "obtain": [14, 157, 161, 174], "forget": 14, "passwd": 14, "reload": 14, "sure": [14, 22, 26, 32, 33, 35, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 141, 156, 157, 160, 161, 173], "browser": [14, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 117, 118, 161], "pem": 14, "kei": [14, 22, 25, 28, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 129, 132, 161], "systemctl": [14, 112, 117, 160], "nomenclatur": 15, "understand": [15, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "account": [15, 16, 18, 136, 137, 141, 152, 167], "restapi": [15, 27, 109], "vpc": [15, 16, 19, 32, 33, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 92, 93, 97, 98, 102, 103, 109, 114, 117, 119, 121, 125, 126, 130, 139, 142, 144, 146, 148, 153, 156, 157, 168, 173], "logic": [15, 117, 170, 176, 179], "segreg": [15, 176, 179], "act": [15, 28, 176, 179], "tradit": [15, 28, 109, 142, 157, 170, 176, 179], "emploi": [15, 142, 176, 179], "overlap": [15, 21, 28, 109, 165, 176, 179], "across": [15, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 176, 177, 179], "role": [15, 16, 21, 28, 109, 117, 126, 139, 153, 157, 168], "separ": [15, 113, 126, 147, 157, 172], "center": [15, 27, 28, 29, 31, 114, 148], "unit": [15, 28, 112, 114], "global": [15, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 109, 157, 163, 174], "AS": [15, 22, 28, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 114, 125, 170], "acl": [15, 16, 34, 35, 39, 40, 44, 45, 49, 50, 54, 55, 59, 60, 64, 65, 69, 70, 74, 75, 79, 80, 84, 85, 89, 90, 94, 95, 99, 100, 104, 105, 109, 126, 148, 156, 157, 163, 173, 174, 179], "mesh": [15, 16, 22, 27, 28, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 140], "vpn": [15, 27, 28, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 109, 140, 142, 179], "level": [15, 22, 27, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 177], "endpoint": [15, 124, 179], "bare": [15, 31, 109, 126, 149, 151, 153, 154, 156, 157, 163, 179], "metal": [15, 16, 31, 109, 125, 126, 149, 151, 153, 154, 156, 157, 179], "cloud": [15, 17, 19, 21, 22, 24, 27, 28, 30, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 117, 121, 127, 130, 131, 142, 151, 154, 155, 162, 171, 179], "bgp": [15, 16, 26, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 113, 117, 123, 124, 125, 126, 134, 145, 163, 164, 177, 178, 179], "peer": [15, 16, 22, 26, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 109, 119, 125, 170], "ibgp": 15, "ebgp": [15, 125], "join": [16, 133, 134, 135], "our": [16, 22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 127, 133, 134, 135, 155, 162, 171], "slack": [16, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 133, 134, 135, 160], "channel": [16, 25, 133, 134, 135], "engin": [16, 27, 133, 134, 135], "introduct": [16, 135, 144, 146, 147, 164, 175], "platform": [16, 112, 113, 115, 117, 118, 131, 157, 161, 163, 174], "matrix": 16, "construct": [16, 30, 117], "overlai": 16, "administr": [16, 24, 28, 117, 165], "hypervisor": [16, 28, 31, 166], "worker": [16, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "plane": [16, 22, 27, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 124, 178], "variat": 16, "compat": [16, 143], "horizont": [16, 109], "scalabl": [16, 109], "dell": [16, 109, 111], "edgecor": [16, 109, 111, 161], "arista": [16, 109], "architectur": [16, 28, 31, 115, 168, 170], "simul": [16, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 115], "lab": [16, 115], "scenario": [16, 22, 32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105, 115, 118, 153, 156, 157, 160, 163, 170, 173], "gpu": [16, 109, 115], "spectrum": [16, 25, 109, 110, 115], "v": [16, 21, 25, 28, 30, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 112, 117, 124, 125, 134, 139, 145, 147, 151, 164, 168, 177], "sitemesh": [16, 109], "nat": [16, 21, 27, 30, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 110, 119, 134, 139, 142, 145, 147, 151, 157, 163, 164, 174, 175], "load": [16, 19, 21, 22, 25, 27, 28, 30, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 110, 117, 119, 134, 139, 142, 145, 147, 151, 157, 161, 163, 164, 175, 177], "balanc": [16, 19, 21, 22, 25, 27, 28, 30, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 110, 119, 134, 139, 142, 145, 147, 151, 164, 175, 177], "upgrad": [16, 17, 117, 120, 132, 133, 149], "rollback": [16, 133], "anywher": [16, 31, 122, 178], "guid": [16, 24, 121, 122, 130, 131, 143, 154], "overview": [16, 135, 144, 146, 147, 164, 175], "upstream": [16, 28, 32, 34, 37, 39, 42, 44, 47, 49, 52, 54, 57, 59, 62, 64, 67, 69, 72, 74, 77, 79, 82, 84, 89, 92, 94, 97, 99, 102, 104, 164], "demand": [16, 22, 24, 30, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104, 109, 127, 134, 145, 147, 151, 164], "elast": [16, 109, 127, 134, 145, 147, 151, 164], "l4": [16, 22, 30, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 109, 127, 134, 145, 147, 151, 164], "equinix": [16, 109, 126], "api": [16, 22, 30, 35, 36, 40, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 95, 96, 100, 101, 105, 106, 134, 145, 151, 157, 163, 179], "integr": [16, 24, 27, 30, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 127, 134, 145, 151, 166, 179], "project": [16, 125, 126, 129, 132, 145], "pool": [16, 21, 24, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 119, 128, 134, 139, 145], "phoenixnap": [16, 109, 153], "bmc": [16, 109, 117, 147, 153], "aw": [16, 18], "gcp": 16, "softwar": [16, 19, 27, 28, 114, 117, 121, 122, 130, 131, 135, 136, 137, 141, 142, 152, 156, 159, 161, 164, 167, 173, 175], "isp": [16, 28, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104, 113, 175, 177, 179], "method": [16, 25, 28, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 127, 155, 162, 171, 177], "manifest": [16, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "loadbalanc": [16, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "l4lb": [16, 21, 26, 28, 30, 36, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106, 179], "reclaim": 16, "calico": [16, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104], "cni": [16, 30, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104], "prepar": 16, "l3": [16, 30, 31, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 91, 94, 96, 99, 101, 104, 106, 113], "anycast": [16, 30, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 91, 94, 96, 99, 101, 104, 106, 109], "lb": [16, 33, 36, 38, 40, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 90, 93, 96, 98, 101, 103, 106, 177], "consum": [16, 17, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 109, 126, 127, 151, 153, 155, 157, 162, 166, 168, 171], "rule": [16, 20, 30, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 88, 92, 93, 97, 98, 102, 103, 107, 109, 114, 117, 122, 126, 131, 138, 153, 156, 157, 168, 173, 174], "approv": [16, 38, 88, 116, 118], "workflow": 16, "process": [16, 18, 22, 27, 33, 35, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 109, 114, 117, 121, 130, 136, 137, 138, 141, 152, 154, 157, 158, 160, 161, 167], "roh": [16, 21, 23, 28, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 91, 93, 96, 98, 101, 103, 106], "alloc": [16, 24, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106, 113, 122, 128, 131, 139, 147, 149, 151, 152, 164, 179], "tree": [16, 21, 22, 31], "basic": [16, 113, 117, 118, 126, 160], "advanc": [16, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 123, 131], "map": [16, 109], "glass": [16, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103, 109, 170], "mainten": [16, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 109, 117], "happen": [16, 120, 132, 149], "scene": [16, 119, 124, 178], "procedur": [16, 26, 27, 117, 133, 151, 166], "restor": [16, 17, 27], "downgrad": 16, "visibl": [16, 28, 157, 163], "graph": 16, "board": 16, "profil": [16, 30, 31, 109, 117, 138], "aggreg": [16, 31, 109, 112, 124], "lag": [16, 109, 112, 178], "evpn": [16, 26, 28, 32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102, 109, 117, 124, 142, 178], "home": [16, 28, 117, 118, 178], "v5": [16, 111], "switchdev": [16, 30, 111], "sonic": [16, 109, 110, 111], "v3": [16, 35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 111, 163], "7": [16, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 76, 80, 85, 90, 95, 100, 105, 107, 109, 111, 117], "minimum": [16, 110, 122, 131, 135, 152, 164], "pro": [16, 27, 109, 117, 120, 128, 132, 149, 175], "bio": [16, 158], "intro": [16, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104], "applic": [16, 17, 22, 23, 27, 30, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 114, 117, 160], "On": [16, 30, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 91, 94, 99, 104, 117, 125], "vm": [17, 27, 28, 31, 109, 117, 126, 130, 136, 146, 153, 156, 157, 163, 174, 179], "prem": [17, 21, 27, 28, 30], "anytim": [17, 117], "ONE": 17, "repo": 17, "info": [17, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 117, 119, 136, 137, 141, 152, 157, 167], "why": [17, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 119, 136, 137, 141, 152, 167], "cach": [17, 27], "almost": [18, 136, 137, 141, 152, 167], "fact": [18, 31, 137, 167], "multipl": [18, 23, 24, 25, 27, 109, 112, 136, 137, 152, 157, 163, 167, 174, 177, 178, 179], "doesn": [18, 31, 137, 153, 156, 167, 173], "matter": [18, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 133, 134, 135, 145, 153, 164, 175], "am": [18, 156, 173], "machin": [18, 28, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 136, 137, 141, 152, 167], "ec2": [18, 121, 144], "got": [18, 30, 136, 152, 167], "54": [18, 122, 137, 141, 167], "219": 18, "211": [18, 42, 43, 46], "71": 18, "like": [18, 19, 21, 22, 28, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 119, 125, 126, 153, 156, 163, 168, 173, 179], "easier": [18, 136, 137, 141, 152, 167], "potenti": [18, 114, 136, 137, 141, 152, 159, 167], "somewher": [18, 136, 137, 152, 167], "cloudflar": [18, 136, 137, 141, 152, 167], "inde": [18, 136, 141, 152, 167], "liner": [18, 120, 132, 149, 158, 160], "stand": [18, 136, 152, 167], "top": [18, 19, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 92, 93, 97, 98, 102, 103, 117, 122, 131, 136, 137, 141, 142, 152, 153, 167, 168, 177], "let": [18, 19, 22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 113, 117, 122, 131, 136, 137, 141, 152, 157, 163, 167], "encrypt": [18, 27, 28, 30, 136, 137, 141, 151, 152, 166, 167], "ssl": [18, 136, 137, 141, 152, 167], "That": [18, 126, 136, 137, 141, 151, 152, 153, 157, 163, 167], "consonl": 18, "immedi": [18, 117, 173], "someth": [18, 117, 125, 126, 153], "strong": 18, "whitelist": 18, "autom": [19, 117], "abstract": [19, 142], "builder": 19, "bring": [19, 119, 120, 132, 149], "continu": [19, 25, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 157], "monitor": [19, 109, 114, 115, 137, 139, 141, 157, 159, 167, 177], "remedi": [19, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "human": 19, "intervent": 19, "awai": [19, 26], "complex": [19, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102, 142], "effici": [19, 121, 130, 157, 163], "down": [19, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 112, 117, 142, 177], "harden": [20, 114, 138], "flow": [20, 114, 138, 151, 156, 166, 173], "As": [20, 22, 25, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 114, 122, 131, 157, 174], "ntp": [20, 30, 109, 114, 138], "free": [20, 22, 28, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 112, 114, 119, 165], "text": [20, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 114, 157], "timezon": [20, 30, 109, 114], "track": [21, 28], "design": [21, 22, 27, 28, 30, 31, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 109, 117, 118, 156, 166, 170, 173], "opportun": [21, 28], "kind": [21, 22, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117], "There": [21, 22, 25, 28, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 113, 117, 139, 151], "main": [21, 27, 28, 30, 113, 114, 117, 122, 131, 156, 157, 173], "organ": [21, 28, 30, 139, 178], "rir": [21, 28, 139], "lir": [21, 28, 139], "child": [21, 28], "parent": [21, 28, 139], "bottom": [21, 23, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103, 117], "owner": [21, 22, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 112, 114, 178], "ust": [21, 28], "experi": [21, 22, 26, 28, 117, 118], "itself": [21, 28, 117], "inact": [21, 28, 122, 131, 153], "reserv": [21, 27, 28, 30, 114, 153, 166], "kube": [22, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "complement": [22, 30], "between": [22, 23, 27, 28, 30, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 109, 114, 117, 119, 121, 122, 126, 130, 151, 153, 157, 163, 166, 170, 174, 178, 179], "accomplish": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 125, 131], "master": [22, 40, 90, 160], "secret": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 150], "cred": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "liter": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "inspect": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "l": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105], "demonstr": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105], "success": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 113], "1629994653": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "6441543": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "logger": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "msg": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "reconcilergroup": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "k8": [22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106], "reconcilerkind": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "count": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 107, 114, 117, 161, 177], "simpl": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 118], "podinfo": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "k": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "stefanprodan": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "kustom": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "po": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "svc": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "clusterip": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "ag": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 160], "576d5bf6bd": [22, 40, 90], "7z9jl": [22, 40, 90], "49": [22, 40, 73, 90, 161], "nhlmh": [22, 40, 90], "33": [22, 40, 68, 90], "172": [22, 32, 33, 36, 37, 38, 40, 41, 42, 43, 46, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 90, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 113, 117, 122, 139, 168, 173, 174], "21": [22, 32, 33, 35, 36, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 90, 92, 95, 97, 100, 102, 105, 107, 117], "65": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 76, 80, 85, 90, 95, 100, 105], "106": [22, 40, 90, 101], "none": [22, 24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 122, 125, 131], "9898": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "9999": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "50": [22, 28, 38, 40, 41, 82, 83, 85, 86, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106], "outsid": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 117, 125, 139, 156, 157, 163, 168, 173, 174, 178], "spec": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110], "again": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105], "32584": [22, 40, 90], "30365": [22, 40, 90], "8m57": [22, 40, 90], "real": [22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 170], "becaus": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 153, 161], "xxxxxxxx": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "been": [22, 26, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 112, 114, 117, 125, 131, 151, 153, 157, 159, 160, 168], "inject": [22, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "117": [22, 38, 40, 41, 43, 45, 46, 82, 83, 85, 86, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106], "59": [22, 38, 40, 41, 82, 83, 85, 86, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106], "202": [22, 67, 68, 71], "9m17": [22, 40, 90], "previou": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 122, 125, 131, 157, 163], "detect": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 157, 163], "them": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117, 126, 151, 153, 157, 163, 166, 168, 177], "frontend": [22, 24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 139, 177], "66d44feb": [22, 40, 90], "0278": [22, 40, 90], "412a": [22, 40, 90], "a32d": [22, 40, 90], "73afe011f2c6": [22, 40, 90], "nyc": [22, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106], "33m": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "32m": [22, 40, 90], "recreat": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "wa": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 116, 131, 168], "origin": [22, 26, 28, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 156, 173], "being": [22, 24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 122, 131, 157, 160, 161, 163], "nativ": [22, 24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 127, 155, 162, 171, 179], "schema": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "backend": [22, 24, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 109, 117, 169, 177], "srv04": [22, 32, 35, 36, 37, 40, 41, 42, 45, 46, 47, 50, 51, 52, 55, 56, 57, 60, 61, 62, 65, 66, 67, 70, 71, 72, 75, 76, 77, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 102, 105, 106], "srv05": [22, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106], "80": [22, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 77, 78, 80, 81, 85, 90, 95, 100, 105, 137, 141, 167], "These": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 160], "index": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "html": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 161], "cat": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "eof": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "apivers": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "v1alpha1": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "metadata": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 119, 151, 157, 163], "ownerten": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "45": [22, 32, 33, 35, 36, 37, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 85, 86, 87, 90, 91, 92, 95, 96, 97, 100, 101, 102, 105, 106, 110], "46": [22, 33, 35, 36, 37, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 87, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106], "timeout": [22, 24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117], "3000": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 151, 157], "And": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 113, 126, 153, 157, 163, 179], "d07acd0f": [22, 40, 90], "51ea": [22, 40, 90], "429a": [22, 40, 90], "89dd": [22, 40, 90], "8e4c1d6d0a86": [22, 40, 90], "2m17": [22, 40, 90], "3m47": [22, 40, 90], "203": [22, 72, 73, 76, 113], "try": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 116, 117, 118, 161], "l2": [22, 29], "segment": [22, 31, 124, 178, 179], "vnet": [22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113], "guestten": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "switchport": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "swp2": [22, 37, 38, 40, 88, 90, 113], "sw22": [22, 32, 38, 40, 42, 47, 52, 57, 62, 67, 72, 77, 82, 88, 90, 92, 97, 102], "incom": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 137, 141, 156, 167, 173], "isp2": [22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105], "neighbora": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "65007": [22, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106], "transport": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 151, 166], "swp14": [22, 88, 90], "sw02": [22, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 87, 88, 90, 93, 95, 96, 98, 100, 101, 103, 105, 106], "vlanid": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 117], "1092": [22, 103, 105, 106], "localip": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "118": [22, 43, 45, 46, 103, 105, 106], "30": [22, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 110, 113, 122, 125, 126, 131, 170], "remoteip": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "prefixlistoutbound": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "28": [22, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 117, 126, 153], "le": [22, 28, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 93, 95, 96, 98, 100, 101, 103, 105, 106, 113], "32": [22, 31, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 67, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 93, 95, 96, 98, 100, 101, 103, 105, 106, 110, 113, 117, 122, 131], "session": [22, 23, 31, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 113, 117, 119, 123, 125, 160, 170, 177], "come": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 113, 117, 118, 143, 168], "neighbor": [22, 28, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 117, 125, 170], "remot": [22, 28, 33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 93, 95, 96, 98, 100, 101, 103, 105, 106, 109, 125, 156, 157, 159, 163, 173, 174, 178], "00": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 161], "51": [22, 90, 113], "2m3": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "feel": [22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 165], "annot": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "previous": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 122, 131, 157, 160], "true": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "complain": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "thei": [22, 33, 35, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 112, 113, 117, 119, 139, 142, 149, 153, 157, 163, 178], "won": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117], "put": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105], "declar": [22, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 157], "suppos": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 126, 153], "prevent": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 128], "reclaimpolici": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "retain": [22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105], "Or": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "leaf": [22, 23, 29, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 110, 117], "tor": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "clean": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117], "doc": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 126, 136, 137, 141, 152, 167], "veri": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "bgpconfigur": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "nodetonodemeshen": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "asnumb": [22, 40, 90, 113], "64512": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 148], "projectcalico": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "org": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "v1": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "logseverityscreen": [22, 40, 90], "06": [22, 40, 90, 161], "7m59": [22, 40, 90], "sandbox9": [22, 102, 103, 105, 106], "srv06": [22, 35, 36, 40, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 95, 96, 100, 101, 105, 106], "110": [22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 98, 100, 101, 105, 106], "66": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 76, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105], "4200070000": [22, 90], "26": [22, 35, 40, 45, 50, 55, 60, 65, 70, 71, 75, 80, 85, 90, 95, 100, 105, 113], "srv07": [22, 35, 36, 40, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 95, 96, 100, 101, 105, 106], "67": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "4200070001": [22, 90], "srv08": [22, 35, 36, 40, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 95, 96, 100, 101, 105, 106], "68": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "4200070002": [22, 90], "notic": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 120, 132, 149], "07": [22, 40, 90], "48": [22, 40, 72, 73, 75, 76, 90, 110], "8m41": [22, 40, 90], "44": [22, 36, 40, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 90, 96, 101, 106], "19": [22, 40, 51, 90, 107, 161], "mean": [22, 25, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 117, 157, 163, 179], "fals": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "final": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 113, 117, 122, 125, 131], "earlier": [22, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105], "ye": [22, 31, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "mfpdt": [22, 40, 90], "revis": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "34577c": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "logo": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "raw": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "githubusercont": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "gh": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "page": [22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 122, 131, 161], "cuddle_clap": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "gif": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "greet": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "v6": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "goo": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "goarch": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "runtim": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "go1": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "num_goroutin": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "num_cpu": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "leverag": [23, 24, 31, 117, 156, 168, 173], "ecmp": [23, 31, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "hash": [23, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "capabl": [23, 25, 31, 122, 126, 131, 153, 156, 173], "spine": [23, 29, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 87, 91, 96, 101, 106, 110, 114, 117], "deliv": [23, 142], "besid": [23, 151, 157, 177], "advertis": [23, 28, 31, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 91, 93, 96, 98, 101, 103, 106, 119, 125, 126, 170], "unicast": 23, "toward": [23, 28, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 117], "pair": [23, 25, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 125], "lifetim": [23, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "identifi": [23, 25, 157, 160, 163, 174], "failur": [23, 114, 159], "rerout": 23, "outag": 23, "instanc": [23, 31, 121, 130, 144, 146, 152, 156, 173, 174], "ellips": 23, "extra": 23, "status": [23, 28, 109, 117, 170, 177], "router": [24, 27, 28, 31, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 92, 93, 96, 97, 98, 101, 102, 103, 106, 109, 119, 125, 170], "layer": [24, 27, 31, 109, 110, 124, 125, 142, 163, 178, 179], "least": [24, 117, 122, 139], "well": [24, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 117, 127, 155, 156, 157, 162, 163, 171, 173, 174], "kubenet": [24, 30], "requestor": 24, "space": [24, 27, 88, 122, 128, 131, 142], "expos": [24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "next": [24, 28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 113, 117, 119, 120, 122, 126, 131, 132, 149, 151, 153, 168, 172, 174], "probe": 24, "uncondition": [24, 25], "millisecond": 24, "known": [25, 28, 31, 114, 142], "bundl": 25, "ethernet": [25, 31, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104, 109], "combin": 25, "parallel": [25, 109, 117], "increas": [25, 26, 31], "throughput": [25, 29], "beyond": [25, 109], "singl": [25, 27, 28, 29, 38, 109, 117, 136, 141, 147, 152, 177, 178, 179], "could": [25, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "sustain": 25, "redund": [25, 29, 125, 142, 151, 166], "lacp": [25, 26, 109], "compon": [25, 148, 179], "collect": [25, 27, 160], "handl": [25, 117, 119, 124, 126, 151, 153, 170], "multihom": [25, 26, 32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102, 109], "particip": [25, 28, 30, 178], "mh": [25, 109, 117], "robust": 25, "concurr": 25, "high": [25, 27, 29, 31, 109, 117, 120, 124, 132, 142, 149, 158, 178], "seamless": 25, "failov": 25, "result": [25, 26, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 107, 117, 157, 174], "enhanc": [25, 142], "resili": [25, 31, 142], "One": [25, 27, 117, 160], "overcom": 25, "asic": [25, 109, 110, 161], "a1": 25, "fallback": 25, "standbi": [25, 109, 178], "without": [25, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 142, 151, 166, 177], "suffici": 25, "howev": [25, 26, 27, 117, 138, 141, 157, 160, 168, 172], "intend": [26, 126, 139, 153, 168, 178], "assist": 26, "smoothli": 26, "redirect": [26, 156, 173], "carri": [26, 28, 117], "impact": [26, 160], "advis": [26, 122, 131], "switchov": 26, "normal": [26, 177], "toggl": [26, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 92, 93, 97, 98, 102, 103, 112], "conveni": 26, "checkbox": [26, 122], "strive": 26, "guarante": 26, "offload": [26, 109], "undertaken": 26, "prefer": [26, 28, 172, 174], "lower": 26, "intern": [26, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 88, 92, 93, 97, 98, 102, 103, 109, 116, 117, 128, 131, 172], "prepend": 26, "tenfold": 26, "outbound": [26, 28, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 93, 96, 98, 101, 103, 106, 122, 126, 156, 173], "med": [26, 28], "decreas": 26, "snat": [26, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 88, 92, 93, 97, 98, 102, 103, 107, 110, 126, 128, 147, 153, 164, 168, 179], "transfer": 26, "second": [26, 33, 35, 37, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 117, 120, 132, 149, 153, 172, 177], "reestablish": 26, "correctli": 26, "id": [26, 27, 32, 33, 36, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 88, 92, 93, 96, 97, 98, 101, 102, 103, 106, 109, 112, 113, 125, 129, 132, 150, 157, 163, 165, 166, 170, 172, 174, 178], "undergo": 26, "modif": [26, 157], "reorgan": 26, "instabl": 26, "flap": 26, "becom": [26, 30, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 151, 166], "revert": 26, "stabil": 26, "preemption": 26, "compos": 27, "element": [27, 113], "statist": [27, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 117, 177], "analyt": 27, "modul": [27, 109, 115, 117], "diagram": [27, 33, 34, 38, 39, 43, 44, 48, 49, 53, 54, 58, 59, 63, 64, 68, 69, 73, 74, 78, 79, 83, 84, 89, 93, 94, 98, 99, 103, 104, 117, 170, 179], "replic": [27, 126, 153], "unreach": [27, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103], "toler": [27, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "stat": [27, 31], "affect": [27, 160], "border": [27, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 119, 142, 170], "translat": [27, 28, 34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104, 139, 142, 156, 164, 173], "firewal": [27, 122, 131, 179], "x86": [27, 110, 120, 136, 142], "dpdk": [27, 30, 109, 120, 132, 149, 158], "enter": [27, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 113, 117, 122, 131, 148, 157], "bypass": 27, "kernel": [27, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 161], "travel": 27, "pcie": 27, "bu": 27, "closest": 27, "last": [27, 117, 172], "header": 27, "rewrit": 27, "mac": [27, 28, 114, 117, 157, 177], "vlan": [27, 34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 112, 117, 125, 151, 157, 163, 164, 165, 166, 172, 174, 178], "100gbp": [27, 30, 109], "frr": [27, 28, 30, 31, 119], "standard": [27, 29, 117, 126, 153, 156, 168, 173], "wireguard": [27, 28, 30, 109, 179], "tunnel": [27, 28, 30], "dynam": [27, 28, 30, 159, 163], "preconfigur": [28, 138, 173, 174, 176, 179], "abil": [28, 30, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 176, 177, 179], "safe": [28, 151, 166], "tabl": [28, 31, 109, 122, 139, 170], "program": 28, "meet": [28, 122, 131], "fine": 28, "expand": [28, 31, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 117, 178], "referenc": 28, "extend": [28, 112, 179], "larg": [28, 126], "consist": [28, 138, 139, 156, 161, 168, 173], "possibl": [28, 126, 157, 163, 168, 178], "mandatori": [28, 161], "len": 28, "ge": 28, "keyword": 28, "aa": 28, "nn": 28, "65535": 28, "associ": [28, 122, 126, 131, 148, 153, 157, 163, 179], "inbound": [28, 31, 36, 38, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106, 119, 122], "sequenc": [28, 122, 131], "drag": [28, 114], "claus": 28, "hop": [28, 122, 131], "tag": [28, 109, 120, 126, 131, 132, 151, 166, 172, 178], "met": 28, "whether": [28, 113, 114, 117], "manipul": 28, "situat": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 156, 173], "dual": [28, 109, 178], "temporari": 28, "interconnect": [28, 30], "scope": [28, 125, 150], "null": 28, "254": [28, 36, 37, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 87, 91, 96, 101, 106, 113, 117, 157, 163], "fornat": 28, "lift": [28, 179], "dnat": [28, 107, 109, 110, 126, 128, 147, 153, 157, 164, 168, 179], "larger": [28, 120, 149], "scale": [28, 109, 110, 117], "carrier": 28, "grade": 28, "overload": 28, "mani": [28, 31, 113, 178], "along": [28, 142, 143, 158], "accept": [28, 141], "silent": 28, "exclus": [28, 172], "masquerad": [28, 128, 147, 164, 174], "predefin": [28, 139, 168], "24subnet": 28, "128a": 28, "130": 28, "8080": 28, "henc": 28, "reach": [28, 33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101, 103, 106, 119, 158, 160], "hub": [28, 122, 131], "transit": 28, "major": [28, 29], "spoke": 28, "small": [28, 120, 136, 149, 152], "offic": [28, 109], "measur": 28, "underneath": 28, "mathemat": 28, "randomli": 28, "converg": [28, 117], "tool": [28, 30, 127, 155, 162, 171], "perspect": 28, "left": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 126, 153, 168, 177], "dropdown": [28, 112, 122, 126, 129, 131, 132, 150, 153, 168, 177], "famili": [28, 32, 33, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 92, 93, 97, 98, 102, 103, 125], "summari": [28, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 177], "adjac": [28, 31, 87], "queri": 28, "lookup": [28, 109], "rib": 28, "tracerout": 28, "conduct": 28, "determin": 28, "ping": [28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 117], "underlai": [28, 109, 117, 124, 178], "switch11": 28, "four": [28, 29, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "brief": 28, "vni": 28, "distinguish": 28, "fall": [29, 139, 168], "pattern": 29, "unmanag": 29, "collaps": 29, "distribut": [29, 30, 161], "27mpp": [30, 107], "12mpp": [30, 107], "built": [30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 109, 161], "andoth": 30, "comprehens": [30, 142, 177], "call": [30, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 177], "sent": [30, 170, 177], "search": [30, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 177], "sort": [30, 177], "column": [30, 177], "filter": [30, 117, 131, 142, 177], "andautomat": 30, "hairpin": 30, "loop": 30, "cabl": [30, 113, 123], "entir": [30, 122, 126, 128, 177, 178], "kvm": [30, 157, 163, 166, 174], "easi": [30, 151, 166], "authent": 30, "improv": 30, "hide": 30, "comfort": [30, 165], "ecosystem": 31, "daemon": [31, 112, 119], "suit": [31, 156, 173], "don": [31, 117, 126, 151, 153, 157, 179], "span": [31, 178, 179], "stp": 31, "miniz": 31, "reliabl": 31, "interim": 31, "proxmox": [31, 109], "Will": 31, "inherit": 31, "zero": [31, 109, 148], "msft": 31, "sometim": 31, "autodetect": [31, 178], "1gbp": [31, 178], "10gpb": 31, "speed": [31, 110, 112, 178], "bulk": [31, 112, 178], "b": [31, 86], "255": [31, 136], "certain": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 112, 117, 122, 131, 156, 173], "explor": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "interact": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 113], "yourself": [32, 35, 37, 40, 42, 45, 47, 50, 52, 55, 57, 60, 62, 65, 67, 70, 72, 75, 77, 80, 82, 85, 87, 90, 92, 95, 97, 100, 102, 105], "visit": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "sandbox1": [32, 33, 35, 36], "examin": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 158], "indic": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 114], "2607": [32, 36, 37, 41, 42, 46, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 92, 96, 97, 101, 102, 106], "f358": [32, 36, 37, 41, 42, 46, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 92, 96, 97, 101, 102, 106], "11": [32, 36, 37, 41, 42, 46, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 92, 96, 97, 101, 102, 106, 107, 109, 117, 122], "ffc1": [32, 36], "swp4": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "sw21": [32, 33, 35, 37, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 92, 93, 95, 97, 98, 100, 102, 103, 105], "properli": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 107, 122], "unchang": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "38": [32, 33, 35, 36, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 117], "161": [32, 33, 35, 36, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 92, 93, 97, 102, 114], "leav": [32, 33, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 92, 93, 97, 98, 102, 103, 129, 132, 150], "choic": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "adrress": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "selecet": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "label": [32, 33, 37, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103], "submit": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103, 116, 125], "thank": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "56": [32, 37, 42, 47, 52, 57, 62, 67, 72, 73, 76, 77, 82, 87, 92, 97, 102], "84": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 78, 81, 82, 87, 92, 97, 102], "byte": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "icmp_seq": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "ttl": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "61": [32, 42, 47, 52, 57, 62, 67, 72, 75, 77, 82, 92, 97, 102, 152], "29": [32, 42, 47, 52, 57, 62, 67, 68, 70, 71, 72, 77, 82, 92, 97, 102, 117, 153, 161], "82": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 86, 92, 97, 102], "79": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "transmit": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "loss": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "4002m": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "rtt": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "min": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "avg": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "max": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 117], "mdev": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "787": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "285": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "572": [32, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "interest": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "asid": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "iri": [32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106], "isp1": [32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106], "sandbox": [32, 33, 34, 35, 37, 38, 39, 40, 42, 43, 44, 45, 47, 48, 49, 50, 52, 53, 54, 55, 57, 58, 59, 60, 62, 63, 64, 65, 67, 68, 69, 70, 72, 73, 74, 75, 77, 78, 79, 80, 82, 83, 84, 85, 87, 88, 89, 90, 92, 93, 94, 95, 97, 98, 99, 100, 102, 103, 104, 105], "fault": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "observ": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 92, 93, 97, 98, 102, 103], "g": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 88, 92, 93, 97, 98, 102, 103, 172], "demo": [32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 91, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106], "216": [32, 33, 36, 37, 38, 41, 42, 43, 46, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106], "201": [32, 33, 36, 137, 141, 167], "30064": [32, 36, 37, 41, 42, 46, 47, 51, 52, 56, 57, 61, 62, 66, 67, 71, 72, 76, 77, 81, 82, 86, 87, 91, 92, 96, 97, 101, 102, 106], "introductori": [32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105], "ping4": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 92, 97, 102], "repli": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "62": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 75, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "wan": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "impos": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102], "granular": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 126, 142, 156, 157, 163, 170, 173, 174], "implement": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 117, 148], "would": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 117, 178], "short": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "exercis": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103], "conjunct": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "intellig": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "automag": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "give": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 117, 126, 157], "30065": [33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101, 103, 106], "eth1": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 117, 172], "proto": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103], "onlink": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103], "fulli": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 117], "reachabl": [33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105], "judg": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "corner": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 122, 131, 153, 168, 177], "mark": [33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 122, 177], "swp5": [33, 35, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 93, 95, 98, 100, 103, 105], "sw12": [33, 35, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 93, 95, 98, 100, 103, 105], "wire": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 109, 110, 117, 159], "55": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "world": [33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 91, 93, 96, 98, 101, 103, 106], "swp16": [33, 35, 36, 38, 40, 41, 43, 45, 46, 48, 50, 51, 53, 55, 56, 58, 60, 61, 63, 65, 66, 68, 70, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 93, 95, 96, 98, 100, 101, 103, 105, 106, 113], "easili": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 125, 156, 173], "1012": [33, 35, 36], "untag": [33, 35, 38, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 93, 95, 98, 100, 103, 105, 109, 117, 178], "uncheck": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "green": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 117, 123, 125, 160], "glean": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "insight": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "face": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 109], "aim": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 172, 179], "23": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 107, 117, 137, 141, 167], "srv01": [33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 91, 93, 96, 98, 101, 103, 106], "srv02": [33, 36, 38, 41, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 91, 93, 96, 98, 101, 103, 106], "webpag": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "repeat": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 114, 120, 122, 131, 132, 149, 172], "had": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "tab": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 122, 131], "bar": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "calcul": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117], "factor": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "websit": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "land": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "trigger": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "around": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 118], "vip": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 177], "name_45": [33, 43, 48, 53, 58, 63, 68, 73, 78], "pop": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103], "opposit": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "al3lb": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "were": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 142], "uc": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103], "longer": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 157, 163], "sync": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 112, 149], "yellow": [33, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 125], "windown": [33, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "relev": [33, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103, 117, 118, 131], "member": [33, 36, 38, 43, 46, 48, 51, 53, 56, 58, 61, 63, 66, 68, 71, 73, 76, 78, 81, 83, 86, 88, 93, 96, 98, 101, 103, 106, 177], "resum": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103], "content": [34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104], "welcom": [34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104, 117, 133, 134, 135], "vxlan": [34, 36, 39, 41, 44, 46, 49, 51, 54, 56, 59, 61, 64, 66, 69, 71, 74, 76, 79, 81, 84, 86, 89, 91, 94, 96, 99, 101, 104, 106, 109, 117, 124, 178], "exterior": [34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 89, 94, 99, 104], "l3lb": [34, 39, 44, 49, 54, 59, 64, 69, 74, 79, 84, 94, 99, 104], "dedic": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 110, 156, 166, 173], "furthermor": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "k3s_url": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "kubeconfig": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "afterward": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 122, 131, 160], "ve": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 157, 163], "6443": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "coredn": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "proxi": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "metric": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "debug": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "diagnos": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "problem": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "dump": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 160], "7cf557d9d7": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "6gfwx": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "34": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 117, 131], "nb2t7": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "43": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "443": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 137, 141, 167], "103": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "37m": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "32486": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "30455": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "3m45": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "29m": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "42": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "190": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "13": [35, 106, 107, 117, 153, 161], "30771": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "30510": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "5m14": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "rememb": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "357009a86331a987811fefc11be1350058da33fc": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "develop": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "decid": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 136, 151], "ic": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "coffe": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "5bdf0a53": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "027d": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "449f": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "8896": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "547e06028c6b": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "7m21": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "15m": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "9m56": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "17m": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "14": [35, 41], "unhealthi": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "didn": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "behavior": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 172], "round": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "robin": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "intermitt": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "went": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "btw": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "2m": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "957240": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "957241": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "16m": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "4230000000": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "37": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "4230000001": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "4230000002": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "957194": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "19m": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "01": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105, 113, 117, 157, 163], "2m7": [35, 45, 50, 55, 60, 65, 70, 75, 80, 85, 95, 100, 105], "question": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 117], "five": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "integrat": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "routabl": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "srv03": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "cluser": [36, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106], "30061": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "30062": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "30063": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "softage1": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106], "sw01": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 87, 96, 101, 106], "1011": 36, "17": [36, 46, 107], "ffc0": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106], "127": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106, 113], "softage2": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106], "___": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 96, 101, 106], "sandbox10": [37, 38, 40, 41], "ffca": [37, 41], "respect": [37, 87, 148, 179], "vrf_netri": 37, "562": 37, "745": 37, "690": 37, "737": 37, "666": 37, "4092m": 37, "680": 37, "065": 37, "210": [37, 38, 41, 66], "1102": [38, 40, 41, 117], "126": [38, 40, 41, 48, 50, 51], "125": [38, 40, 41, 47, 51], "208": [38, 40, 41, 97, 98, 101], "209": [38, 66, 102, 103, 106], "212": [38, 41, 51], "name_50": [38, 83, 93, 98, 103], "kubesprai": [40, 90], "apiserv": [40, 90], "221": 40, "222": 40, "prefixlistinbound": [40, 113], "160": [40, 92, 93, 95, 96, 117], "27": 40, "beer": [40, 90], "1101": 41, "122": [41, 51], "121": [41, 48, 50, 51], "220": 41, "sandbox11": [42, 43, 45, 46], "ffcb": [42, 46], "96": [42, 43, 45, 46, 110, 117], "1112": [43, 45, 46], "97": [43, 96], "104": [43, 46], "109": [45, 98, 100, 101], "1111": 46, "114": [46, 106], "113": [46, 106, 113], "108": 46, "sandbox12": [47, 48, 50, 51], "1251": 47, "1122": [47, 48, 50, 51], "1121": [48, 50, 51], "129": [48, 51, 83], "132": [48, 51, 83, 86], "140": [50, 86], "141": [50, 85], "142": [51, 85], "ffcc": 51, "136": [51, 83, 86], "sandbox13": [52, 53, 55, 56], "ffcd": [52, 56], "144": [52, 53, 55, 56, 88, 90, 91, 110, 122], "213": [52, 53, 56, 63, 65, 66], "1132": [53, 55, 56], "166": [53, 55, 56], "165": [53, 55, 56], "145": 53, "148": [53, 56], "152": [53, 56], "157": 55, "158": 55, "1131": 56, "162": 56, "1b": 56, "1a": 56, "156": 56, "sandbox15": [57, 58, 60, 61, 62, 63, 65, 66], "ffce": [57, 61], "176": [57, 58, 60, 61, 97, 98, 100, 101, 122], "214": [57, 58, 61, 63, 65, 66], "1142": [58, 60, 61], "174": [58, 60, 61, 95], "173": [58, 60, 61, 95], "177": [58, 98], "180": [58, 61, 98, 101], "184": [58, 61, 98, 101], "189": [60, 100], "1141": 61, "170": 61, "169": 61, "1d": 61, "1c": 61, "188": [61, 101], "ffcf": [62, 66], "215": [62, 63, 66], "1152": [63, 65, 66], "193": [63, 103], "196": [63, 66, 103, 106], "200": [63, 66, 103, 106], "205": [65, 82, 83, 86, 105], "206": [65, 87, 88, 91, 105], "1151": 66, "1f": 66, "1e": 66, "204": [66, 77, 78, 81, 106], "sandbox2": [67, 68, 70, 71], "ffc2": [67, 71], "1022": [68, 70, 71], "36": [68, 71, 110, 117, 161], "40": [68, 71], "1021": 71, "25": [71, 113], "sandbox3": [72, 73, 75, 76], "ffc3": [72, 76], "1032": [73, 75, 76], "70": [73, 75, 76], "69": [73, 75, 76], "52": [73, 76], "1031": 76, "60": 76, "sandbox4": [77, 78, 80, 81], "ffc4": [77, 81], "1042": [78, 80, 81], "78": [78, 80, 81], "77": [78, 80, 81], "81": [78, 86], "88": [78, 81], "93": [80, 88, 90, 91], "94": [80, 88, 90, 91], "1041": 81, "74": 81, "73": 81, "92": 81, "sandbox5": [82, 83, 85, 86], "ffc5": [82, 86], "1052": [83, 85, 86], "86": [83, 85, 86], "85": [83, 85, 86, 131], "1051": 86, "sandbox6": [87, 88, 90, 91], "radio": 87, "row": 87, "543": 87, "436": 87, "445": 87, "354": 87, "345": 87, "4074m": 87, "424": 87, "075": 87, "insid": [87, 119, 151, 178], "screen": [87, 122, 131], "pubblic": 87, "disallow": 87, "nfv": 88, "unselect": 88, "1062": [88, 90, 91], "150": [88, 131], "durat": 88, "v0": 90, "154": [90, 152], "155": 90, "1061": 91, "90": 91, "89": [91, 136], "sandbox7": [92, 93, 95, 96], "ffc7": [92, 96], "207": [92, 93, 96], "1072": [93, 95, 96], "102": [93, 95, 96], "101": [93, 95, 96], "164": [93, 96], "1071": 96, "98": 96, "sandbox8": [97, 98, 100, 101], "ffc8": [97, 101], "1082": [98, 100, 101], "1081": 101, "105": 101, "ffc9": [102, 106], "1091": 106, "help": [107, 117], "mpp": [107, 109], "unmatch": 107, "e5": 107, "2660": 107, "fw": 107, "schedul": 107, "kni": 107, "17mpp": 107, "7mpp": 107, "9mpp": 107, "2mpp": 107, "gold": [107, 109], "6130": 107, "odd": 107, "31mpp": 107, "22mpp": 107, "15mpp": 107, "11mpp": 107, "3mpp": 107, "3m": 107, "conntrack": 107, "20k": 107, "38mpp": 107, "28mpp": 107, "19mpp": 107, "eo": [109, 110], "day0": 109, "day1": 109, "day2": 109, "nov": 109, "2024": [109, 110, 117], "east": [109, 117], "west": [109, 117], "north": [109, 117], "south": [109, 117], "tenanc": [109, 126, 153, 168], "ing": 109, "laser": 109, "signal": [109, 120, 126, 132, 149, 153, 177], "sensor": [109, 177], "temperatur": [109, 177], "fan": [109, 117, 177], "suppli": [109, 177], "dec": [109, 110], "unnumb": [109, 124, 178], "chain": 109, "l3vpn": [109, 117], "l2vpn": [109, 117], "templat": [109, 115], "tbd": 109, "pat": 109, "region": [109, 131, 136, 141, 148, 152, 179], "awar": [109, 178], "unawar": 109, "comput": [109, 117, 157, 163, 174], "esi": 109, "mc": 109, "rail": [109, 117], "best": 109, "practic": 109, "qo": 109, "roce": 109, "workload": [109, 117, 151, 166], "adapt": 109, "congest": 109, "dpu": [109, 117], "touch": 109, "supern": 109, "central": 109, "protect": 109, "unwant": 109, "audit": 109, "aspect": 109, "intuit": 109, "iac": 109, "vmware": [109, 157, 163, 166, 174], "apach": 109, "stack": 109, "openstack": 109, "harvest": 109, "hbn": 109, "vtep": 109, "acceler": [109, 120, 132, 149], "2025": 109, "flavor": [109, 132, 136, 152], "edg": [109, 125], "dot1q": [109, 122, 131], "amd": [109, 110], "64gb": [109, 117], "300gb": 109, "6336y": 109, "48c": 109, "3ghz": 109, "11gbp": 109, "8mpp": 109, "nsx": 109, "100gbe": [109, 110], "128gb": 109, "platinum": 109, "25mpp": 109, "hyperscal": 109, "256gb": 109, "8352y": 109, "64c": 109, "2ghz": 109, "22gbp": 109, "q2": 109, "202211": 109, "331": 109, "os": 110, "ssd": 110, "nvme": 110, "2x": 110, "1gbe": 110, "600": 110, "tb": 110, "10gbe": 110, "su": [110, 117], "256": 110, "stateless": 110, "necessit": 110, "prod": 110, "25gbe": 110, "1x": 110, "manufactur": 110, "caveat": 110, "sn2010": 110, "sfp28": 110, "qsfp28": 110, "sn2100": 110, "sn2201": 110, "rj45": 110, "sn2410": 110, "sn2700": 110, "sn3420": 110, "sn3700c": 110, "sn3700": 110, "qsfp56": 110, "200gbe": 110, "sn4410": 110, "100g": [110, 175], "qsfp": 110, "400gbe": 110, "sn4600c": 110, "sn4600": 110, "sn4700": 110, "sn5400": 110, "sn5600": 110, "osfp": 110, "800gbe": 110, "powerswitch": 110, "seri": 110, "s5212f": 110, "ON": 110, "broadcom": [110, 161], "trident": 110, "iii": 110, "25g": 110, "s5224f": 110, "s5232f": 110, "s5248f": 110, "s5296f": 110, "s5448f": 110, "iv": 110, "z": 110, "z9664f": 110, "tomahawk": 110, "z9432f": 110, "dcs201": 110, "as5835": 110, "54x": 110, "10g": [110, 175], "sfp": 110, "dcs202": 110, "54t": 110, "rj": 110, "dcs203": 110, "as7326": [110, 161], "56x": [110, 161], "as7726": 110, "32x": 110, "dcs510": 110, "as9716": 110, "32d": 110, "400g": 110, "dcs511": 110, "as9737": 110, "32db": 110, "ais800": 110, "64o": 110, "osfp800": 110, "7020r": 110, "qumran": 110, "qsfp100": 110, "1000mb": 110, "7050x3": 110, "sfp25": 110, "7050x4": 110, "2sfp": 110, "dsfp": 110, "qsfpdd": 110, "200g": 110, "7060x4": 110, "800g": 110, "7060x5": 110, "7280r3a": 110, "jericho2": 110, "7280r3": 110, "7358x4": 110, "7368x4": 110, "7300r3": 110, "40gbe": 110, "7500r3": 110, "jericho": 110, "288": 110, "whom": 112, "breakout": 112, "split": [112, 147], "switchd": 112, "mtu": 112, "maximum": 112, "transmiss": [112, 117], "autoneg": 112, "autonegoti": 112, "duplex": 112, "extens": [112, 113], "quick": 112, "detach": 112, "hand": [113, 139], "tutori": [113, 143, 161, 179], "hashicorp": 113, "tf": 113, "At": [113, 117, 120, 125, 126, 132, 149], "begin": [113, 125], "required_provid": 113, "netris_address": 113, "netris_login": 113, "netris_password": 113, "init": [113, 115, 117], "folder": [113, 161], "properti": 113, "Such": 113, "netris_sit": 113, "clear": 113, "tenantid": 113, "netris_ten": 113, "netris_alloc": 113, "198": 113, "netris_subnet": 113, "defaultgatewai": 113, "siteid": 113, "depends_on": 113, "With": 113, "netris_softg": 113, "mainip": 113, "mgmtip": 113, "netris_switch": 113, "cumulus_linux": 113, "portcount": 113, "netris_link": 113, "sg": [113, 160], "sw": [113, 160], "swp1": 113, "netris_vnet": 113, "1050": 113, "swp3": 113, "10th": 113, "netris_port": 113, "swp10_my_switch": 113, "swp10": 113, "netris_bgp": 113, "23456": 113, "portid": 113, "confirm": 113, "progress": 113, "destroi": [113, 117, 118], "resid": [114, 139], "preliminari": 114, "regist": [114, 122, 131], "synchron": [114, 117, 151, 172, 177], "administ": [114, 126, 153, 157, 168], "watch": [114, 159], "against": [114, 159], "long": [114, 159], "nhn": 115, "plugin": 115, "cleanup": 115, "r": 116, "partner": 116, "educ": 116, "subject": 116, "trial": [116, 118], "www": [116, 161], "blank": 117, "tfvar": [117, 118], "block": [117, 122, 134, 145, 157], "blueprint": [117, 118], "guidelin": 117, "increment": 117, "els": 117, "product": [117, 151, 156, 160, 166, 170, 173], "pxe": 117, "conveninec": 117, "teast": 117, "thumb": 117, "4th": 117, "equal": 117, "although": [117, 126, 153], "tofu": [117, 118], "insert": 117, "cloudsim": [117, 118], "pulumi": [117, 118], "icon": [117, 177], "donut": 117, "psu": 117, "hw": 117, "tennant": 117, "middl": 117, "zoom": 117, "eth": 117, "dataset": 117, "aliv": 117, "care": 117, "healthcheck": 117, "background": 117, "quantiti": 117, "rare": [117, 178], "debian": [117, 161], "gnu": 117, "pod00": 117, "su0": 117, "r3": 117, "smp": 117, "preempt_dynam": 117, "cl5": 117, "1u6": 117, "05": [117, 161], "x86_64": [117, 161], "thu": [117, 151, 156, 157, 161, 166, 173], "sep": [117, 161], "haven": 117, "119": 117, "landscap": 117, "canon": 117, "fri": 117, "utc": [117, 161], "95": 117, "ens4": [117, 131], "39": [117, 161], "esm": 117, "hgx": 117, "h00": 117, "know": [117, 119, 166], "scheme": 117, "respond": 117, "rail0": 117, "rail1": 117, "rail2": 117, "rail3": 117, "rail4": 117, "rail5": 117, "rail6": 117, "rail7": 117, "224": 117, "gw": 117, "ipmi": 117, "eth11": 117, "sensit": 117, "therefor": [117, 121, 122, 130, 131, 151, 156, 166, 173], "shall": 117, "lldp": 117, "tlv": 117, "pluggin": 117, "hnp": 117, "systemd": 117, "preset": 117, "sinc": [117, 157, 163], "09": 117, "21h": 117, "ago": 117, "pid": 117, "2906": 117, "task": [117, 139, 156, 173], "7m": 117, "3min": 117, "913": 117, "slice": 117, "align": [117, 158, 168], "ask": 117, "confogur": 117, "deal": [117, 129, 132, 150], "area": 117, "tell": 117, "postfix": 117, "servern": 117, "eth2": [117, 172], "eth3": 117, "eth4": 117, "eth5": 117, "eth6": 117, "eth7": 117, "eth8": 117, "eth9": 117, "eth10": 117, "ipv4gatewai": 117, "relat": 117, "datacent": 117, "primit": 117, "driven": 117, "ahead": 117, "doubl": 117, "book": 117, "avoid": 117, "conflict": 117, "conclud": 117, "colleagu": [117, 126, 153, 157, 168], "recycl": 117, "anywai": 117, "feedback": 117, "tip": 118, "trick": 118, "netrisprovidedpassword": 118, "written": 118, "hcl": 118, "sa": 118, "solut": [118, 142, 151, 166, 172], "architect": 118, "subdirectori": 118, "chapter": [119, 126, 157, 163], "pull": 119, "crit": [120, 132, 149], "heart": [120, 132, 149], "beat": [120, 132, 149], "premis": [121, 130], "commenc": [122, 131], "due": [122, 131, 151, 163], "numer": [122, 131], "equip": [122, 131], "t2": 122, "medium": [122, 131], "t3": 122, "satisfi": [122, 131, 168], "drive": [122, 131], "crucial": [122, 131], "upcom": [122, 131], "involv": [122, 131], "input": [122, 131], "65500": [122, 131], "asn": [122, 125, 131, 148], "subsequ": [122, 131], "netmask": [122, 131], "choos": [122, 125, 131, 141, 148, 149, 152, 159], "cidr": 122, "31": [122, 149, 152], "int": [122, 131, 172], "modifi": [122, 172], "target": [122, 131], "unrout": [124, 178, 179], "secondari": 125, "coupl": 125, "unus": [125, 153], "1000": 125, "vc": 125, "bill": 125, "token": 125, "onto": 125, "shoudl": 125, "crush": 126, "smaller": [126, 153], "worri": [126, 151, 153], "word": [126, 153], "repurpos": [126, 153], "e2": 131, "167": 131, "manner": 131, "regardless": 131, "googl": 131, "accur": 131, "smallest": [132, 136, 152], "video": 134, "walk": [134, 154], "enough": [136, 152, 160], "139": 136, "178": 136, "idea": [136, 152, 167], "netrisctl": [136, 137, 141, 152, 167], "issuer": [136, 137, 141, 152, 167], "hint": [136, 137, 141, 152, 167], "183": [137, 141, 167], "2003": [137, 141, 167], "stream": [137, 141, 167], "telemetri": [137, 141, 167], "collectd": [137, 141, 167], "3033": [137, 141, 167], "50051": [137, 141, 167], "effect": 139, "hierarch": 139, "facilit": [139, 173], "rfc": [139, 168], "1918": [139, 168], "encompass": [139, 168], "reason": 141, "technologi": [142, 158], "capac": 142, "autonom": 148, "65534": 148, "trust": 148, "s2": 149, "c1": 149, "jammi": 149, "client": 150, "introduc": [151, 160, 166], "disrupt": [151, 166], "4094": 151, "802": [151, 166], "1q": [151, 166], "s0": 152, "d1": 152, "131": 152, "153": 152, "usabl": 153, "broadcast": 153, "offici": 154, "meanwhil": [155, 162, 171], "goal": [156, 173], "ideal": [156, 173], "unlik": [156, 173], "often": [156, 173], "55022": [156, 173], "launch": [157, 163, 174], "live": [157, 163], "significantli": 157, "simpler": 157, "compar": 157, "cannot": 157, "perman": 157, "jumphost": 157, "proxycommand": 157, "exclud": [157, 163], "build": [157, 161, 163], "sai": [157, 163], "dialogu": [157, 163], "convert": [157, 163], "made": 157, "advantag": [158, 168], "anyth": 160, "hypothet": 160, "goe": 160, "wrong": 160, "check_ag": 160, "svclb": 160, "haproxi": 160, "6tkgj": 160, "38d": 160, "bcb944b7c": 160, "qcbf8": 160, "13d": 160, "squid": 160, "7f6fdc6cf9": 160, "7fdx8": 160, "58rnp": 160, "graphit": 160, "mongodb": 160, "redi": 160, "smtp": 160, "76778cf85f": 160, "lw5v5": 160, "10d": 160, "8b9dbbcd8": 160, "8snhd": 160, "notifi": 160, "647975848f": 160, "fs5dn": 160, "b9b8d8f8d": 160, "4ssqb": 160, "987669fb9": 160, "jjskp": 160, "777c98c5d9": 160, "mqwl6": 160, "lqmq7": 160, "20h": 160, "sshing": 160, "event": 160, "advers": 160, "031": 160, "sonic_20220929_052156_ec202012_420": 161, "wipe": 161, "author": [161, 168], "ident": 161, "prompt": 161, "mismatch": 161, "switch15": 161, "commit": 161, "895d178f6": 161, "2022": 161, "225": 161, "accton_as7326_56x": 161, "r0": 161, "hwsku": 161, "accton": 161, "serial": 161, "redact": 161, "uptim": 161, "averag": 161, "08": 161, "aren": 163, "dmz": 164, "700": [165, 172, 174], "900": [165, 172], "imper": 168, "reconfigur": 168, "worth": 168, "illustr": 168, "meant": 168, "uncom": 169, "1m": 170, "alter": 170, "trobleshoot": 170, "propos": 172, "interface_nam": 172, "mission": 172, "1g": 175, "sum": 177, "visual": 177, "titl": 177, "bp": 177, "pp": 177, "optic": 177, "histori": 177, "uplink": 177, "pc06": 177, "explan": 177, "summar": 177, "pie": 177, "alarm": 177, "collabor": 178, "guest": 178, "circuit": 178, "backbon": 178, "alia": 178, "swp": 178, "signific": 178, "egress": 178, "assum": 178, "svi": 178, "nose": 178, "10gbp": 178, "entiti": 179, "hierarchi": 179, "spread": 179, "zone": 179, "multisit": 179}, "objects": {}, "objtypes": {}, "objnames": {}, "titleterms": {"netri": [0, 6, 7, 11, 14, 16, 18, 19, 22, 27, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 108, 109, 110, 113, 115, 116, 117, 118, 120, 122, 123, 124, 131, 132, 133, 134, 136, 137, 141, 142, 143, 149, 151, 152, 154, 160, 167, 179], "local": 0, "repositori": 0, "setup": [0, 1, 2, 3, 4, 5, 8, 111, 134, 138, 139, 144, 145, 146, 147, 148, 153, 159, 164, 168, 176], "when": 0, "why": 0, "us": [0, 22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106, 134, 145, 147, 155, 156, 157, 162, 163, 164, 170, 171, 172, 173, 174], "how": 0, "enabl": [0, 24, 28, 122, 126, 127, 128, 129, 131, 132, 144, 146, 150, 153, 168], "control": [0, 10, 11, 14, 17, 18, 22, 27, 32, 33, 35, 36, 37, 38, 40, 41, 42, 43, 45, 46, 47, 48, 50, 51, 52, 53, 55, 56, 57, 58, 60, 61, 62, 63, 65, 66, 67, 68, 70, 71, 72, 73, 75, 76, 77, 78, 80, 81, 82, 83, 85, 86, 87, 88, 90, 92, 93, 95, 96, 97, 98, 100, 101, 102, 103, 105, 106, 110, 117, 118, 122, 131, 136, 137, 141, 152, 160, 167], "consum": [0, 24], "upgrad": [0, 11, 160, 161], "repo": [0, 12], "cach": 0, "dell": [1, 110], "sonic": [1, 2, 161], "switch": [1, 2, 3, 4, 5, 8, 16, 26, 27, 108, 109, 110, 111, 112, 114, 117, 143, 160, 175], "initi": [1, 2, 3, 4, 5, 8, 111, 117], "edgecor": [2, 110], "nvidia": [3, 4, 5, 110, 117], "cumulu": [3, 4, 5], "v3": 3, "7": 3, "v5": [4, 5], "9": 5, "softgat": [6, 7, 26, 27, 107, 108, 109, 110, 114, 120, 122, 131, 132, 149, 158, 160, 169, 172], "pro": [6, 158], "instal": [6, 7, 11, 12, 13, 14, 16, 17, 18, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 136, 137, 141, 143, 152, 154, 167, 169], "minimum": [6, 7, 169], "hardwar": [6, 7, 110, 169], "requir": [6, 7, 11, 12, 14, 110, 141, 167, 169], "bio": 6, "configur": [6, 12, 32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 113, 122, 131, 170], "agent": [6, 27, 143, 160], "provis": [7, 120, 122, 131, 149, 158, 169], "softwar": [7, 158, 160, 169], "ubuntu": 8, "switchdev": 8, "account": 9, "user": 9, "tenant": 9, "permiss": 9, "group": 9, "role": 9, "access": [10, 14, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 179], "list": [10, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 179], "acl": [10, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "default": [10, 165], "polici": [10, 16, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "rule": [10, 28], "approv": 10, "workflow": 10, "process": [10, 13], "order": 10, "gener": [11, 16, 133], "linux": [11, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "host": [11, 31, 117], "specif": [11, 109], "name": 11, "let": 11, "": [11, 26, 131], "encrypt": 11, "ssl": [11, 14], "custom": [11, 22, 25, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "issuer": 11, "uninstal": [11, 12], "backup": [11, 160], "restor": [11, 160], "secret": 11, "kei": 11, "helm": [12, 22], "chart": [12, 22], "get": [12, 134, 135, 145, 147, 164, 175], "info": 12, "quickstart": 13, "virtual": [14, 157, 163, 174], "machin": 14, "kvm": 14, "hypervisor": [14, 109], "vm": [14, 131], "replac": 14, "certif": 14, "definit": 15, "welcom": [16, 36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "document": 16, "inform": 16, "try": [16, 115], "learn": [16, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 88, 90, 93, 95, 98, 100, 103, 105, 115], "tutori": [16, 133], "cloud": [16, 109], "nativ": 16, "tool": 16, "network": [16, 29, 32, 33, 36, 37, 38, 41, 42, 43, 46, 47, 48, 51, 52, 53, 56, 57, 58, 61, 62, 63, 66, 67, 68, 71, 72, 73, 76, 77, 78, 81, 82, 83, 86, 87, 88, 91, 92, 93, 96, 97, 98, 101, 102, 103, 106, 108, 109, 111, 117, 157, 163, 168, 170, 174], "servic": [16, 24, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 117, 126, 127, 128, 153, 155, 156, 157, 162, 163, 168, 171, 173, 174, 179], "oper": [16, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 161], "fabric": [16, 109, 123, 124, 125, 142, 175], "manag": [16, 21, 28, 108, 109, 114, 123, 124, 142, 175], "detail": 16, "miscellan": 16, "introduct": [19, 22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 121, 130, 142, 151, 166], "inventori": [20, 114, 138], "profil": [20, 114], "field": [20, 21, 28, 114, 178], "ip": [21, 28, 126, 157, 170], "address": [21, 28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 126, 168, 170], "ipam": [21, 28, 139, 153, 168, 179], "alloc": [21, 28, 153, 168], "subnet": [21, 28, 122, 131, 153, 157], "add": [21, 28, 114, 122, 131], "an": [21, 23, 28, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113, 122, 123, 125, 157, 168], "kubernet": [22, 35, 36, 40, 41, 45, 46, 50, 51, 55, 56, 60, 61, 65, 66, 70, 71, 75, 76, 80, 81, 85, 86, 90, 91, 95, 96, 100, 101, 105, 106], "integr": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 129, 132, 150], "method": 22, "regular": 22, "manifest": 22, "type": 22, "loadbalanc": 22, "resourc": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 113], "l4lb": [22, 24, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "v": [22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 126, 157, 163, 172, 174, 178, 179], "net": [22, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 126, 157, 163, 172, 174, 178, 179], "bgp": [22, 28, 32, 33, 35, 37, 38, 40, 42, 43, 45, 47, 48, 50, 52, 53, 55, 57, 58, 60, 62, 63, 65, 67, 68, 70, 72, 73, 75, 77, 78, 80, 82, 83, 85, 87, 88, 90, 92, 93, 95, 97, 98, 100, 102, 103, 105, 119, 170], "import": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "exist": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 157], "from": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "reclaim": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "calico": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "cni": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "disabl": [22, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "l3": [23, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "load": [23, 24, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 126, 127, 153, 155, 162, 168, 171, 179], "balanc": [23, 24, 33, 35, 38, 40, 43, 45, 48, 50, 53, 55, 58, 60, 63, 65, 68, 70, 73, 75, 78, 80, 83, 85, 90, 93, 95, 98, 100, 103, 105, 126, 127, 153, 155, 162, 168, 171, 179], "anycast": [23, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "lb": 23, "creat": [23, 33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 88, 93, 98, 103, 113, 122, 125, 131, 157, 168, 172, 174], "l4": [24, 155, 162, 168, 171], "link": [25, 114, 172], "aggreg": 25, "lag": 25, "automat": 25, "evpn": 25, "multi": [25, 172], "home": 25, "mainten": 26, "mode": 26, "overview": [26, 121, 130, 142, 151, 166, 170], "what": 26, "happen": 26, "behind": 26, "scene": 26, "architectur": [27, 108, 117], "vpc": [28, 122, 131, 134, 135, 145, 147, 151, 164, 166, 170, 175, 176, 179], "ad": [28, 31, 114, 122, 132, 157], "new": [28, 126, 148, 157], "tree": 28, "view": [28, 114], "basic": 28, "peer": 28, "advanc": 28, "object": 28, "ipv4": 28, "prefix": 28, "ipv6": 28, "commun": 28, "rout": [28, 31, 109, 122, 131], "map": 28, "static": [28, 122, 131, 157], "nat": [28, 32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 126, 128, 153, 156, 168, 173, 179], "defin": 28, "sitemesh": [28, 179], "look": 28, "glass": 28, "refer": [29, 108], "design": 29, "releas": 30, "note": [30, 153], "roh": 31, "provid": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 113], "exampl": [32, 37, 42, 47, 52, 57, 62, 67, 72, 77, 82, 87, 92, 97, 102, 134], "ethernet": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "vlan": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 170], "vxlan": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "e": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "exterior": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "border": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "gatewai": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "protocol": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103], "translat": [32, 33, 37, 38, 42, 43, 47, 48, 52, 53, 57, 58, 62, 63, 67, 68, 72, 73, 77, 78, 82, 83, 87, 88, 92, 93, 97, 98, 102, 103, 168], "l3lb": [33, 38, 43, 48, 53, 58, 63, 68, 73, 78, 83, 93, 98, 103], "sandbox1": 34, "intro": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "deploi": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 122, 131, 157], "applic": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "On": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "demand": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 126, 136, 153, 155, 162, 168, 171], "mileston": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105], "1": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 126, 153, 156], "2": [35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 105, 126, 153, 156], "sandbox": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "topologi": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 114, 117, 159], "diagram": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106], "server": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 117, 124, 136, 152, 157], "cluster": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 117], "upstream": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 170], "isp": [36, 41, 46, 51, 56, 61, 66, 71, 76, 81, 86, 91, 96, 101, 106, 123], "sandbox10": 39, "sandbox11": 44, "sandbox12": 49, "sandbox13": 54, "sandbox14": 59, "sandbox15": 64, "sandbox2": 69, "sandbox3": 74, "sandbox4": 79, "sandbox5": 84, "sandbox6": 89, "gui": 91, "sandbox7": 94, "sandbox8": 99, "sandbox9": 104, "perform": 107, "unmanag": [108, 157, 163], "ha": 108, "scalabl": [108, 110], "data": [108, 109], "center": 108, "support": [109, 110], "function": 109, "platform": [109, 116], "matrix": 109, "extern": [109, 179], "construct": 109, "overlai": 109, "featur": [109, 140], "ai": 109, "secur": [109, 136, 137, 141, 152, 167], "administr": 109, "interfac": [109, 172], "worker": 109, "node": [109, 120, 122, 131, 132, 149, 160], "plane": 109, "variat": 109, "NOS": 109, "version": [109, 110, 160], "compat": 109, "h": 110, "The": [110, 160], "horizont": 110, "arista": 110, "port": 112, "terraform": 113, "To": 113, "your": [113, 160], "first": 113, "directori": 113, "file": 113, "prepar": 113, "infrastructur": [113, 116, 118], "plan": 113, "delet": 113, "simul": [116, 117, 118], "lab": 117, "scenario": 117, "gpu": 117, "spectrum": 117, "x": 117, "start": [117, 118, 134, 135, 145, 147, 164, 175], "monitor": 117, "dashboard": [117, 177], "ssh": [117, 118], "nhn": 117, "plugin": 117, "templat": 117, "check": [117, 122, 160, 165], "connect": [117, 123, 124, 170, 179], "cleanup": 117, "test": 118, "web": 118, "consol": 118, "init": 118, "modul": 118, "stop": [118, 160], "activ": 119, "equinix": [119, 120, 125, 129, 132, 134, 136, 145, 163], "metal": [119, 120, 129, 132, 134, 136, 145, 163], "project": [119, 120, 134, 163], "site": [121, 122, 130, 131, 144, 146, 148, 165, 179], "mesh": [121, 122, 130, 131, 144, 146], "aw": [121, 122, 144], "concept": [121, 130, 142, 151, 166], "ec2": 122, "instanc": [122, 131], "pre": [122, 131], "requisit": [122, 131], "step": [122, 131, 134, 136, 141, 152, 167], "sourc": 122, "destin": 122, "interconnect": 125, "pool": 126, "request": 126, "public": [126, 170], "block": 126, "elast": [126, 153, 155, 162, 168, 171], "3": 126, "4": 126, "api": [129, 132, 150, 177], "gcp": [130, 131, 146], "guid": [134, 135, 164], "anywher": [135, 164, 166], "matter": [136, 137, 141, 152, 167], "more": 140, "phoenixnap": [147, 149, 150, 151, 152, 154, 155, 156, 157], "bmc": [149, 150, 151, 152, 154, 155, 156, 157], "option": [153, 156, 170], "separ": 153, "each": 153, "purpos": 153, "split": 153, "singl": 153, "differ": 153, "masquerad": [156, 173], "snat": [156, 173], "dnat": [156, 173], "isol": [157, 163, 174], "w": 157, "dynam": 157, "tag": [157, 163], "100g": 158, "1g": 158, "10g": 158, "rollback": 160, "procedur": [160, 161], "current": 160, "databas": 160, "downgrad": 160, "system": 161, "set": 165, "one": 170, "two": 170, "dmz": 170, "visibl": 177, "telescop": 177, "graph": 177, "board": 177, "log": 177, "work": 179}, "envversion": {"sphinx.domains.c": 2, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 8, "sphinx.domains.index": 1, "sphinx.domains.javascript": 2, "sphinx.domains.math": 2, "sphinx.domains.python": 3, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.todo": 2, "sphinx.ext.viewcode": 1, "sphinx": 57}, "alltitles": {"Netris Local Repository Setup": [[0, "netris-local-repository-setup"]], "When & Why to Use the Local Repository?": [[0, "when-why-to-use-the-local-repository"]], "How to Enable the Local Repository on the Netris Controller?": [[0, "how-to-enable-the-local-repository-on-the-netris-controller"]], "How to consume local repository": [[0, "how-to-consume-local-repository"]], "Upgrade Local-Repo Cache": [[0, "upgrade-local-repo-cache"]], "Dell SONiC Switch Initial Setup": [[1, "dell-sonic-switch-initial-setup"]], "EdgeCore SONiC Switch Initial Setup": [[2, "edgecore-sonic-switch-initial-setup"]], "Nvidia Cumulus v3.7 Switch Initial Setup": [[3, "nvidia-cumulus-v3-7-switch-initial-setup"]], "Nvidia Cumulus v5 Switch Initial Setup": [[4, "nvidia-cumulus-v5-switch-initial-setup"]], "Nvidia Cumulus v5.9+ Switch Initial Setup": [[5, "nvidia-cumulus-v5-9-switch-initial-setup"]], "SoftGate PRO Installation": [[6, "softgate-pro-installation"]], "Minimum Hardware Requirements": [[6, "minimum-hardware-requirements"], [7, "minimum-hardware-requirements"], [169, "minimum-hardware-requirements"]], "BIOS Configuration": [[6, "bios-configuration"]], "Install the Netris Agent": [[6, "install-the-netris-agent"]], "SoftGate Installation": [[7, "softgate-installation"], [169, "softgate-installation"]], "Provision Netris SoftGate software": [[7, "provision-netris-softgate-software"]], "Ubuntu SwitchDev Switch Initial Setup": [[8, "ubuntu-switchdev-switch-initial-setup"]], "Accounts": [[9, "accounts"]], "Users": [[9, "users"]], "Tenants": [[9, "tenants"]], "Permission Groups": [[9, "permission-groups"]], "User Roles": [[9, "user-roles"]], "Access Control Lists (ACL)": [[10, "access-control-lists-acl"]], "ACL Default Policy": [[10, "acl-default-policy"]], "ACL Rules": [[10, "acl-rules"]], "ACL Approval Workflow": [[10, "acl-approval-workflow"]], "ACL Processing Order": [[10, "acl-processing-order"]], "Netris Controller installation on a generic Linux host": [[11, "netris-controller-installation-on-a-generic-linux-host"]], "Linux Host requirements": [[11, "linux-host-requirements"]], "Installation": [[11, "installation"]], "Installation with the specific host name": [[11, "installation-with-the-specific-host-name"]], "Installation with the Let\u2019s Encrypt SSL": [[11, "installation-with-the-let-s-encrypt-ssl"]], "Installation with the Custom SSL Issuer": [[11, "installation-with-the-custom-ssl-issuer"]], "Upgrading": [[11, "upgrading"]], "Uninstalling": [[11, "uninstalling"]], "Backup and Restore": [[11, "backup-and-restore"]], "Backup": [[11, "backup"]], "Backup the Secret Key": [[11, "backup-the-secret-key"]], "Restore": [[11, "restore"]], "Restore the Secret Key": [[11, "restore-the-secret-key"]], "Helm Chart Installation": [[12, "helm-chart-installation"]], "Requirements": [[12, "requirements"], [14, "requirements"]], "Get Repo Info": [[12, "get-repo-info"]], "Installing the Chart": [[12, "installing-the-chart"]], "Uninstalling the Chart": [[12, "uninstalling-the-chart"]], "Chart Configuration": [[12, "chart-configuration"]], "Quickstart Installation": [[13, "quickstart-installation"]], "Quickstart Process": [[13, "quickstart-process"]], "Virtual Machine Installation": [[14, "virtual-machine-installation"]], "KVM Hypervisor Installation": [[14, "kvm-hypervisor-installation"]], "VM Controller Installation": [[14, "vm-controller-installation"]], "Accessing the Netris Controller": [[14, "accessing-the-netris-controller"]], "Replacing the SSL certificate": [[14, "replacing-the-ssl-certificate"]], "Definitions": [[15, "definitions"]], "Welcome to Netris Documentation": [[16, "welcome-to-netris-documentation"]], "General Information": [[16, null]], "Try & Learn Netris": [[16, null]], "Tutorials": [[16, null]], "Cloud Native Tools": [[16, null]], "Network Services": [[16, null]], "Network Policies": [[16, null]], "Operations": [[16, null]], "Switch-fabric Management": [[16, null]], "Detailed Installation": [[16, null]], "Miscellaneous": [[16, null]], "Controller Installation": [[17, "controller-installation"], [17, null]], "Installing a Netris Controller": [[18, "installing-a-netris-controller"], [137, "installing-a-netris-controller"]], "Introduction to Netris": [[19, "introduction-to-netris"]], "Inventory Profiles": [[20, "inventory-profiles"], [114, "inventory-profiles"]], "Inventory Profile Fields": [[20, "id1"], [114, "id1"]], "IP Address Management (IPAM)": [[21, "ip-address-management-ipam"]], "Allocations and Subnets": [[21, "allocations-and-subnets"], [28, "allocations-and-subnets"]], "Add an Allocation": [[21, "add-an-allocation"], [28, "add-an-allocation"]], "Allocation Fields": [[21, "id1"], [28, "id1"]], "Add a Subnet": [[21, "add-a-subnet"], [28, "add-a-subnet"]], "Subnet fields": [[21, "id2"], [28, "id2"]], "Kubernetes Integration": [[22, "kubernetes-integration"]], "Install Netris Operator": [[22, "install-netris-operator"], [35, "install-netris-operator"], [40, "install-netris-operator"], [45, "install-netris-operator"], [50, "install-netris-operator"], [55, "install-netris-operator"], [60, "install-netris-operator"], [65, "install-netris-operator"], [70, "install-netris-operator"], [75, "install-netris-operator"], [80, "install-netris-operator"], [85, "install-netris-operator"], [90, "install-netris-operator"], [95, "install-netris-operator"], [100, "install-netris-operator"], [105, "install-netris-operator"]], "Helm Chart Method": [[22, "helm-chart-method"]], "Regular Manifest Method": [[22, "regular-manifest-method"]], "Using Type \u2018LoadBalancer\u2019": [[22, "using-type-loadbalancer"]], "Using Netris Custom Resources": [[22, "using-netris-custom-resources"], [35, "using-netris-custom-resources"], [40, "using-netris-custom-resources"], [45, "using-netris-custom-resources"], [50, "using-netris-custom-resources"], [55, "using-netris-custom-resources"], [60, "using-netris-custom-resources"], [65, "using-netris-custom-resources"], [70, "using-netris-custom-resources"], [75, "using-netris-custom-resources"], [80, "using-netris-custom-resources"], [85, "using-netris-custom-resources"], [90, "using-netris-custom-resources"], [95, "using-netris-custom-resources"], [100, "using-netris-custom-resources"], [105, "using-netris-custom-resources"]], "Introduction to Netris Custom Resources": [[22, "introduction-to-netris-custom-resources"], [35, "introduction-to-netris-custom-resources"], [40, "introduction-to-netris-custom-resources"], [45, "introduction-to-netris-custom-resources"], [50, "introduction-to-netris-custom-resources"], [55, "introduction-to-netris-custom-resources"], [60, "introduction-to-netris-custom-resources"], [65, "introduction-to-netris-custom-resources"], [70, "introduction-to-netris-custom-resources"], [75, "introduction-to-netris-custom-resources"], [80, "introduction-to-netris-custom-resources"], [85, "introduction-to-netris-custom-resources"], [90, "introduction-to-netris-custom-resources"], [95, "introduction-to-netris-custom-resources"], [100, "introduction-to-netris-custom-resources"], [105, "introduction-to-netris-custom-resources"]], "L4LB Custom Resource": [[22, "l4lb-custom-resource"], [35, "l4lb-custom-resource"], [40, "l4lb-custom-resource"], [45, "l4lb-custom-resource"], [50, "l4lb-custom-resource"], [55, "l4lb-custom-resource"], [60, "l4lb-custom-resource"], [65, "l4lb-custom-resource"], [70, "l4lb-custom-resource"], [75, "l4lb-custom-resource"], [80, "l4lb-custom-resource"], [85, "l4lb-custom-resource"], [90, "l4lb-custom-resource"], [95, "l4lb-custom-resource"], [100, "l4lb-custom-resource"], [105, "l4lb-custom-resource"]], "V-Net Custom Resource": [[22, "v-net-custom-resource"], [35, "v-net-custom-resource"], [40, "v-net-custom-resource"], [45, "v-net-custom-resource"], [50, "v-net-custom-resource"], [55, "v-net-custom-resource"], [60, "v-net-custom-resource"], [65, "v-net-custom-resource"], [70, "v-net-custom-resource"], [75, "v-net-custom-resource"], [80, "v-net-custom-resource"], [85, "v-net-custom-resource"], [90, "v-net-custom-resource"], [95, "v-net-custom-resource"], [100, "v-net-custom-resource"], [105, "v-net-custom-resource"]], "BGP Custom Resource": [[22, "bgp-custom-resource"], [35, "bgp-custom-resource"], [40, "bgp-custom-resource"], [45, "bgp-custom-resource"], [50, "bgp-custom-resource"], [55, "bgp-custom-resource"], [60, "bgp-custom-resource"], [65, "bgp-custom-resource"], [70, "bgp-custom-resource"], [75, "bgp-custom-resource"], [80, "bgp-custom-resource"], [85, "bgp-custom-resource"], [90, "bgp-custom-resource"], [95, "bgp-custom-resource"], [100, "bgp-custom-resource"], [105, "bgp-custom-resource"]], "Importing existing resources from Netris Controller to Kubernetes": [[22, "importing-existing-resources-from-netris-controller-to-kubernetes"]], "Reclaim Policy": [[22, "reclaim-policy"], [35, "reclaim-policy"], [40, "reclaim-policy"], [45, "reclaim-policy"], [50, "reclaim-policy"], [55, "reclaim-policy"], [60, "reclaim-policy"], [65, "reclaim-policy"], [70, "reclaim-policy"], [75, "reclaim-policy"], [80, "reclaim-policy"], [85, "reclaim-policy"], [90, "reclaim-policy"], [95, "reclaim-policy"], [100, "reclaim-policy"], [105, "reclaim-policy"]], "Calico CNI Integration": [[22, "calico-cni-integration"]], "Disabling Netris-Calico Integration": [[22, "disabling-netris-calico-integration"], [35, "disabling-netris-calico-integration"], [40, "disabling-netris-calico-integration"], [45, "disabling-netris-calico-integration"], [50, "disabling-netris-calico-integration"], [55, "disabling-netris-calico-integration"], [60, "disabling-netris-calico-integration"], [65, "disabling-netris-calico-integration"], [70, "disabling-netris-calico-integration"], [75, "disabling-netris-calico-integration"], [80, "disabling-netris-calico-integration"], [85, "disabling-netris-calico-integration"], [90, "disabling-netris-calico-integration"], [95, "disabling-netris-calico-integration"], [100, "disabling-netris-calico-integration"], [105, "disabling-netris-calico-integration"]], "L3 Load Balancer (Anycast LB)": [[23, "l3-load-balancer-anycast-lb"]], "Creating an L3 Load Balancer": [[23, "creating-an-l3-load-balancer"]], "L4 Load Balancer (L4LB)": [[24, "l4-load-balancer-l4lb"]], "Enabling L4LB service": [[24, "enabling-l4lb-service"]], "Consuming L4LB service": [[24, "consuming-l4lb-service"]], "Link Aggregation (LAG)": [[25, "link-aggregation-lag"]], "Automatic LAG with EVPN Multi-homing": [[25, "automatic-lag-with-evpn-multi-homing"]], "Custom LAG": [[25, "custom-lag"]], "Maintenance Mode": [[26, "maintenance-mode"]], "Overview": [[26, "overview"]], "Maintenance Mode for Softgate - What\u2019s happening behind the scenes?": [[26, "maintenance-mode-for-softgate-whats-happening-behind-the-scenes"]], "Maintenance Mode for Switch - What\u2019s happening behind the scenes?": [[26, "maintenance-mode-for-switch-whats-happening-behind-the-scenes"]], "Netris Architecture": [[27, "netris-architecture"]], "Netris Controller": [[27, "netris-controller"], [36, "netris-controller"], [41, "netris-controller"], [46, "netris-controller"], [51, "netris-controller"], [56, "netris-controller"], [61, "netris-controller"], [66, "netris-controller"], [71, "netris-controller"], [76, "netris-controller"], [81, "netris-controller"], [86, "netris-controller"], [96, "netris-controller"], [101, "netris-controller"], [106, "netris-controller"], [110, "netris-controller"]], "Netris Switch Agent": [[27, "netris-switch-agent"]], "Netris SoftGate": [[27, "netris-softgate"]], "VPC": [[28, "vpc"]], "Adding new VPC": [[28, "adding-new-vpc"]], "IP Address Management": [[28, "ip-address-management"]], "IPAM Tree View": [[28, "ipam-tree-view"]], "Basic BGP": [[28, "basic-bgp"]], "Adding BGP Peers": [[28, "adding-bgp-peers"]], "Advanced BGP": [[28, "advanced-bgp"]], "BGP Objects": [[28, "bgp-objects"]], "IPv4 Prefix": [[28, "ipv4-prefix"]], "IPv6 Prefix": [[28, "ipv6-prefix"]], "Community": [[28, "community"]], "BGP route-maps": [[28, "bgp-route-maps"]], "Static Routing": [[28, "static-routing"]], "NAT": [[28, "nat"]], "Enabling NAT": [[28, "enabling-nat"]], "Defining NAT rules": [[28, "defining-nat-rules"]], "NAT Rule Fields": [[28, "id3"]], "SiteMesh": [[28, "sitemesh"]], "Looking Glass": [[28, "looking-glass"]], "Network Reference Designs": [[29, "network-reference-designs"]], "Release notes": [[30, "release-notes"]], "ROH (Routing on the Host)": [[31, "roh-routing-on-the-host"]], "Adding ROH Hosts": [[31, "adding-roh-hosts"]], "Provided Example Configurations": [[32, "provided-example-configurations"], [37, "provided-example-configurations"], [42, "provided-example-configurations"], [47, "provided-example-configurations"], [52, "provided-example-configurations"], [57, "provided-example-configurations"], [62, "provided-example-configurations"], [67, "provided-example-configurations"], [72, "provided-example-configurations"], [77, "provided-example-configurations"], [82, "provided-example-configurations"], [87, "provided-example-configurations"], [92, "provided-example-configurations"], [97, "provided-example-configurations"], [102, "provided-example-configurations"]], "V-Net (Ethernet/Vlan/VXlan) Example": [[32, "v-net-ethernet-vlan-vxlan-example"], [37, "v-net-ethernet-vlan-vxlan-example"], [42, "v-net-ethernet-vlan-vxlan-example"], [47, "v-net-ethernet-vlan-vxlan-example"], [52, "v-net-ethernet-vlan-vxlan-example"], [57, "v-net-ethernet-vlan-vxlan-example"], [62, "v-net-ethernet-vlan-vxlan-example"], [67, "v-net-ethernet-vlan-vxlan-example"], [72, "v-net-ethernet-vlan-vxlan-example"], [77, "v-net-ethernet-vlan-vxlan-example"], [82, "v-net-ethernet-vlan-vxlan-example"], [87, "v-net-ethernet-vlan-vxlan-example"], [92, "v-net-ethernet-vlan-vxlan-example"], [97, "v-net-ethernet-vlan-vxlan-example"], [102, "v-net-ethernet-vlan-vxlan-example"]], "E-BGP (Exterior Border Gateway Protocol) Example": [[32, "e-bgp-exterior-border-gateway-protocol-example"], [37, "e-bgp-exterior-border-gateway-protocol-example"], [42, "e-bgp-exterior-border-gateway-protocol-example"], [47, "e-bgp-exterior-border-gateway-protocol-example"], [52, "e-bgp-exterior-border-gateway-protocol-example"], [57, "e-bgp-exterior-border-gateway-protocol-example"], [62, "e-bgp-exterior-border-gateway-protocol-example"], [67, "e-bgp-exterior-border-gateway-protocol-example"], [72, "e-bgp-exterior-border-gateway-protocol-example"], [77, "e-bgp-exterior-border-gateway-protocol-example"], [82, "e-bgp-exterior-border-gateway-protocol-example"], [87, "e-bgp-exterior-border-gateway-protocol-example"], [92, "e-bgp-exterior-border-gateway-protocol-example"], [97, "e-bgp-exterior-border-gateway-protocol-example"], [102, "e-bgp-exterior-border-gateway-protocol-example"]], "NAT (Network Address Translation) Example": [[32, "nat-network-address-translation-example"], [37, "nat-network-address-translation-example"], [42, "nat-network-address-translation-example"], [47, "nat-network-address-translation-example"], [52, "nat-network-address-translation-example"], [57, "nat-network-address-translation-example"], [62, "nat-network-address-translation-example"], [67, "nat-network-address-translation-example"], [72, "nat-network-address-translation-example"], [77, "nat-network-address-translation-example"], [82, "nat-network-address-translation-example"], [87, "nat-network-address-translation-example"], [92, "nat-network-address-translation-example"], [97, "nat-network-address-translation-example"], [102, "nat-network-address-translation-example"]], "ACL (Access Control List) Example": [[32, "acl-access-control-list-example"], [37, "acl-access-control-list-example"], [42, "acl-access-control-list-example"], [47, "acl-access-control-list-example"], [52, "acl-access-control-list-example"], [57, "acl-access-control-list-example"], [62, "acl-access-control-list-example"], [67, "acl-access-control-list-example"], [72, "acl-access-control-list-example"], [77, "acl-access-control-list-example"], [82, "acl-access-control-list-example"], [87, "acl-access-control-list-example"], [92, "acl-access-control-list-example"], [97, "acl-access-control-list-example"], [102, "acl-access-control-list-example"]], "Learn by Creating Services": [[33, "learn-by-creating-services"], [38, "learn-by-creating-services"], [43, "learn-by-creating-services"], [48, "learn-by-creating-services"], [53, "learn-by-creating-services"], [58, "learn-by-creating-services"], [63, "learn-by-creating-services"], [68, "learn-by-creating-services"], [73, "learn-by-creating-services"], [78, "learn-by-creating-services"], [83, "learn-by-creating-services"], [88, "learn-by-creating-services"], [93, "learn-by-creating-services"], [98, "learn-by-creating-services"], [103, "learn-by-creating-services"]], "V-Net (Ethernet/Vlan/VXlan)": [[33, "v-net-ethernet-vlan-vxlan"], [38, "v-net-ethernet-vlan-vxlan"], [43, "v-net-ethernet-vlan-vxlan"], [48, "v-net-ethernet-vlan-vxlan"], [53, "v-net-ethernet-vlan-vxlan"], [58, "v-net-ethernet-vlan-vxlan"], [63, "v-net-ethernet-vlan-vxlan"], [68, "v-net-ethernet-vlan-vxlan"], [73, "v-net-ethernet-vlan-vxlan"], [78, "v-net-ethernet-vlan-vxlan"], [83, "v-net-ethernet-vlan-vxlan"], [88, "v-net-ethernet-vlan-vxlan"], [93, "v-net-ethernet-vlan-vxlan"], [98, "v-net-ethernet-vlan-vxlan"], [103, "v-net-ethernet-vlan-vxlan"]], "E-BGP (Exterior Border Gateway Protocol)": [[33, "e-bgp-exterior-border-gateway-protocol"], [38, "e-bgp-exterior-border-gateway-protocol"], [43, "e-bgp-exterior-border-gateway-protocol"], [48, "e-bgp-exterior-border-gateway-protocol"], [53, "e-bgp-exterior-border-gateway-protocol"], [58, "e-bgp-exterior-border-gateway-protocol"], [63, "e-bgp-exterior-border-gateway-protocol"], [68, "e-bgp-exterior-border-gateway-protocol"], [73, "e-bgp-exterior-border-gateway-protocol"], [78, "e-bgp-exterior-border-gateway-protocol"], [83, "e-bgp-exterior-border-gateway-protocol"], [88, "e-bgp-exterior-border-gateway-protocol"], [93, "e-bgp-exterior-border-gateway-protocol"], [98, "e-bgp-exterior-border-gateway-protocol"], [103, "e-bgp-exterior-border-gateway-protocol"]], "NAT (Network Address Translation)": [[33, "nat-network-address-translation"], [38, "nat-network-address-translation"], [43, "nat-network-address-translation"], [48, "nat-network-address-translation"], [53, "nat-network-address-translation"], [58, "nat-network-address-translation"], [63, "nat-network-address-translation"], [68, "nat-network-address-translation"], [73, "nat-network-address-translation"], [78, "nat-network-address-translation"], [83, "nat-network-address-translation"], [88, "nat-network-address-translation"], [93, "nat-network-address-translation"], [98, "nat-network-address-translation"], [103, "nat-network-address-translation"]], "L3LB (Anycast L3 Load Balancer)": [[33, "l3lb-anycast-l3-load-balancer"], [43, "l3lb-anycast-l3-load-balancer"], [48, "l3lb-anycast-l3-load-balancer"], [53, "l3lb-anycast-l3-load-balancer"], [58, "l3lb-anycast-l3-load-balancer"], [63, "l3lb-anycast-l3-load-balancer"], [68, "l3lb-anycast-l3-load-balancer"], [73, "l3lb-anycast-l3-load-balancer"], [78, "l3lb-anycast-l3-load-balancer"], [83, "l3lb-anycast-l3-load-balancer"], [93, "l3lb-anycast-l3-load-balancer"], [98, "l3lb-anycast-l3-load-balancer"], [103, "l3lb-anycast-l3-load-balancer"]], "ACL (Access Control List)": [[33, "acl-access-control-list"], [38, "acl-access-control-list"], [43, "acl-access-control-list"], [48, "acl-access-control-list"], [53, "acl-access-control-list"], [58, "acl-access-control-list"], [63, "acl-access-control-list"], [68, "acl-access-control-list"], [73, "acl-access-control-list"], [78, "acl-access-control-list"], [83, "acl-access-control-list"], [88, "acl-access-control-list"], [93, "acl-access-control-list"], [98, "acl-access-control-list"], [103, "acl-access-control-list"]], "Sandbox1": [[34, "sandbox1"]], "Learn Netris Operations with Kubernetes": [[35, "learn-netris-operations-with-kubernetes"], [40, "learn-netris-operations-with-kubernetes"], [45, "learn-netris-operations-with-kubernetes"], [50, "learn-netris-operations-with-kubernetes"], [55, "learn-netris-operations-with-kubernetes"], [60, "learn-netris-operations-with-kubernetes"], [65, "learn-netris-operations-with-kubernetes"], [70, "learn-netris-operations-with-kubernetes"], [75, "learn-netris-operations-with-kubernetes"], [80, "learn-netris-operations-with-kubernetes"], [85, "learn-netris-operations-with-kubernetes"], [95, "learn-netris-operations-with-kubernetes"], [100, "learn-netris-operations-with-kubernetes"], [105, "learn-netris-operations-with-kubernetes"]], "Intro": [[35, "intro"], [40, "intro"], [45, "intro"], [50, "intro"], [55, "intro"], [60, "intro"], [65, "intro"], [70, "intro"], [75, "intro"], [80, "intro"], [85, "intro"], [90, "intro"], [95, "intro"], [100, "intro"], [105, "intro"]], "Deploy an Application with an On-Demand Netris Load Balancer": [[35, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [40, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [45, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [50, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [55, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [60, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [65, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [70, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [75, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [80, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [85, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [90, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [95, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [100, "deploy-an-application-with-an-on-demand-netris-load-balancer"], [105, "deploy-an-application-with-an-on-demand-netris-load-balancer"]], "Milestone 1": [[35, null], [40, null], [45, null], [50, null], [55, null], [60, null], [65, null], [70, null], [75, null], [80, null], [85, null], [90, null], [95, null], [100, null], [105, null]], "Importing Existing Resources from Netris Controller to Kubernetes": [[35, "importing-existing-resources-from-netris-controller-to-kubernetes"], [40, "importing-existing-resources-from-netris-controller-to-kubernetes"], [45, "importing-existing-resources-from-netris-controller-to-kubernetes"], [50, "importing-existing-resources-from-netris-controller-to-kubernetes"], [55, "importing-existing-resources-from-netris-controller-to-kubernetes"], [60, "importing-existing-resources-from-netris-controller-to-kubernetes"], [65, "importing-existing-resources-from-netris-controller-to-kubernetes"], [70, "importing-existing-resources-from-netris-controller-to-kubernetes"], [75, "importing-existing-resources-from-netris-controller-to-kubernetes"], [80, "importing-existing-resources-from-netris-controller-to-kubernetes"], [85, "importing-existing-resources-from-netris-controller-to-kubernetes"], [90, "importing-existing-resources-from-netris-controller-to-kubernetes"], [95, "importing-existing-resources-from-netris-controller-to-kubernetes"], [100, "importing-existing-resources-from-netris-controller-to-kubernetes"], [105, "importing-existing-resources-from-netris-controller-to-kubernetes"]], "Netris Calico CNI Integration": [[35, "netris-calico-cni-integration"], [40, "netris-calico-cni-integration"], [45, "netris-calico-cni-integration"], [50, "netris-calico-cni-integration"], [55, "netris-calico-cni-integration"], [60, "netris-calico-cni-integration"], [65, "netris-calico-cni-integration"], [70, "netris-calico-cni-integration"], [75, "netris-calico-cni-integration"], [80, "netris-calico-cni-integration"], [85, "netris-calico-cni-integration"], [90, "netris-calico-cni-integration"], [95, "netris-calico-cni-integration"], [100, "netris-calico-cni-integration"], [105, "netris-calico-cni-integration"]], "Milestone 2": [[35, null], [40, null], [45, null], [50, null], [55, null], [60, null], [65, null], [70, null], [75, null], [80, null], [85, null], [90, null], [95, null], [100, null], [105, null]], "Welcome to Netris Sandbox": [[36, "welcome-to-netris-sandbox"], [41, "welcome-to-netris-sandbox"], [46, "welcome-to-netris-sandbox"], [51, "welcome-to-netris-sandbox"], [56, "welcome-to-netris-sandbox"], [61, "welcome-to-netris-sandbox"], [66, "welcome-to-netris-sandbox"], [71, "welcome-to-netris-sandbox"], [76, "welcome-to-netris-sandbox"], [81, "welcome-to-netris-sandbox"], [86, "welcome-to-netris-sandbox"], [91, "welcome-to-netris-sandbox"], [96, "welcome-to-netris-sandbox"], [101, "welcome-to-netris-sandbox"], [106, "welcome-to-netris-sandbox"]], "Topology diagram": [[36, "topology-diagram"], [41, "topology-diagram"], [46, "topology-diagram"], [51, "topology-diagram"], [56, "topology-diagram"], [61, "topology-diagram"], [66, "topology-diagram"], [71, "topology-diagram"], [76, "topology-diagram"], [81, "topology-diagram"], [86, "topology-diagram"], [91, "topology-diagram"], [96, "topology-diagram"], [101, "topology-diagram"], [106, "topology-diagram"]], "Linux servers": [[36, "linux-servers"], [41, "linux-servers"], [46, "linux-servers"], [51, "linux-servers"], [56, "linux-servers"], [61, "linux-servers"], [66, "linux-servers"], [71, "linux-servers"], [76, "linux-servers"], [81, "linux-servers"], [86, "linux-servers"], [91, "linux-servers"], [96, "linux-servers"], [101, "linux-servers"], [106, "linux-servers"]], "Kubernetes cluster": [[36, "kubernetes-cluster"], [41, "kubernetes-cluster"], [46, "kubernetes-cluster"], [51, "kubernetes-cluster"], [56, "kubernetes-cluster"], [61, "kubernetes-cluster"], [66, "kubernetes-cluster"], [71, "kubernetes-cluster"], [76, "kubernetes-cluster"], [81, "kubernetes-cluster"], [86, "kubernetes-cluster"], [91, "kubernetes-cluster"], [96, "kubernetes-cluster"], [101, "kubernetes-cluster"], [106, "kubernetes-cluster"]], "Upstream ISP": [[36, "upstream-isp"], [41, "upstream-isp"], [46, "upstream-isp"], [51, "upstream-isp"], [56, "upstream-isp"], [61, "upstream-isp"], [66, "upstream-isp"], [71, "upstream-isp"], [76, "upstream-isp"], [81, "upstream-isp"], [86, "upstream-isp"], [91, "upstream-isp"], [96, "upstream-isp"], [101, "upstream-isp"], [106, "upstream-isp"]], "Networks Used": [[36, "networks-used"], [41, "networks-used"], [46, "networks-used"], [51, "networks-used"], [56, "networks-used"], [61, "networks-used"], [66, "networks-used"], [71, "networks-used"], [76, "networks-used"], [81, "networks-used"], [86, "networks-used"], [96, "networks-used"], [101, "networks-used"], [106, "networks-used"]], "L3LB (Anycast L3 load balancer)": [[38, "l3lb-anycast-l3-load-balancer"]], "Sandbox10": [[39, "sandbox10"]], "Sandbox11": [[44, "sandbox11"]], "Sandbox12": [[49, "sandbox12"]], "Sandbox13": [[54, "sandbox13"]], "Sandbox14": [[59, "sandbox14"]], "Sandbox15": [[64, "sandbox15"]], "Sandbox2": [[69, "sandbox2"]], "Sandbox3": [[74, "sandbox3"]], "Sandbox4": [[79, "sandbox4"]], "Sandbox5": [[84, "sandbox5"]], "Sandbox6": [[89, "sandbox6"]], "Learn Netris operations with Kubernetes": [[90, "learn-netris-operations-with-kubernetes"]], "Netris GUI": [[91, "netris-gui"]], "Networks used": [[91, "networks-used"]], "Sandbox7": [[94, "sandbox7"]], "Sandbox8": [[99, "sandbox8"]], "Sandbox9": [[104, "sandbox9"]], "SoftGate Performance": [[107, "softgate-performance"], [107, "id1"]], "Reference Network Architectures": [[108, "reference-network-architectures"]], "Unmanaged Switch & Netris SoftGate": [[108, "unmanaged-switch-netris-softgate"]], "Unmanaged Switch & SoftGate (HA)": [[108, "unmanaged-switch-softgate-ha"]], "Netris Managed Switch & SoftGate (HA)": [[108, "netris-managed-switch-softgate-ha"]], "Netris Managed Switch & SoftGate scalable data center (HA)": [[108, "netris-managed-switch-softgate-scalable-data-center-ha"]], "Netris Supported Functionality & Platforms Matrix": [[109, "netris-supported-functionality-platforms-matrix"]], "Switch Fabric Management Functions": [[109, "switch-fabric-management-functions"]], "External Routing Functions": [[109, "external-routing-functions"]], "Cloud Networking Functions & Constructs": [[109, "cloud-networking-functions-constructs"]], "Overlay Network Features": [[109, "overlay-network-features"]], "AI Specific Functions": [[109, "ai-specific-functions"]], "Security": [[109, "security"]], "Administration": [[109, "administration"]], "Management Interfaces": [[109, "management-interfaces"]], "Hypervisor/Worker node specific functionality": [[109, "hypervisor-worker-node-specific-functionality"]], "SoftGate Data Plane Variations": [[109, "softgate-data-plane-variations"]], "Netris and NOS versions compatibility matrix": [[109, "netris-and-nos-versions-compatibility-matrix"]], "Hardware Requirements": [[110, "hardware-requirements"]], "Netris SoftGate HS (The Horizontally Scalable version)": [[110, "netris-softgate-hs-the-horizontally-scalable-version"]], "Supported Switch Hardware": [[110, "supported-switch-hardware"]], "Nvidia": [[110, "nvidia"]], "Dell": [[110, "dell"]], "EdgeCore": [[110, "edgecore"]], "Arista": [[110, "arista"]], "Network Switch Initial Setup": [[111, "id1"]], "Switch Ports": [[112, "switch-ports"]], "Terraform: Netris provider": [[113, "terraform-netris-provider"]], "To create your first Terraform configuration:": [[113, "to-create-your-first-terraform-configuration"]], "Install Terraform": [[113, "install-terraform"]], "Create a directory for Terraform files": [[113, "create-a-directory-for-terraform-files"]], "Configure a provider": [[113, "configure-a-provider"]], "Prepare an infrastructure plan": [[113, "prepare-an-infrastructure-plan"]], "Create resources": [[113, "create-resources"]], "Delete resources": [[113, "delete-resources"]], "Inventory": [[114, "inventory"]], "Adding Switches": [[114, "adding-switches"]], "Add Inventory Fields - Switch": [[114, "id2"]], "Adding SoftGates": [[114, "adding-softgates"]], "Add Inventory Fields - SoftGate": [[114, "id3"]], "Viewing Inventory": [[114, "viewing-inventory"]], "Topology Manager": [[114, "topology-manager"]], "Adding Links": [[114, "adding-links"]], "Netris Try & Learn": [[115, "netris-try-learn"]], "Netris Infrastructure Simulation Platform": [[116, "netris-infrastructure-simulation-platform"]], "Lab Scenario: GPU-as-a-Service network with NVIDIA Spectrum-X architecture": [[117, "lab-scenario-gpu-as-a-service-network-with-nvidia-spectrum-x-architecture"]], "Initialize the Netris controller": [[117, "initialize-the-netris-controller"]], "Start a simulation": [[117, "start-a-simulation"]], "Monitoring Dashboard": [[117, "monitoring-dashboard"]], "Topology": [[117, "topology"]], "SSH to Switches": [[117, "ssh-to-switches"]], "SSH to GPU servers": [[117, "ssh-to-gpu-servers"]], "NHN (Netris host networking plugin)": [[117, "nhn-netris-host-networking-plugin"]], "Server Cluster Template": [[117, "server-cluster-template"]], "Server Cluster": [[117, "server-cluster"]], "Checking the connectivity": [[117, "checking-the-connectivity"]], "Cleanup the Controller": [[117, "cleanup-the-controller"]], "Netris Test Controller & Infrastructure Simulation": [[118, "netris-test-controller-infrastructure-simulation"]], "Web Console": [[118, "web-console"]], "SSH": [[118, "ssh"]], "Netris Init Modules": [[118, "netris-init-modules"]], "Start/Stop a Simulation": [[118, "start-stop-a-simulation"]], "Activating BGP on Equinix Metal Project": [[119, "activating-bgp-on-equinix-metal-project"]], "Provisioning Netris SoftGate nodes in Equinix Metal Project": [[120, "provisioning-netris-softgate-nodes-in-equinix-metal-project"]], "Site Mesh with AWS Overview": [[121, "site-mesh-with-aws-overview"]], "Introduction": [[121, "introduction"], [130, "introduction"], [142, "introduction"], [151, "introduction"], [166, "introduction"]], "Concept": [[121, "concept"], [130, "concept"], [142, "concept"], [151, "concept"], [166, "concept"]], "Deploy a Softgate in AWS": [[122, "deploy-a-softgate-in-aws"]], "Create an EC2 instance": [[122, "create-an-ec2-instance"]], "Configure Netris Controller": [[122, "configure-netris-controller"], [131, "configure-netris-controller"]], "Pre-Requisite Steps": [[122, "pre-requisite-steps"], [131, "pre-requisite-steps"]], "Add AWS VPC Subnet into the Netris Controller": [[122, "add-aws-vpc-subnet-into-the-netris-controller"]], "Static route for AWS VPC Subnet in Netris Controller": [[122, "static-route-for-aws-vpc-subnet-in-netris-controller"]], "Create the Softgate in the Netris Controller": [[122, "create-the-softgate-in-the-netris-controller"], [131, "create-the-softgate-in-the-netris-controller"]], "Netris SoftGate node provisioning": [[122, "netris-softgate-node-provisioning"], [131, "netris-softgate-node-provisioning"]], "Configure AWS VPC": [[122, "configure-aws-vpc"]], "Adding routes": [[122, "adding-routes"]], "EC2 Source / destination check": [[122, "ec2-source-destination-check"]], "Enabling Site Mesh": [[122, "enabling-site-mesh"], [131, "enabling-site-mesh"]], "Connecting Netris managed fabric to an ISP": [[123, "connecting-netris-managed-fabric-to-an-isp"]], "Connecting servers to the Netris managed fabric": [[124, "connecting-servers-to-the-netris-managed-fabric"]], "Creating an Interconnection to Equinix Fabric": [[125, "creating-an-interconnection-to-equinix-fabric"]], "Enabling services (NAT, V-Net, Load Balancer, IP pools)": [[126, "enabling-services-nat-v-net-load-balancer-ip-pools"]], "1) Requesting new Public IP address block": [[126, "requesting-new-public-ip-address-block"]], "2) Enable on-demand (elastic) Load Balancer": [[126, "enable-on-demand-elastic-load-balancer"]], "3) Enable V-Net": [[126, "enable-v-net"]], "4) Enable NAT": [[126, "enable-nat"]], "Enabling Load-balancing services": [[127, "enabling-load-balancing-services"]], "Enabling NAT services": [[128, "enabling-nat-services"]], "Enable Equinix Metal API integration": [[129, "enable-equinix-metal-api-integration"]], "Site Mesh with GCP Overview": [[130, "site-mesh-with-gcp-overview"]], "Deploy a Softgate in GCP": [[131, "deploy-a-softgate-in-gcp"]], "Create a VM instance": [[131, "create-a-vm-instance"]], "Add GCP VPC Subnet(s) into the Netris Controller": [[131, "add-gcp-vpc-subnet-s-into-the-netris-controller"]], "Static route for GCP VPC Subnet(s) in Netris Controller": [[131, "static-route-for-gcp-vpc-subnet-s-in-netris-controller"]], "Configure routing in GCP VPC": [[131, "configure-routing-in-gcp-vpc"]], "Equinix Metal API integration enablement": [[132, "equinix-metal-api-integration-enablement"]], "Adding Netris SoftGate nodes": [[132, "adding-netris-softgate-nodes"]], "Netris Generic Tutorials": [[133, "netris-generic-tutorials"]], "Netris VPC for Equinix Metal Getting Started Guide": [[134, "netris-vpc-for-equinix-metal-getting-started-guide"]], "Step-by-step Netris Setup Guide for Equinix Metal": [[134, null]], "Examples of Using Netris VPC on Equinix Metal Project": [[134, null]], "VPC for Anywhere Getting Started Guide": [[135, "vpc-for-anywhere-getting-started-guide"]], "Installing a Netris Controller on Equinix Metal on-demand server": [[136, "installing-a-netris-controller-on-equinix-metal-on-demand-server"]], "Installation Steps": [[136, "installation-steps"], [152, "installation-steps"]], "Security Matters": [[136, "security-matters"], [137, "security-matters"], [141, "security-matters"], [152, "security-matters"], [167, "security-matters"]], "Inventory setup": [[138, "inventory-setup"]], "IPAM setup": [[139, "ipam-setup"]], "More features": [[140, "more-features"]], "Install a Netris Controller": [[141, "install-a-netris-controller"], [167, "install-a-netris-controller"]], "Requirements and Installation steps": [[141, "requirements-and-installation-steps"], [167, "requirements-and-installation-steps"]], "Netris managed fabric Overview": [[142, "netris-managed-fabric-overview"]], "Netris Switch Agent Installation": [[143, "netris-switch-agent-installation"]], "Enabling Site Mesh with AWS": [[144, "enabling-site-mesh-with-aws"]], "Setup": [[144, null], [145, null], [146, null], [147, null], [164, null]], "Getting Started with Equinix Metal VPC": [[145, "getting-started-with-equinix-metal-vpc"]], "Use": [[145, null], [147, null], [164, null]], "Enabling Site Mesh with GCP": [[146, "enabling-site-mesh-with-gcp"]], "Getting Started with PhoenixNAP VPC": [[147, "getting-started-with-phoenixnap-vpc"]], "New Site setup": [[148, "new-site-setup"]], "Provisioning Netris SoftGate nodes in phoenixNAP BMC": [[149, "provisioning-netris-softgate-nodes-in-phoenixnap-bmc"]], "Enable phoenixNAP BMC API integration": [[150, "enable-phoenixnap-bmc-api-integration"]], "Netris VPC for phoenixNAP BMC Overview": [[151, "netris-vpc-for-phoenixnap-bmc-overview"]], "Installing a Netris Controller on phoenixNAP BMC server": [[152, "installing-a-netris-controller-on-phoenixnap-bmc-server"]], "IPAM Setup for Services": [[153, "ipam-setup-for-services"], [168, "ipam-setup-for-services"]], "Option 1 - Separated allocation for each purpose": [[153, "option-1-separated-allocation-for-each-purpose"]], "1. Enable on-demand (elastic) Load Balancer": [[153, "enable-on-demand-elastic-load-balancer"], [153, "id1"]], "2. Enable NAT": [[153, "enable-nat"], [153, "id2"]], "Option 2 - Splitting single allocation into different subnets": [[153, "option-2-splitting-single-allocation-into-different-subnets"]], "Note*": [[153, "note"]], "Installing Netris on phoenixNAP BMC": [[154, "installing-netris-on-phoenixnap-bmc"]], "Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC": [[155, "using-on-demand-elastic-l4-load-balancer-service-in-phoenixnap-bmc"]], "Using NAT services in phoenixNAP BMC": [[156, "using-nat-services-in-phoenixnap-bmc"]], "Option 1 - MASQUERADE": [[156, "option-1-masquerade"]], "Option 2 - SNAT": [[156, "option-2-snat"]], "DNAT": [[156, "dnat"], [173, "dnat"]], "Using V-Net (isolated virtual network) services in phoenixNAP BMC": [[157, "using-v-net-isolated-virtual-network-services-in-phoenixnap-bmc"]], "Adding Subnets for V-Net": [[157, "adding-subnets-for-v-net"]], "Creating a V-Net": [[157, "creating-a-v-net"], [172, "creating-a-v-net"], [174, "creating-a-v-net"]], "Deploy a new server w/ dynamic IP into an existing V-Net": [[157, "deploy-a-new-server-w-dynamic-ip-into-an-existing-v-net"]], "Deploy a new server w/ static IP into an existing V-Net": [[157, "deploy-a-new-server-w-static-ip-into-an-existing-v-net"]], "Tags": [[157, "tags"], [163, "tags"]], "Unmanaged": [[157, "unmanaged"], [163, "unmanaged"]], "SoftGate software provisioning": [[158, "softgate-software-provisioning"], [169, "softgate-software-provisioning"]], "SoftGate PRO (100G) software provisioning": [[158, "softgate-pro-100g-software-provisioning"]], "SoftGate (1G, 10G) software provisioning": [[158, "softgate-1g-10g-software-provisioning"]], "Topology setup": [[159, "topology-setup"]], "Upgrade and Rollback": [[160, "upgrade-and-rollback"]], "Netris upgrade Procedure": [[160, "netris-upgrade-procedure"]], "Backup current database": [[160, "backup-current-database"]], "Stop Netris Agents": [[160, "stop-netris-agents"], [160, "id1"]], "Check your current version": [[160, "check-your-current-version"]], "Upgrade the Controller": [[160, "upgrade-the-controller"]], "Check the upgraded version": [[160, "check-the-upgraded-version"]], "Upgrade Switches and SoftGate nodes": [[160, "upgrade-switches-and-softgate-nodes"]], "Rollback Procedure": [[160, "rollback-procedure"]], "Restore The Database": [[160, "restore-the-database"]], "Downgrade the Controller Software": [[160, "downgrade-the-controller-software"]], "Downgrade Netris Agent Software": [[160, "downgrade-netris-agent-software"]], "Upgrading SONiC Operating System": [[161, "upgrading-sonic-operating-system"]], "Upgrade Procedure": [[161, "upgrade-procedure"]], "Using on-demand (elastic) L4 Load Balancer service": [[162, "using-on-demand-elastic-l4-load-balancer-service"], [171, "using-on-demand-elastic-l4-load-balancer-service"]], "Using V-Net (isolated virtual network) services in Equinix Metal Project": [[163, "using-v-net-isolated-virtual-network-services-in-equinix-metal-project"]], "VPC Anywhere Getting Started Guide": [[164, "vpc-anywhere-getting-started-guide"]], "Check Default Site Settings": [[165, "check-default-site-settings"]], "VPC Anywhere Overview": [[166, "vpc-anywhere-overview"]], "Create an allocation": [[168, "create-an-allocation"]], "Enable on-demand L4 (elastic) Load Balancer": [[168, "enable-on-demand-l4-elastic-load-balancer"]], "Enable Network Address Translation (NAT)": [[168, "enable-network-address-translation-nat"]], "Connecting VPC to upstream networks (use one of two options)": [[170, "connecting-vpc-to-upstream-networks-use-one-of-two-options"]], "Using a VLAN with public IP addresses (DMZ)": [[170, "using-a-vlan-with-public-ip-addresses-dmz"]], "BGP Upstream": [[170, "bgp-upstream"]], "BGP Upstream: overview": [[170, "bgp-upstream-overview"]], "BGP Upstream: configuration": [[170, "bgp-upstream-configuration"]], "Using Multi-interface SoftGate": [[172, "using-multi-interface-softgate"]], "SoftGate to SoftGate Link": [[172, "softgate-to-softgate-link"]], "Using NAT services": [[173, "using-nat-services"]], "MASQUERADE": [[173, "masquerade"]], "SNAT": [[173, "snat"]], "Using V-Net (isolated virtual network) services": [[174, "using-v-net-isolated-virtual-network-services"]], "Getting Started with Switch-Fabric Manager & VPC": [[175, "getting-started-with-switch-fabric-manager-vpc"]], "VPC setup": [[176, "vpc-setup"]], "Visibility (Telescope)": [[177, "visibility-telescope"]], "Graph Boards": [[177, "graph-boards"]], "API Logs": [[177, "api-logs"]], "Dashboard": [[177, "dashboard"]], "V-Net": [[178, "v-net"]], "V-Net Fields": [[178, "v-net-fields"]], "Netris VPC": [[179, "netris-vpc"]], "Sites": [[179, "sites"]], "IPAM": [[179, "ipam"]], "V-Nets": [[179, "v-nets"]], "External connections": [[179, "external-connections"]], "SiteMesh connections": [[179, "sitemesh-connections"]], "NAT services": [[179, "nat-services"]], "Load-balancing service": [[179, "load-balancing-service"]], "Access lists": [[179, "access-lists"]], "Working with Netris VPC": [[179, "working-with-netris-vpc"]]}, "indexentries": {}}) \ No newline at end of file diff --git a/en/latest/softgate-performance.html b/en/latest/softgate-performance.html new file mode 100644 index 0000000000..3eedde37ad --- /dev/null +++ b/en/latest/softgate-performance.html @@ -0,0 +1,902 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate Performance — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

SoftGate Performance

+

The following tested results are offered to help properly size the hardware needed for a SoftGate with various types of services:

+ + +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SoftGate Performance

Routing(mpps)

Routing with unmatched DNAT rules (mpps)

Routing with unmatched DNAT and SNAT rules (mpps)

DNAT

SNAT

DNAT with unmatched SNAT rules

Xeon E5-2660 CPU cores: (6 core for fw) Scheduler 0 Forward 1-7 KNI 8-10 OS 11-15

~27mpps

~17mpps

7mpps

9mpps

~2mpps

Xeon Gold 6130 CPU cores: (6 core for fw) Scheduler 0 Forward 3-13(odd numbers) KNI 15 17 19

~31mpps

~22mpps

~15mpps

~11mpps

~3mpps(3M conntrack count) ~7mpps(20k conntrack count)

~11mpps

Xeon Gold 6130 CPU cores: (8 core for fw) Scheduler 0 Forward 3-17(odd numbers) KNI 19 21 23

~38mpps

~28mpps

19mpps

12mpps

~12mpps

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/supported-networks.html b/en/latest/supported-networks.html new file mode 100644 index 0000000000..00f0612113 --- /dev/null +++ b/en/latest/supported-networks.html @@ -0,0 +1,859 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Reference Network Architectures — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Reference Network Architectures
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Reference Network Architectures

+
+

Unmanaged Switch & Netris SoftGate

+Unmanaged Switch & SoftGate +
+
+
+

Unmanaged Switch & SoftGate (HA)

+Unmanaged Switch & SoftGate (HA) +
+
+
+

Netris Managed Switch & SoftGate (HA)

+Netris Managed Switch & SoftGate small data center (HA) +
+
+
+

Netris Managed Switch & SoftGate scalable data center (HA)

+Netris Managed Switch & SoftGate scalable data center (HA) +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/supported-platform-matrix.html b/en/latest/supported-platform-matrix.html new file mode 100644 index 0000000000..390b7268ac --- /dev/null +++ b/en/latest/supported-platform-matrix.html @@ -0,0 +1,1639 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Supported Functionality & Platforms Matrix — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris Supported Functionality & Platforms Matrix
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Netris Supported Functionality & Platforms Matrix

+
+

Switch Fabric Management Functions

+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Function

Description

Nvidia Spectrum

Dell SONiC

Arista EOS

EdgeCore SONiC

Equinix Metal

PhoenixNAP

Fabric Manager

Day0, Day1, and Day2 switch fabric operations.

Nov/2024

N/A

N/A

Parallel Fabrics

Manage multiple isolated switch fabrics. (example: East-West and North-South)

Nov/2024

N/A

N/A

Topology Manager

Design and operate the switch fabric.

Nov/2024

N/A

N/A

Maintenance Mode

Offload a network node for a maintenance.

Nov/2024

IPAM

Manage IP subnets. Assign RBAC, multi-tenancy, and service-based rules and roles to IP address resources.

Nov/2024

Looking Glass

Lookup underlay and overlay routing info of any managed network node without SSH-ing.

Nov/2024

Monitoring: Switch Ports

Automatic monitoring of Link statuses, link utilization, laser signal levels, errors, packets.

Nov/2024

N/A

N/A

Monitoring: Resources

Automatic monitoring of CPU, RAM, Disk, and ASIC resources.

Nov/2024

Monitoring: Sensors

Automatic monitoring of temperature, fans, power supply statuses.

Nov/2024

N/A

N/A

Monitoring: System Processes

Automatic monitoring of critical system processes.

Nov/2024

Topology Validation

Detect wiring errors.

Dec/2024

Nov/2024

Dec/2024

N/A

N/A

BGP Unnumbered

Any network topology with BGP unnumbered underlay

Nov/2024

N/A

N/A

BGP Numbered

Any network topology with BGP numbered underlay

Dec/2024

Nov/2024

Dec/2024

N/A

N/A

+
+
+

External Routing Functions

+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Function

Description

Nvidia Spectrum

Dell-SONiC

Arista EOS

EdgeCore-SONiC

Equinix Metal

PhoenixNAP

External BGP (SoftGate)

Terminate full routing table on SoftGate Gateway-server.

Nov/2024

External BGP (Switch)

Peer with external routers.

Nov/2024

N/A

N/A

BGP Route-Maps

Create chain of BGP rules.

Nov/2024

Static Routes

Define static routing rules.

Nov/2024

+
+
+

Cloud Networking Functions & Constructs

+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Function

Description

Nvidia Spectrum

Dell-SONiC

Arista EOS

EdgeCore-SONiC

Equinix Metal

PhoenixNAP

VPC (Virtual Private Cloud)

Isolated VPCs, VRFs. Overlapping IPs supported.

Nov/2024

V-Net (Subnet)

L3VPN VXLAN or L2VPN VXLAN with an anycast default Gateway, and built-in DHCP.

Nov/2024

Server Cluster (Profiling)

Create network constructs template, then apply it on groups of servers.

Dec/2024

Nov/2024

Dec/2024

TBD

TBD

Internet Gateway

Provide shared Internet access to V-Nets and VPC

Nov/2024

Dec/2024

✔ (single VPC)

✔ (single VPC)

NAT Gateway

Provide shared DNAT, PAT, 1:1 NAT to multiple V-Nets and multiple VPCs

Nov/2024

Dec/2024

✔ (single VPC)

✔ (single VPC)

L4 Load Balancer

Provide on-demand elastic load balancer service to hosts in multiple V-Nets and multiple VPCs

Nov/2024

Dec/2024

✔ (single VPC)

✔ (single VPC)

SiteMesh

Wireguard-based Site-to-Site VPN between multiple regions/sites. (single VPC)

TBD

Nov/2024

+
+
+

Overlay Network Features

+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Function

Description

Nvidia Spectrum

Dell-SONiC

Arista EOS

EdgeCore-SONiC

Equinix Metal

PhoenixNAP

L2VPN VXLAN VLAN Aware

L2VPN VXLAN with VLAN tagged or untagged termination on switch port.

Nov/2024

N/A

N/A

L2VPN VXLAN VLAN Unaware

L2VPN VXLAN with VLAN tagged or untagged termination on switch port supporting different VLAN IDs on different end points.

N/A

N/A

Nov/2024

N/A

N/A

N/A

L3VPN VXLAN

L3VPN VXLAN, Commonly used in high performance computing, such as AI clusters.

TBD

TBD

Dec/2024

N/A

N/A

EVPN-MH / VXLAN-ESI

EVPN MultiHoming based on VXLAN and ESI for automatic Active-Active server network multihoming

Dec/2024

Nov/2024

N/A

N/A

N/A

LACP

Link Aggregation or Active-Standby server multihoming.

Nov/2024

Dec/2024

N/A

N/A

MC-LAG

Traditional MC-LAG-based server multihoming

TBD

TBD

Dec/2024

N/A

N/A

+
+
+

AI Specific Functions

+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Function

Description

Nvidia Spectrum

Dell-SONiC

Arista EOS

EdgeCore-SONiC

Equinix Metal

PhoenixNAP

Spectrum-X

AI GPU cluster switch fabric operation for Nvidia Spectrum-X

N/A

N/A

N/A

N/A

N/A

Rail-optimized designs

Topology and best practices initialization module for rail-optimized fabrics

N/A

N/A

Dec/2024

N/A

N/A

QoS for RoCE

Enable QoS for RoCE workloads

N/A

N/A

N/A

N/A

N/A

RoCE Adaptive Routing

Enable RoCE adaptive routing

N/A

N/A

N/A

N/A

N/A

RoCE Congestion Control

Enable automatic congestion control for RoCE workloads

N/A

N/A

N/A

N/A

N/A

DPU/Host zero-touch configuration

Automatically configure IP addresses, routing, RoCE and other DPU/SuperNIC specific configuration on GPU servers

N/A

N/A

N/A

N/A

N/A

+
+
+

Security

+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Function

Description

Nvidia Spectrum

Dell-SONiC

Arista EOS

EdgeCore-SONiC

Equinix Metal

PhoenixNAP

Network ACLs

Centralized Network Access Control Lists.

Nov/2024

Dec/2024

N/A

N/A

Managed Device Profiling

Managed switch & SoftGate protection from unwanted access, push administrative and system settings (NTP, DNS, timezone, etc.)

Nov/2024

Dec/2024

N/A

N/A

Audit Logs

Log all controller access and changes.

Nov/2024

N/A

N/A

+
+
+

Administration

+ +++++ + + + + + + + + + + + + + + +

Function

Description

Globally

Role Based Access Control

Who can view and edit which aspects of the system.

Multi-Tenancy

Network resource delegation to tenants.

+
+
+

Management Interfaces

+ +++++ + + + + + + + + + + + + + + + + + + +

Function

Description

Globally

Web Console

Manage through intuitive web interface.

RestAPI

Integrate your other systems or your customer-facing portal with Netris consuming RestAPIs.

IaC: Terraform

Manage your infrastructure as a code using Terraform.

+
+
+

Hypervisor/Worker node specific functionality

+ ++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Function

Description

Kubernetes

Vmware

Apache Cloud Stack

OpenStack

Harvester

Proxmox

L4 Load Balancer

Layer-4 container or vm/server load balancer with health checks.

✔ (native & automatic)

✔ (need to specify backend IPs)

Dec/2024

✔ (need to specify backend IPs)

✔ (need to specify backend IPs)

✔ (need to specify backend IPs)

VPC to internal routing peering

Automatically route internal networks into VPC routing table (allow containers communicate with VMs).

N/A

Dec/2024

Dec/2024

TBD

TBD

Automatic VXLAN/VLAN

Automatically provision VXLAN/VLAN on switch fabric and include appropriate switch ports when virtual network is created in the hypervisor.

TBD

Dec/2024

TBD

TBD

HBN Host-based networking.

Terminate VTEPs on the hypervisor host. Scale beyond VLAN limits

Dec/2024

TBD

Dec/2024

Dec/2024

TBD

TBD

HBN on DPU

Host-based networking. Terminate VTEPs on the hypervisor host DPU. Scale beyond VLAN limits with accelerated performance

2025

TBD

2025

2025

TBD

TBD

+
+
+
+

SoftGate Data Plane Variations

+

SoftGate is Netris data plane for Internet Gateway, NAT Gateway, Network Access Control, Elastic Load Balancer, and Site-to-Site VPN functions.

+ +++++++++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Flavor

Common Use Case

Availability

Tenancy/VPC

Handoff

Packet Forwarding

HA & Scalability

Ethernet Environment

NIC

CPU

RAM

Disk

Performance (w/ 100 NAT rules)

SoftGate

Bare metal cloud site, Edge site, Remote office.

Single

VLAN

Linux w/ Netris optimizations

Active/Standby - 2 nodes

Dot1q: Equinix Metal, PhoenixNAP, pre-configured VLAN-range on any Ethernet switches.

Any

Intel or AMD

16-64GB

300GB

Dual Gold 6336Y (48c x 2.3GHz) - 11Gbps / 1.8Mpps

SoftGate PRO

Private Cloud, Public Cloud Border Gateway, Enterprise Cloud, Vmware NSX alternative.

Single

VLAN

Netris DPDK

Active/Standby - 2 nodes

Netris Switch-Fabric

Nvidia Connect-X 5, 6 100Gbe

Intel XEON (required for DPDK)

128GB

300GB

Intel XEON Platinum 20+ cores - 100Gbps / 25Mpps

SoftGate HS (HyperScale)

Scalable GPU & CPU Cloud Services Provider.

Multi

VXLAN

Linux w/ Netris optimizations

Active/Active - Horizontally scalable

Netris Switch-Fabric

Any OK. Nvidia Connect-X is recommended

Intel or AMD

128-256GB

300GB

Dual Platinum 8352Y (64c x 2.2GHz) - 22Gbps / 3.5 Mpps

SoftGate HS PRO

Scalable GPU & CPU Cloud Services Provider.

2025/Q2

Multi

VXLAN

Netris

Active/Active - Horizontally scalable

Netris Switch-Fabric

Nvidia Connect-X 5, 6, 7

Intel, AMD (TBD)

256GB+

300GB

TBD

+
+
+

Netris and NOS versions compatibility matrix

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Netris Version

Switch & OS

Bare Metal Cloud

SoftGate OS

Availability

4.4.0

Nvidia Cumulus 5.11, Dell SONiC 4.4, EdgeCore SONiC 202211-331

Equinix Metal, PhoenixNAP BMC

SoftGate HS: Ubuntu 24.04, SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04

Dec/2024

4.3.0

Nvidia Cumulus 5.9, Dell SONiC 4.1, EdgeCore SONiC 12.3

Equinix Metal, PhoenixNAP BMC

SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04 (non-pro)

4.2.0

Nvidia Cumulus 5.7, Dell SONiC 4.1, EdgeCore SONiC 12.3

Equinix Metal, PhoenixNAP BMC

SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04

4.1.1

Nvidia Cumulus 5.7, EdgeCore SONiC 12.3

Equinix Metal, PhoenixNAP BMC

SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04

4.0.0

Nvidia Cumulus 5.7, EdgeCore SONiC 12.3

Equinix Metal, PhoenixNAP BMC

SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04

3.5.0

Nvidia Cumulus 5.7, EdgeCore SONiC 12.3

Equinix Metal, PhoenixNAP BMC

SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04

3.4.1

Nvidia Cumulus 5.7, EdgeCore SONiC 12.3

Equinix Metal, PhoenixNAP BMC

SoftGate Pro: Ubuntu 20.04, SoftGate: Ubuntu 22.04

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/supported-switch-hardware.html b/en/latest/supported-switch-hardware.html new file mode 100644 index 0000000000..df6512a06d --- /dev/null +++ b/en/latest/supported-switch-hardware.html @@ -0,0 +1,1386 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Hardware Requirements — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Hardware Requirements
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Hardware Requirements

+
+

Netris Controller

+

We recommend two Ubuntu 24.04 servers with the below specs. Netris controller can also run on other Linux OSes or on a shared Kubernetes cluster.

+ +++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Use Case

CPU Cores

RAM

SSD/NVME

Network

Leaf/Spine 1-30 switches

4

32 GB

300 GB

2x 1GbE+ NIC

Leaf/Spine 30-100 switches

8

32 GB

600 GB

2x 1GbE+ NIC

Leaf/Spine 100-300 switches

16

64 GB

1 TB

2x 10GbE+ NIC

Leaf/Spine 300+ switches

32

128 GB

2 TB

2x 10GbE+ NIC

Spectrum-X 1-8 SUs

16

64 GB

1 TB

2x 10GbE+ NIC

Spectrum-X 16-32 SUs

32

128 GB

2 TB

2x 10GbE+ NIC

Spectrum-X 32+ SUs

64

256 GB

10 TB

2x 10GbE+ NIC

+
+
+

Netris SoftGate HS (The Horizontally Scalable version)

+

A minimum of 4 dedicated servers are required for an HA (highly available) active-active SoftGate HS cluster. Two SoftGates will forward stateful traffic (SNAT), and two others will forward the stateless traffic (DNAT, 1:1 NAT, Layer-4 Load Balancing, etc.) Each group (stateful and stateless) can be scaled horizontally by deploying more servers as CPU & RAM utilization necessitates.

+

Server specs:

+ +++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Minimum

Recommended

CPU (Modern Intel/AMD X86)

16 Cores

32 Cores

RAM

128 GB

256 GB

NIC prod

2x 10GbE

2x 25GbE

NIC OOB

1x 1GbE

1x 1GbE

Disk

300 GB

300 GB

OS

Ubuntu 24.04

Ubuntu 24.04

+
+
+
+

Supported Switch Hardware

+
+

Nvidia

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Manufacturer

Model

ASIC

Ports

NOS

Caveats

Supported

Nvidia

SN2010

Spectrum

18 x SFP28 25GbE + 4 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN2100

Spectrum

16 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN2201

Spectrum

48 x RJ45 + 4 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN2410

Spectrum

48 x SFP28 25GbE + 8 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN2700

Spectrum

32 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN3420

Spectrum 2

48 x SFP28 25GbE + 12 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN3700C

Spectrum 2

32 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN3700

Spectrum 2

32 x QSFP56 200GbE

Cumulus Linux

Nvidia

SN4410

Spectrum 3

24 x QSFP28-DD 100G + 8 x QSFP-DD 400GbE

Cumulus Linux

Nvidia

SN4600C

Spectrum 3

64 x QSFP28 100GbE

Cumulus Linux

Nvidia

SN4600

Spectrum 3

64 QSFP56 200GbE

Cumulus Linux

Nvidia

SN4700

Spectrum 3

32 x QSFP-DD 400GbE

Cumulus Linux

Nvidia

SN5400

Spectrum 4

64 x QSFP-DD 400GbE + 2 x SFP28 25GbE

Cumulus Linux

Nvidia

SN5600

Spectrum 4

64 x OSFP 800GbE + 1 x SFP28 25GbE

Cumulus Linux

+
+
+

Dell

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Manufacturer

Model

ASIC

Ports

NOS

Caveats

Supported

Dell

PowerSwitch S Series S5212F-ON

Broadcom Trident III

12 x 25G SFP28 + 3 x 100GbE QSFP28

Dell-SONiC

Dell

PowerSwitch S Series S5224F-ON

Broadcom Trident III

24 x 25G SFP28 + 4 x 100GbE QSFP28

Dell-SONiC

Dell

PowerSwitch S Series S5232F-ON

Broadcom Trident III

32 x 100GbE QSFP28

Dell-SONiC

Dell

PowerSwitch S Series S5248F-ON

Broadcom Trident III

48 x 25G SFP28 + 4 x 100GbE QSFP28 + 2 x 200GbE QSFP28

Dell-SONiC

Dell

PowerSwitch S Series S5296F-ON

Broadcom Trident III

96 x 25G SFP28 + 8 x 100GbE QSFP28

Dell-SONiC

Dell

PowerSwitch S Series S5448F-ON

Broadcom Trident IV

48 x QSFP-DD 100GbE + 8 x QSFP-DD 400GbE

Dell-SONiC

Dell

PowerSwitch Z Series Z9664F-ON

Broadcom Tomahawk IV

64 x QSFP-DD 400GbE

Dell-SONiC

Dell

PowerSwitch Z Series Z9432F-ON

Broadcom Trident IV

32 x QSFP-DD 400GbE

Dell-SONiC

+
+
+

EdgeCore

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Manufacturer

Model

ASIC

Ports

NOS

Caveats

Supported

EdgeCore

DCS201 (AS5835-54X)

Broadcom Trident III

48 x 10G SFP+ + 6 x 100G QSFP28

EC-SONiC

EdgeCore

DCS202 (AS5835-54T)

Broadcom Trident III

48 x 10G RJ-45 + 6 x 100G QSFP28

EC-SONiC

EdgeCore

DCS203 (AS7326-56X)

Broadcom Trident III

48 x 25G SFP28 + 8 x 100G QSFP28+ 2 x 10G

EC-SONiC

EdgeCore

AS7726-32X

Broadcom Trident III

32 x 100G QSFP28 + 2 x 10G SFP+

EC-SONiC

EdgeCore

DCS510 (AS9716-32D)

Broadcom Tomahawk 3

32 x 400G QSFP-DD

EC-SONiC

EdgeCore

DCS511 (AS9737-32DB)

Broadcom Tomahawk 4

32 x 400G QSFP56-DD

EC-SONiC

EdgeCore

AIS800-64O

Broadcom Tomahawk 5

64 x OSFP800

EC-SONiC

+
+
+

Arista

+ +++++++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Manufacturer

Model

ASIC

Ports

NOS

Caveats

Supported

Arista

7020R

Qumran

24 x 10G + 2 QSFP100; 32 x 10G + 2 QSFP100; 48 x 100/1000Mb + 6 SFP+; 48 x 100/1000Mb + 6 SFP+

EOS

Dec/2024

Arista

7050X3

Broadcom Trident III

32 x QSFP100; 48 x SFP25 + 12 x QSFP100; 48 x SFP25 + 8 x QSFP100; 48 x 10G-T + 8 x QSFP100

EOS

Dec/2024

Arista

7050X4

Trident-4

32 QSFP-DD 400G + 2SFP+; 32 OSFP 400G + 2SFP+; 48 SFP-DD 100G + 8 QSFP-DD 400G; 48 DSFP 100G + 8 QSFPDD 400G; 24 QSFP56 200G + 8 QSFPDD 400G + 2SFP+; 48 QSFP28 + 8 QSFP-DD 400G + 2SFP+

EOS

Dec/2024

Arista

7060X4

Trident-4

32 x QSFP-DD 800G + 2 x SFP+; 32 x QSFP-DD 800G + 2 x SFP+; 32 x OSFP 800G + 2 x SFP+; 64 x QSFP-DD 400G, 2 x SFP+; 32 x QSFP-DD + 1x SFP+; 56x QSFP100, 8 x QSFP-DD 400G + 1x SFP+

EOS

Dec/2024

Arista

7060X5

Tomahawk 4

32 x QSFP-DD 800G + 2x SFP+; 32 x QSFP-DD 800G + 2x SFP+; 32 x OSFP 800G + 2x SFP+; 64 x QSFP-DD 400G + 2x SFP+; : 32 x QSFP-DD + 1 x SFP+; 56x QSFP100, 8 x QSFP-DD 400G, 1x SFP+

EOS

Dec/2024

Arista

7280R3A

Jericho2

144 x 100G or 36 x 400G

EOS

Dec/2024

Arista

7280R3

Jericho2

24 x 400G; 96 x 100G; 25G + 8 x 100G

EOS

Dec/2024

Arista

7358X4

Trident-4

128 x QSFP or 32 x OSFP / QSFP-DD

EOS

Dec/2024

Arista

7358X4

Trident-4

128 x QSFP or 32 x OSFP / QSFP-DD

EOS

Dec/2024

Arista

7368X4

Tomahawk 3

128 x 100G or 32 x 400G

EOS

Dec/2024

Arista

7300R3

Trident-4

256 wire-speed 40GbE ports

EOS

Dec/2024

Arista

7500R3

Jericho, Jericho2

Up to 288 wire-speed 400G ports

EOS

Dec/2024

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/switch-agent-installation.html b/en/latest/switch-agent-installation.html new file mode 100644 index 0000000000..c173967473 --- /dev/null +++ b/en/latest/switch-agent-installation.html @@ -0,0 +1,865 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Network Switch Initial Setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Network Switch Initial Setup
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/switch-ports.html b/en/latest/switch-ports.html new file mode 100644 index 0000000000..9fd05e560b --- /dev/null +++ b/en/latest/switch-ports.html @@ -0,0 +1,879 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Switch Ports — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Switch Ports

+

Switch ports can be directly managed in the Switch Port UI section. Both physical and virtual ports (extended, aggregate, etc…) will appear in this section once they have been added to inventory. The Netris Controller will automatically sync the list of available ports that appear on each device.

+

The following options are available for editing on each port:

+
    +
  • Description - Description of the port.

  • +
  • Tenant - Tenant to whom the port is assigned, by default it is the owner tenant of the device to whom the port belongs to.

  • +
  • Breakout - Available only for physical switch ports, used to split physical ports into multiple physical ports. When there is a need to use other supported option supported by switch not shown in the dropdown list user must set breakout to “Manual” and configure breakout manually on the switch. For certain platforms some ports need to be disabled to support breakout into other ports, for that option use “Disable” mode of breakout. For Cumulus, after configuration, user must manually restart switchd daemon on the switch via command “systemctl restart switchd”.

  • +
  • MTU - Maximum transmission unit of the port.

  • +
  • Autoneg - Toggle autonegotiation. Available only for physical ports.

  • +
  • Speed - Toggle speed. Available only for physical ports.

  • +
  • Duplex - Toggle duplex. Available only for physical ports.

  • +
  • Extension - Create extension ports. Available for physical and aggregate ports.

  • +
  • Extension Name - Name for new extension.

  • +
  • VLAN Range - VLAN id range for new extension port.

  • +
+_images/edit-port.png +

Example: Edit physical port

+

Quick action menu provides following actions for ports (note that Bulk Action also available for multiple ports:

+

Edit - Edit the port. +Admin UP/Down - Toggle admin status of the port. +Add to V-net - Add selected port(s) to a V-net.

+_images/quick-action-ports.png +

Add to LAG - Add selected ports into a LAG.

+_images/add-to-lag-port.png +

Free Up Port - Detach port from all resources.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/terraform-integration.html b/en/latest/terraform-integration.html new file mode 100644 index 0000000000..52ebdbf7c7 --- /dev/null +++ b/en/latest/terraform-integration.html @@ -0,0 +1,1140 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Terraform: Netris provider — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Terraform: Netris provider
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Terraform: Netris provider

+

Use Netris provider to interact with the many resources supported by Netris. You must configure the provider with the proper credentials before you can use it. +To learn the basics of Terraform using this provider, follow the hands-on get started tutorials on HashiCorp’s Learn platform.

+_images/diagrams_terraform.png +

When you make changes in the Terraform files and apply them, Terraform automatically decides which part of your configuration is already deployed into Netris controller and what should be added or removed.

+ +
+

Install Terraform

+

Download and install the Terraform

+
+
+

Create a directory for Terraform files

+
    +
  1. Create a directory with any name, for example, netris-terraform. It stores the configuration files and saved states for Terraform and your infrastructure.

  2. +
  3. Create a configuration file with the .tf extension in this directory, such as main.tf.

  4. +
+
+
+

Configure a provider

+
    +
  1. At the beginning of the configuration file, specify the provider settings.

  2. +
+
terraform {
+  required_providers {
+    netris = {
+      source  = "netrisai/netris"
+      version = ">= 2.0.0"
+    }
+  }
+}
+
+provider "netris" {
+  address = "<controller address>"
+  login = "<controller login>"
+  password = "<controller password>"
+}
+
+
+

Specify the provider required arguments:

+
    +
  • address - This is your Netris-Controller address (http://example.com). This can also be specified with the NETRIS_ADDRESS environment variable.

  • +
  • login - This is your Netris-Controller login. This can also be specified with the NETRIS_LOGIN environment variable.

  • +
  • password - This is your Netris-Controller password. This can also be specified with the NETRIS_PASSWORD environment variable.

  • +
+
    +
  1. Execute the command terraform init in the folder with the configuration file. This command initializes the providers specified in the configuration files and lets you work with the provider resources and data sources.

  2. +
+
+
+

Prepare an infrastructure plan

+

By using Netris Terraform Provider, you can create all kinds of resources, such as Sites, IPAMs, Topology, Inventory, etc. +To create a resource, specify a set of required and optional parameters that define the resource properties. Such resource descriptions make up an infrastructure plan.

+

Infrastructure provisioning in Netris starts with Site resources. The Netris-Controller comes with the initial site Default. You can use it in your Terraform configuration files by getting its ID with the Terraform Data source element.

+

Let’s create a separate file for site resource, and get its ID via Terraform Data source element.

+
cat << EOF > site.tf
+data "netris_site" "default" {
+  name = "Default"
+}
+EOF
+
+
+

Or, you can create a new Site resource, here is the detailed documentation with examples.

+

Now, when we’re clear on Site resource usage, let’s define our IPAM. There are two types of IPAM resources in the Netris-Controller it’s Allocation and Subnet. +IPAM resources only require tenantid field, let’s get our default Admin tenant ID with the Data source element.

+
cat << EOF > tenant.tf
+data "netris_tenant" "admin"{
+  name = "Admin"
+}
+EOF
+
+
+

Then, when we have the tenantid, we can create IPAM resources.

+
cat << EOF > ipam.tf
+resource "netris_allocation" "my-allocation-mgmt" {
+  name = "my-allocation-mgmt"
+  prefix = "192.0.2.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_allocation" "my-allocation-loopback" {
+  name = "my-allocation-loopback"
+  prefix = "198.51.100.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_allocation" "my-allocation-common" {
+  name = "my-allocation-common"
+  prefix = "203.0.113.0/24"
+  tenantid = data.netris_tenant.admin.id
+}
+
+resource "netris_subnet" "my-subnet-mgmt" {
+  name = "my-subnet-mgmt"
+  prefix = "192.0.2.0/24"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "management"
+  defaultgateway = "192.0.2.254"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-mgmt,
+  ]
+}
+
+resource "netris_subnet" "my-subnet-loopback" {
+  name = "my-subnet-loopback"
+  prefix = "198.51.100.0/24"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "loopback"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-loopback,
+  ]
+}
+
+resource "netris_subnet" "my-subnet-common" {
+  name = "my-subnet-common"
+  prefix = "203.0.113.0/25"
+  tenantid = data.netris_tenant.admin.id
+  purpose = "common"
+  siteids = [data.netris_site.default.id]
+  depends_on = [
+    netris_allocation.my-allocation-common,
+  ]
+}
+EOF
+
+
+

With the command above, we’ve defined 6 resources, 3 of the type of Allocation, 3 of the type of Subnet, each Subnet resource has a different purpose. +For more details, get familiar with the IPAM docs.

+

Now, when we have all the required resources let’s define our Inventory. +We’re going to create 1 SoftGate, 1 switch and connect them with a link.

+
cat << EOF > inventory.tf
+resource "netris_softgate" "my-softgate" {
+  name = "my-softgate"
+  tenantid = data.netris_tenant.admin.id
+  siteid = data.netris_site.default.id
+  description = "Softgate 1"
+  mainip = "auto"
+  mgmtip = "auto"
+  depends_on = [
+    netris_subnet.my-subnet-mgmt,
+    netris_subnet.my-subnet-loopback,
+  ]
+}
+
+resource "netris_switch" "my-switch" {
+  name = "my-switch"
+  tenantid = data.netris_tenant.admin.id
+  siteid = data.netris_site.default.id
+  description = "Switch 01"
+  nos = "cumulus_linux"
+  asnumber = "auto"
+  mainip = "auto"
+  mgmtip = "auto"
+  portcount = 16
+  depends_on = [
+    netris_subnet.my-subnet-mgmt,
+    netris_subnet.my-subnet-loopback,
+  ]
+}
+
+resource "netris_link" "sg-to-sw" {
+  ports = [
+    "swp1@my-softgate",
+    "swp16@my-switch"
+  ]
+  depends_on = [
+    netris_softgate.my-softgate,
+    netris_switch.my-switch,
+  ]
+}
+EOF
+
+
+

Next, let’s define a local L3 network for our servers, suppose we want to connect 3 servers to our switch first 3 ports

+
cat << EOF > vnet.tf
+resource "netris_vnet" "my-vnet" {
+  name = "my-vnet"
+  tenantid = data.netris_tenant.admin.id
+  state = "active"
+  sites{
+    id = data.netris_site.default.id
+    gateways {
+      prefix = "203.0.113.1/25"
+    }
+    ports {
+      name = "swp1@my-switch"
+      vlanid = 1050
+    }
+    ports {
+      name = "swp2@my-switch"
+      vlanid = 1050
+    }
+    ports {
+      name = "swp3@my-switch"
+    }
+  }
+  depends_on = [
+    netris_switch.my-switch,
+    netris_subnet.my-subnet-common,
+  ]
+}
+EOF
+
+
+

And finally, we have to provide internet connectivity to our fabric, for that we’ll define BGP resource. Suppose we’re going to connect our ISP cable to the 10th port of our switch, and want to establish the BGP session on our Softgate.

+
cat << EOF > bgp.tf
+data "netris_port" "swp10_my_switch"{
+  name = "swp10@my-switch"
+  depends_on = [netris_switch.my-switch]
+}
+
+resource "netris_bgp" "my-bgp" {
+  name = "my-bgp"
+  siteid = data.netris_site.default.id
+  hardware = "my-softgate"
+  neighboras = 23456
+  portid = data.netris_port.swp10_my_switch.id
+  vlanid = 3000
+  localip = "172.16.0.2/30"
+  remoteip = "172.16.0.1/30"
+  description = "My First BGP"
+  prefixlistinbound = ["deny 127.0.0.0/8 le 32", "permit 0.0.0.0/0 le 24"]
+  prefixlistoutbound = ["permit 192.0.2.0/24", "permit 198.51.100.0/24 le 25", "permit 203.0.113.0/24 le 26"]
+  depends_on = [netris_link.sg-to-sw]
+}
+EOF
+
+
+
+

Note

+

For more information about all resources, how to create and manage them in Terraform, see the provider’s documentation.

+
+

Now, when we’ve done with the configuration files, let’s check whether they are valid

+
terraform validate
+
+
+

If the configuration is valid, the following message is returned:

+
Success! The configuration is valid.
+
+
+
+
+

Create resources

+
    +
  1. After preparing and checking the configuration, run the command:

  2. +
+
terraform plan
+
+
+

The terminal will display a list of resources with parameters. This is a test step. No resources are created. If there are errors in the configuration, Terraform points them out.

+
    +
  1. To create resources, run the command:

  2. +
+
terraform apply
+
+
+
    +
  1. Confirm the resource creation: type yes in the terminal and press Enter.

  2. +
+

Terraform will create all the required resources and the terminal will display the progress. After creation, you can check resource availability and their settings in the Netris-Controller UI.

+
+
+

Delete resources

+
    +
  1. To delete resources created using Terraform:

  2. +
+

Run the command:

+
terraform destroy
+
+
+

After the command is executed, the terminal will display a list of resources to be deleted.

+
    +
  1. Type yes to confirm their deletion and press Enter.

  2. +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/topology-management.html b/en/latest/topology-management.html new file mode 100644 index 0000000000..ca42e13f22 --- /dev/null +++ b/en/latest/topology-management.html @@ -0,0 +1,1053 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Inventory — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Inventory

+

The Inventory section allows you to add/edit/delete network switches and SoftGates (VPC gateways). Initial setup of a Netris managed network is a three step process:

+
    +
  1. Create Inventory Profiles.

  2. +
  3. Adding Switches.

  4. +
  5. Adding Softgates.

  6. +
+
+

Inventory Profiles

+

Inventory profiles allow security hardening of inventory devices. By default all traffic flow destined to switch/SoftGate is allowed. As soon as the inventory profile is attached to a device it denies all traffic destined to the device except Netris-defined and user-defined custom flows. Automatically allowed flows include:

+
    +
  • SSH from user defined subnets

  • +
  • NTP from user defined ntp services

  • +
  • DNS from user defined DNS servers

  • +
  • Custom user defined rules

  • +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + +
Inventory Profile Fields

Name

Profile name

Description

Free text description

Allow SSH from IPv4

List of IPv4 subnets allowed to ssh (one address per line)

Allow SSH from IPv6

List of IPv6 subnets allowed to ssh (one address per line)

Timezone

Devices using this inventory profile will adjust their system time to the selected timezone.

NTP servers

List of domain names or IP addresses of NTP servers (one address per line). You can use your Netris Controller address as an NTP server for your switches and SoftGate.

DNS servers

List of IP addresses of DNS servers (one address per line). You can use your Netris Controller address as a DNS server for your switches and SoftGate.

+

Example: This example Inventory profile is used to provide NTP and DNS services to the switches (common setup). A custom rule is created to allow UDP connections to the port 161.

+_images/inventory_profile_custom.png +
+
+

Adding Switches

+

Every switch needs to be added to the Netris Controller inventory. You can add new devices with the following process:

+
    +
  1. Navigate to Network → Inventory

  2. +
  3. Click the Add button

  4. +
  5. Fill in the fields as described below

  6. +
  7. Click the Add button

  8. +
+
+
+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Add Inventory Fields - Switch

Name

Name of the device

Owner Tenant

Owner tenant of the device (usually Admin)

Description

Description of the device

Type

Select Switch

NOS

Operating system of the device; applicable to switches only

Site

Site where the devices reside

AS Number

Private AS number of the device; applicable to switches only; recommended to be assigned automatically

Profile

Inventory profile for the current device. Profiles are used for the security hardening of the devices

Main IP address

Main loopback IP address for the device. A subnet with purpose set to loopback needs to be defined in IPAM for the current site

Management IP address

Management IP address for the device. A subnet with purpose set to Management needs to be defined in IPAM for the current site

MAC address

MAC address of the device; applicable to switches only (reserved for future use)

Preliminary port count

Used for definition of topology. When the device registers with the controller the actual ports will synchronize with inventory

Add Link

Used for describing physical connections between switches and SoftGates

+
+

Example: Add a new Switch.

+
+
_images/add-new-hardware.png +
+
+

Note

+

Repeat this process to define all your switches.

+
+
+
+

Adding SoftGates

+

Every SoftGate node needs to be added to the Netris Controller inventory. To add a SoftGate node:

+
    +
  1. Navigate to Network → Topology

  2. +
  3. Click Add

  4. +
  5. Fill in the fields as described below

  6. +
  7. Click the Add button

  8. +
+ + ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Add Inventory Fields - SoftGate

Name

Descriptive name

Owner Tenant

Tenant(typically Admin); who administers this node

Description

Free text description

Hardware Type

Select SoftGate

Site

The data center where the current SoftGate node belongs.

Profile

Inventory Profile describing the timezone; DNS; NTP; and Security features

IP Address

IPv4 address for the loopback interface

Management IP address

IPv4 address for the out of band management interface

Add Links

Used for describing physical connections between switches and SoftGates

+

Example: Adding a SoftGate Node to Topology.

+_images/add-softgate.png +
+
+

Viewing Inventory

+

Inventory Listing provides information about network nodes known to Netris.

+

Heartbeat - Indicates whether the node can communicate with Netris Controller. +Health - Indicates the health status of the node.

+
+
_images/inventory-listing.png +
+
+

Note

+

You can also add new devices in the Topology view.

+
+
+
+
+

Topology Manager

+

The topology manager is for describing and monitoring the desired network topology. Netris Switch Agents software will automatically configure the underlying network devices according to this topology and will watch against potential failures.

+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/try-learn/index.html b/en/latest/try-learn/index.html new file mode 100644 index 0000000000..91389148b4 --- /dev/null +++ b/en/latest/try-learn/index.html @@ -0,0 +1,880 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Try & Learn — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/try-learn/netris-cloudsim.html b/en/latest/try-learn/netris-cloudsim.html new file mode 100644 index 0000000000..b62be24141 --- /dev/null +++ b/en/latest/try-learn/netris-cloudsim.html @@ -0,0 +1,859 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Infrastructure Simulation Platform — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Infrastructure Simulation Platform

+

Netris infrastructure simulation platform was initially created by and for the Netris R&D team for internal use. Netris Infrastructure Simulation Platform is also available to customers and partners for testing and educational purposes per request. Each request is subject to approval by the Netris team.

+

Please use the below link to submit a Netris trial request. +https://www.netris.io/try/

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/try-learn/nvidia-spectrum-x-scenario.html b/en/latest/try-learn/nvidia-spectrum-x-scenario.html new file mode 100644 index 0000000000..0e8790eb26 --- /dev/null +++ b/en/latest/try-learn/nvidia-spectrum-x-scenario.html @@ -0,0 +1,1104 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Lab Scenario: GPU-as-a-Service network with NVIDIA Spectrum-X architecture — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Lab Scenario: GPU-as-a-Service network with NVIDIA Spectrum-X architecture

+
+

Initialize the Netris controller

+

Start with a blank Netris Controller. SSH to the Netris controller server and cd /home/ubuntu/netris-init/netris-spectrum-x-init.

+

Optionally Edit terraform.tfvars file to set cluster scale parameters.

+

Below, we describe the role of a few parameters that directly define the scale. The description of the rest of the parameters is available in the terraform.tfvars file itself. For the purpose of this try & learn scenario, there is no need to change the other parameters.

+

The east-west switch fabric is responsible for high performance data transmission between GPU servers. It rail-optimized design allows to non-blocking max-rate data transmission between any GPUs on the network. You only need to define the number of GPU servers in the terraform.tfvars file. When you execute the initialization module, it will automatically calculate the proper number of links and will generate the rail-optimized blueprint in the Netris controller according to the NVIDIA Spectrum-X guidelines.

+
    +
  • Define gpu-server-count using increments of 32 (1 SU = 32 servers, 2 SUs = 64 servers, etc.)

  • +
+

The north-south switch fabric is responsible for everything else - for connectivity from the outside, to manage the GPU nodes and operate workloads. OOB management switches are responsible for out-of-band management of the network switches and GPU servers. In production, OOB management is also used for PXE booting the GPU servers. In this simulation scenario, GPU servers will be booted by means of the Netris infrastructure simulation platform for your conveninece of teasting and learning.

+
    +
  • Define leaf-count - the rule of thumb is that at least 1/4th of the number of SUs - so 4 leaf switches can handle up to 4 SUs

  • +
  • Define oob-leaf-count - Should be equal to the number of SUs.

  • +
  • Define spine-count - Typically 2, although other values are welcome.

  • +
+

Save the changes and exit.

+

Execute tofu-apply or tofu-destroy to insert/clean up relevant declarations into the Netris controller.

+

Navigate to the Netris controller in your web browser to see the results.

+
+
+

Start a simulation

+

Check Inventory, IPAM, and Topology sections under Network menu in the Netris controller web console. (In the topology section, you may need to select the right site to see a diagram)

+

Go back to the SSH session and cd to /home/ubuntu/netris-cloudsim

+

Execute pulumi up or pulumi destroy to start/stop a simulation of what’s described in the Netris Controller.

+
+
+

Monitoring Dashboard

+

Once simulation creation is done, go back to the Netris web console and wait up to 5 minutes for the infrastructure to come up. You can monitor the status of the network either from the dashboard (click on the Netris icon in the top left corner) or from the Topology section.

+

Click on Agent Heartbeats donut to see its detailed view on the right. Agent Heartbeats section shows whether Netris agent heartbeat is being received from each Netris-managed host.

+

Once heartbeats are received, automatic configuration will start, as well as automatic monitoring. Netris automatic monitoring provides information about the health of your network, such as Interface up/down status, BGP status, Topology/Wiring errors, RAM/CPU/PSU/Fan status, and else.

+

Click on the Managed HW Health donut to see monitoring check statuses for each Netris-managed node on the right side.

+
+
+

Topology

+

The group of spine and leaf switches at the top part is the east-west network (backend network). The group of spine and leaf switches in the bottom is the north-south network (Tennant Access Network). GPU servers are located in the middle between two switch fabrics. If you zoom in, you can see that eth ports 1-8 of each GPU server are connected to the east-west fabric through rail-optimized design - that’s where high-performance computing traffic runs. Interfaces 9-10 are connected to the leaf switches of north-south fabric, later you will see that interfaces 9 & 10 will be bonded from the GPU server side - that’s where production traffic, storage traffic, dataset management, and workload management traffic runs. Finally, interface 11 is connected to the OOB (out of band) management switch. OOB interfaces are used for PXE booting the GPU nodes. (in the current simulation there’s no PXE booting - the VMs that simulate GPU servers just come alive from an image)

+

Network->Topology The main purpose of the topology section is to define the topology. In this scenario, the topology has been defined by means of the initialization module. However, manual changes can still be done through the web console.

+

When deploying with physical hardware, the procedure would be to wire the switches and servers according to the topology diagram in Netris. During that process, the MAC address of each physical switch should be entered into the Netris controller through Topology, by editing every switch node (only for switches) and entering the actual MAC address. These MAC addresses will be required for binding the physical switches to logical switches in the Netris controller.

+

When running a simulation, like in this scenario there’s no need to enter any MAC addresses. The simulation platform will take care of everything.

+

The Topology section also reflects some monitoring information. Links change their color based on link status and utilization. You can zoom in/out and then right-click on any link to check its details to see traffic statistics and any relevant healthcheck info. Switch and SoftGate nodes show numbers on a red/yellow/green background reflecting the quantity of critical/warning/ok checks per each node.

+

Once your newly created simulation is converged, you will see only 1 check in critical state on every switch - that’s the time synchronization, which takes up to 10 minutes to go green.

+
+
+

SSH to Switches

+

During regular operation, there’s no need to SSH to any switches because configuration and management is fully automated by Netris software.

+

There are rare cases when the administrator needs to SSH to a switch. To do so, Find the switch management IP from Topology or Inventory sections in the Netris web console. SSH from the Netris Controller to the management IP of the switch using ‘cumulus’ username. No password is needed when working with the simulation.

+

Example:

+
ubuntu@test-ctl:~/netris-cloudsim$ ssh cumulus@10.7.0.4
+Debian GNU/Linux 12
+Linux leaf-pod00-su0-r3 6.1.0-cl-1-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.38-4+cl5.9.1u6 (2024-05-13) x86_64
+Last login: Thu Sep 12 16:21:35 2024 from 10.8.0.2
+cumulus@leaf-pod00-su0-r3:mgmt:~$
+
+
+
+
+

SSH to GPU servers

+

GPU servers are connected to the east-west and north-south fabrics. At this point of the lab scenario, we haven’t created any VPC/V-Net/VLAN services to instruct the fabric to provide connectivity to any interfaces of any GPU nodes. However, for the purpose of learning and experimenting, the simulation platform has an additional management network that allows you to connect to GPU servers anytime.

+

SSH from the Netris controller server to a few GPU servers using ‘root’ username and IP addresses starting 192.168.16.2. ( 192.168.16.2 is host 0 in SU0, 192.168.16.3 is host 1 in SU0, etc.)

+

Example:

+
ubuntu@test-ctl:~$ ssh root@192.168.16.2
+Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64)
+
+ * Documentation:  https://help.ubuntu.com
+ * Management:     https://landscape.canonical.com
+ * Support:        https://ubuntu.com/pro
+
+ System information as of Fri Sep 13 01:34:28 UTC 2024
+
+  System load:  0.13              Processes:             95
+  Usage of /:   29.2% of 5.64GB   Users logged in:       0
+  Memory usage: 22%               IPv4 address for ens4: 192.168.16.2
+  Swap usage:   0%
+
+
+Expanded Security Maintenance for Applications is not enabled.
+
+39 updates can be applied immediately.
+12 of these updates are standard security updates.
+To see these additional updates run: apt list --upgradable
+
+Enable ESM Apps to receive additional future security updates.
+See https://ubuntu.com/esm or run: sudo pro status
+
+New release '24.04.1 LTS' available.
+Run 'do-release-upgrade' to upgrade to it.
+
+
+Last login: Thu Sep 12 23:20:36 2024 from 10.8.0.2
+root@hgx-pod00-su0-h00:~#
+
+
+

On the GPU host, you’ll find ./cluster-ping.sh, which is a bash script that helps you execute parallel pings across every east-west and north-south interface towards any GPU node. The script knows the IP addressing scheme used in this scenario, and it only needs the SU number and host number.

+

In the below example, the host pings itself. So, local interface IPs are responding, while the default gateways are not. If you ping another host, you’ll get timeouts on all interfaces.

+
root@hgx-pod00-su0-h00:~# ./cluster-ping.sh 0 0
+Usage: ./cluster-ping.sh <SU> <Host>
+
+Ping from hgx-pod00-su0-h00 to SU:0 host:0
+
+------ East-West Fabric ------
+ping rail0 (172.0.0.0)    : OK
+ping rail1 (172.32.0.0)   : OK
+ping rail2 (172.64.0.0)   : OK
+ping rail3 (172.96.0.0)   : OK
+ping rail4 (172.128.0.0)  : OK
+ping rail5 (172.160.0.0)  : OK
+ping rail6 (172.192.0.0)  : OK
+ping rail7 (172.224.0.0)  : OK
+
+------ North-South Fabric ------
+ping bond0  (192.168.0.1)  : OK
+ping default GW (192.168.7.254)  : Timeout
+
+------ IPMI/BMC ------
+ping eth11 (192.168.8.1)  : OK
+ping default GW (192.168.15.254)  : Timeout
+
+
+root@hgx-pod00-su0-h00:~#
+
+
+
+
+

NHN (Netris host networking plugin)

+

Netris host networking plugin is an optional plugin that runs on a GPU host for automatic configuration of IP addresses, static routes, and DPU parameters. The plugin does not use any management network and does not carry any sensitive information. It’s important for multi-tenant situations because the cloud provider should not have access to the tenant servers – therefore, any host configuration method shall not use any kind of shared management network. Also, the tenant should not be able to access any sensitive information of the cloud provider or other tenants. Netris host networking plugin addresses both issues. The plugin reads the necessary IP and static route information by leveraging LLDP, topology discovery, and custom TLVs.

+

The below example shows how to check if the pluggin is running:

+
root@hgx-pod00-su0-h00:~# systemctl status netris-hnp.service
+● netris-hnp.service - Netris Host Networking Plugin
+     Loaded: loaded (/etc/systemd/system/netris-hnp.service; enabled; vendor preset: enabled)
+     Active: active (running) since Thu 2024-09-12 23:01:22 UTC; 21h ago
+   Main PID: 2906 (netris-hnp)
+      Tasks: 4 (limit: 1102)
+     Memory: 7.7M
+        CPU: 3min 35.913s
+     CGroup: /system.slice/netris-hnp.service
+             └─2906 /opt/netris/bin/netris-hnp
+
+
+

You can also check the running IP addresses and static routes on the GPU server, and if you right-click on the server to switch links in the Network->Topology and check the details, you will see that the actual IP addresses on the GPU servers are aligned with those in the Topology blueprint.

+
+
+

Server Cluster Template

+

Now, when the switch fabric is in an operational state, the underlay is established, it is the time to start defining cloud networking constructs such as VPCs, Subnets, etc., in order to ask the system to provision network access to certain groups of servers.

+

One way to do that would be to navigate to Network->VPC, Network->IPAM, and Services->V-Net sections and create these objects, list switch ports, and then Netris will implement the necessary confogurations.

+

Before that, there is one more important concept that we want you to learn. Server Cluster and Server Cluster Template. +Server Cluster allows the creation of VPC, IPAM, and V-Net objects by listing server names instead of switch ports – this is critical for cloud providers because cloud users don’t want to deal with switch ports.

+

In the web console, navigate to Services->Server Cluster Template - click +Add, give the template some name ‘GPU-Cluster-Template’ or something, and copy/paste the below in the JSON area.

+

The Template is basically telling the system to what server interfaces should be groupped into what V-Nets. Netris will find out the appropriate switch ports by looking up the topology.

+
[
+    {
+        "postfix": "East-West",
+        "type": "l3vpn",
+        "vlan": "untagged",
+        "vlanID": "auto",
+        "serverNics": [
+            "eth1",
+            "eth2",
+            "eth3",
+            "eth4",
+            "eth5",
+            "eth6",
+            "eth7",
+            "eth8"
+        ]
+    },
+    {
+        "postfix": "North-South-in-band-and-storage",
+        "type": "l2vpn",
+        "vlan": "untagged",
+        "vlanID": "auto",
+        "serverNics": [
+            "eth9",
+            "eth10"
+        ],
+        "ipv4Gateway": "192.168.7.254/21"
+    },
+    {
+        "postfix": "OOB-Management",
+        "type": "l2vpn",
+        "vlan": "untagged",
+        "vlanID": "auto",
+        "serverNics": [
+            "eth11"
+        ],
+        "ipv4Gateway": "192.168.15.254/21"
+    }
+]
+
+
+
+
+

Server Cluster

+

Now, navigate to Services->Server Cluster and click +Add. Give the new cluster some name, set Admin to Admin (this is related to Netris internal permissions of who can edit/delete this cluster), set the site to your site (Datacenter-1 is the default name), set VPC to ‘create new’, select the Template (you’ll see the Template created in the last step), and click +Add server and include first 10 servers (from 0 to 9). Click Add.

+

While the Server Cluster is being provisioned, check out what primitive objects have been created in the Netris controller driven by the Server Cluster and Server Cluster Template constructs. Navigate to Network->VPC, and you’ll see a newly created VPC. Navigate to Network->IPAM, then open the VPC filter and make it filter the IPAM by your new VPC, you’ll see some subnets created and assigned to that new VPC. Navigate to Services->V-Net, and you’ll see some 3 V-Nets created, one for the east-west fabric (L3VPN VXLAN), one for north-south (L2VPN VXLAN - this one will have EVPN-MH bonding enabled, you’ll see in next steps), and one for OOB.

+

Go ahead, and create another Server Cluster, and include the next 10 servers - or any other servers. The system won’t let you “double-book” any server in more than one cluster to avoid conflicts.

+
+
+

Checking the connectivity

+

SSH to GPU server host 0 SU 0.

+
ubuntu@test-ctl:~$ ssh root@192.168.16.2
+Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64)
+
+
+

Cluster-ping the neighboring GPU servers. SU0 host 0-9

+
root@hgx-pod00-su0-h00:~# ./cluster-ping.sh 0 9
+Usage: ./cluster-ping.sh <SU> <Host>
+
+Ping from hgx-pod00-su0-h00 to SU:0 host:9
+
+------ East-West Fabric ------
+ping rail0 (172.0.0.18)    : OK
+ping rail1 (172.32.0.18)   : OK
+ping rail2 (172.64.0.18)   : OK
+ping rail3 (172.96.0.18)   : OK
+ping rail4 (172.128.0.18)  : OK
+ping rail5 (172.160.0.18)  : OK
+ping rail6 (172.192.0.18)  : OK
+ping rail7 (172.224.0.18)  : OK
+
+------ North-South Fabric ------
+ping bond0  (192.168.0.10)  : OK
+ping default GW (192.168.7.254)  : OK
+
+------ IPMI/BMC ------
+ping eth11 (192.168.8.10)  : OK
+ping default GW (192.168.15.254)  : OK
+
+
+root@hgx-pod00-su0-h00:~#
+
+
+

Since GPU servers from 0 to 9 are in the same cluster, you should be able to cluster-ping all of them. If you try to cluster-ping other nodes, you will get timeouts because they are not in the same Server Cluster - so the Netris-generated configuration of the switches will contain the access within a single VPC using various configurations throughout the network.

+

You can SSH to GPU server SU0 host 10, which belongs in the second cluster, and cluster-ping its neighbors.

+
ubuntu@test-ctl:~$ ssh root@192.168.16.12
+Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 5.15.0-119-generic x86_64)
+
+
+
+
+

Cleanup the Controller

+

At this point this Netris Try & Learn scenario has been concluded. You may want to clean up the lab to let your colleagues run through the scenario or if you are working on another one. There is no need to clean up if you are about to return the environment to the Netris team – we are going to recycle and reinstall the environment anyway.

+
    +
  1. Delete Server Clusters from the Services->Server Cluster menu.

  2. +
  3. Delete Server Cluster Profile from the Services->Server Cluster Profile menu.

  4. +
  5. SSH to the Netris controller server, cd /home/ubuntu/netris-cloudsim, and execute pulumi destroy to destroy the infrastructure simulation.

  6. +
  7. cd /home/ubuntu/netris-init/netris-spectrum-x-init and execute tofu-destroy to remove the objects from the Netris controller that were created through the initialization module.

  8. +
+

Please let us know your feedback and questions.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/try-learn/using-netris-cloudsim.html b/en/latest/try-learn/using-netris-cloudsim.html new file mode 100644 index 0000000000..90a7e73bc9 --- /dev/null +++ b/en/latest/try-learn/using-netris-cloudsim.html @@ -0,0 +1,891 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Test Controller & Infrastructure Simulation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Test Controller & Infrastructure Simulation

+

This document provides general tips and tricks for using the Netris test controller and Netris infrastructure simulation. Please refer to individual scenarios in this section to experiment around a specific use case.

+

Once your Try & Learn through Netris Infrastructure Simulation platform is approved you will receive credentials for accessing a Netris controller with a trial license hosted in Netris Infrastructure Simulation platform.

+

Controller FQDN: example-ctl.netris.dev +Password: NetrisProvidedPassword

+
+

Web Console

+

Navigate your browser to the Controller FQDN then use +username: netris +password: <NetrisProvidedPassword>

+
+
+

SSH

+

Controller initialization modules and simulation control packages are installed on the Netris controller server.

+

Connect to the controller server using ssh. Username: ubuntu Password: <NetrisProvidedPassword>

+
ssh ubuntu@<Controller FQDN>
+
+
+
+
+

Netris Init Modules

+

Netris init modules are designed to generate Inventory, IPAM, and Topology data based on simple arguments. Most init modules are written using Terraform/HCL.

+

Netris test environment may come with a module that is relevant to your use case. If you can’t find the right module for your use case, please contact your SA (Solutions Architect)

+

Init modules are stored in the /home/ubuntu/netris-init/ directory.

+

Each module is stored in its own subdirectory. To use a module ‘cd’ to the appropriate subdirectory, review terraform.tfvars file, make changes to the arguments if needed, and save.

+

Execute tofu-apply or tofu-destroy in the init module subdirectory to apply/destroy the Netris controller configuration.

+
+
+

Start/Stop a Simulation

+

Once the topology blueprint, Inventory, and IPAM data are created in Netris – basically, Netris controller is describing an infrastructure - you can start its simulation.

+

Go to:

+
cd /home/ubuntu/netris-cloudsim
+
+
+

Execute pulumi up or pulumi destroy to start/stop a simulation of what’s described in the Netris Controller

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/activating-bgp-on-equinix-metal-project.html b/en/latest/tutorials/activating-bgp-on-equinix-metal-project.html new file mode 100644 index 0000000000..d36d19e2ce --- /dev/null +++ b/en/latest/tutorials/activating-bgp-on-equinix-metal-project.html @@ -0,0 +1,867 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Activating BGP on Equinix Metal Project — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Activating BGP on Equinix Metal Project

+

Why use BGP with Equinix Metal? +SoftGate nodes are like border routers to your VPC, they are routing traffic between hosts inside your project and the Internet. We are going to establish 2 BGP sessions between SoftGate nodes and Equinix Metal. So there will be 4 BGP sessions total.

+

We need these BGP sessions for moving further. In the next chapters we are going to request pools of public IP addresses, that Netris will automatically advertise to Equinix Metal, so inbound traffic “knows” how to reach Load Balancer, NAT, and other services that you will use within your VPC.

+../_images/equinix-metal-bgp-diagram.png +

You only need to activate BGP on the Equinix Metal Project. Netris will handle the rest. +In the Equinix Metal web console go to Networking → BGP then click Activate BGP on This Project. (see below screenshots)

+../_images/equinix-metal-activate-bgp.png +

Netris will handle the rest behind the scenes automatically. Netris will enable BGP peering on the Equinix Metal side, Netris will pull the metadata with the BGP info, and will automatically configure FRR (Free Range Routing BGP daemon) on both SoftGate nodes to bring up the BGP sessions up.

+

After a few minutes you should see 4 new BGP sessions in your Netris web console under Net → E-BGP. (example screenshot below).

+../_images/equinix-metal-netris-bgp-up.png +

Now your Netris VPC has established BGP sessions with Equinix Metal Project, and you can proceed to the next step.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html b/en/latest/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html new file mode 100644 index 0000000000..0aea22a79c --- /dev/null +++ b/en/latest/tutorials/adding-netris-softgate-nodes-in-equinix-metal.html @@ -0,0 +1,880 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Provisioning Netris SoftGate nodes in Equinix Metal Project — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Provisioning Netris SoftGate nodes in Equinix Metal Project

+

For SoftGate nodes you can start with two c3.small.x86 or larger servers. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one.

+

Request two servers(c3.small.x86) from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned.

+
    +
  1. At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory

  2. +
+../_images/softgate-nodes-created-in-equinix.png +
    +
  1. When Equinix finishes provisioning of the servers, click on each server name, then click “Tags”, and add a tag “netris-softgate”.

    +
    +

    Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location).

    +
    +
  2. +
+

Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+../_images/softgate-nodes-recognized-in-netris.png +
    +
  1. Provision SoftGate nodes.

  2. +
+

Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there.

+../_images/softgate-one-liner-provisioning.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/softgate-green.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/aws-concept.html b/en/latest/tutorials/aws-concept.html new file mode 100644 index 0000000000..dfd94303b0 --- /dev/null +++ b/en/latest/tutorials/aws-concept.html @@ -0,0 +1,866 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Site Mesh with AWS Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Site Mesh with AWS Overview

+
+

Introduction

+

This guide provides a step-by-step process to set up and configure Netris Softgate in AWS for establishing a site mesh network between the user’s on-premises, AWS, and other cloud environments.

+
+
+

Concept

+

Netris Softgate in AWS is an EC2 instance that runs the Netris software. Therefore, you’ll first need to create an EC2 instance for Netris Softgate and install the Netris software on it. Once that’s done, you’ll need to configure the routes in your AWS VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your AWS VPC to access those destinations through the Netris Softgate EC2 instance.

+../_images/aws-concept-traffic-flows.png +

Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/aws-deploy-softgate.html b/en/latest/tutorials/aws-deploy-softgate.html new file mode 100644 index 0000000000..51331e4768 --- /dev/null +++ b/en/latest/tutorials/aws-deploy-softgate.html @@ -0,0 +1,991 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Deploy a Softgate in AWS — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Deploy a Softgate in AWS

+

As stated in the previous section, the following sequence of actions must be taken in order to proceed: create an EC2 instance, add Softgate into the Netris Controller, install Netris Softgate software on the EC2 instance, and configure routes in AWS VPC. Let us commence with these steps in the specified order.

+
+

Create an EC2 instance

+

Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated EC2. To achieve this, create a security group with the “All traffic” type and “Anywhere” source for both inbound and outbound rules. Afterward, an EC2 instance can be created using the security group above.

+../_images/aws-security-group.png +

To enable connectivity with other Netris sites, it is essential to create the EC2 instance in the desired VPC. Therefore, deploy a new EC2 instance with the Ubuntu 22.04 operating system installed, utilizing an instance type that meets the minimum hardware requirements of 2 virtual CPUs and 4 GB of RAM, such as t2.medium/t3.medium or any other type that satisfies these specifications. It is also recommended to allocate at least 30 GB of drive space.

+../_images/aws-softgate-deployed.png +

After successfully deploying the EC2 instance, it is crucial to take note of its Public IPv4 address. This address will be required in the upcoming step.

+
+
+

Configure Netris Controller

+

Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed.

+
+

Pre-Requisite Steps

+

In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below:

+
    +
  1. Open the Netris Web Console.

  2. +
  3. Navigate to “Net” and select “Sites”.

  4. +
  5. Click on the “+ Add” button.

  6. +
  7. Select “Dot1q Trunk” as the “Switch Fabric”.

  8. +
  9. Input a descriptive name for the site.

  10. +
  11. Specify 65500 in the “Public ASN” field.

  12. +
  13. Click “Add” to create the new site.

  14. +
+../_images/aws-netris-site-create.png +

Subsequently, it is necessary to create a subnet with the “Purpose” of loopback within Netris IPAM. The subnet’s “Prefix” will match the External IP of the AWS EC2 instance, with a netmask length of /32. For instance, if the EC2 instance’s External IP address is 54.176.11.144, then the “Prefix” for the loopback subnet will be 54.176.11.144/32. It’s important to note that an allocation for this “Prefix” must be created prior to creating the loopback subnet. To achieve this, follow the steps below:

+
    +
  1. Go to the “IPAM” section under the “Net” tab.

  2. +
  3. Click on the “+ Add” button located at the top-right corner.

  4. +
  5. Enter the “Prefix” for the new allocation, such as 54.176.11.144/32.

  6. +
  7. Type a descriptive “Name” for the allocation.

  8. +
  9. Select the desired tenant name from the “Tenant” dropdown menu.

  10. +
  11. Click on the “Add” button to create the allocation.

  12. +
+

Once the allocation is created, proceed with the creation of the subnet as follows:

+
    +
  1. Click on the “+ Add” button located at the top-right corner.

  2. +
  3. Enter the “Prefix” for the new subnet, such as 54.176.11.144/32.

  4. +
  5. Type a descriptive “Name” for the subnet.

  6. +
  7. Select the desired tenant name from the “Tenant” dropdown menu.

  8. +
  9. From the “Type” dropdown menu, select “Subnet”.

  10. +
  11. Select “loopback” from the “Purpose” dropdown menu.

  12. +
  13. Choose the appropriate site from the “Sites” dropdown menu.

  14. +
  15. Click on the “Add” button to create the subnet.

  16. +
+../_images/aws-netris-ipam-lo.png +
+
+

Add AWS VPC Subnet into the Netris Controller

+

To register your AWS VPC’s entire CIDR block into Netris IPAM, follow these steps:

+
    +
  1. From the AWS Console, navigate to your VPC and take note of your CIDR blocks.

  2. +
  3. In Netris Controller, go to the “IPAM” section under the “Net” tab.

  4. +
  5. Click the “+ Add” button located at the top-right corner.

  6. +
  7. Enter the VPC CIDR block into the “Prefix” field for the new subnet. For example, if your CIDR block is “172.31.0.0/16”, enter that value.

  8. +
  9. Type a descriptive name for the subnet.

  10. +
  11. From the “Tenant” dropdown menu, select the desired tenant name.

  12. +
  13. From the “Type” dropdown menu, select “Subnet”.

  14. +
  15. Select “inactive” from the “Purpose” dropdown menu.

  16. +
  17. Choose the appropriate site from the “Sites” dropdown menu.

  18. +
  19. Click the “Add” button.

  20. +
+../_images/aws-vpc-cidr-to-netris.png +
+
+

Static route for AWS VPC Subnet in Netris Controller

+

We need to create route entry in Netris. The prefix for the route will be the AWS VPC CIDR block, and the next-hop will be the default gateway of Netris Softgate EC2.

+

Here are the steps to create the static route:

+
    +
  1. Securely log in to the Netris Softgate’s EC2 instance using SSH.

  2. +
  3. Retrieve the default gateway address by typing the command ip route show default.

  4. +
  5. In Netris Controller, go to the “Routes” section under the “Net” tab.

  6. +
  7. Click on the “+ Add” button located at the top-right corner of the screen.

  8. +
  9. Enter your VPC CIDR block in the “Prefix” field.

  10. +
  11. Enter the default gateway of the Netris Softgate EC2 instance in the “Next-Hop” field.

  12. +
  13. Select the appropriate site from the “Site” dropdown menu.

  14. +
  15. Finally, click on the “Add” button to create the static route.

  16. +
+../_images/aws-netris-static-route.png +
+
+

Create the Softgate in the Netris Controller

+

After completing all the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide:

+
    +
  1. Ensure that you have completed all the pre-required steps.

  2. +
  3. Navigate to the “Net” tab in the Netris Controller and select the “Inventory” section.

  4. +
  5. Click on the “+Add” button to create a new Softgate.

  6. +
  7. Provide a descriptive name for the Softgate in the “Name” field.

  8. +
  9. From the “Tenant” dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets.

  10. +
  11. From the “Type” dropdown menu, select “SoftGate”.

  12. +
  13. Choose the appropriate site from the “Site” dropdown menu.

  14. +
  15. For the “Main IP address” field, select “Assign automatically”.

  16. +
  17. For the “Management IP Address” field, select “None”.

  18. +
  19. In the “Description” field, add int=eth0 to specify that Netris should use the softgate’s eth0 interface instead of the default bond0 interface that Netris Softgate Agent looks for.

  20. +
  21. Finally, click on the “Add” button to create the Softgate.

  22. +
+../_images/aws-netris-create-sg.png +
+
+

Netris SoftGate node provisioning

+

After creating a softgate resource in Netris Controller, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting “Install Agent”. Copy the one-line installer command to your clipboard and connect to your EC2 instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it’s done, reboot the server.

+../_images/aws-netris-provision-sg.png +
+
+
+

Configure AWS VPC

+
+

Adding routes

+

To enable specific traffic to be routed to the Netris Softgate EC2 instance in your AWS VPC, you need to modify your VPC’s Route Table. To get started, go to the AWS Console and navigate to your VPC’s Route Table. From there, click the “Edit routes” button to access the routing table, and then click “Add route” to create a new routing entry.

+

In the “Destination” field, enter the subnet CIDR block for the Netris other Sites’ subnets you want to access from this VPC. Next, in the “Target” field, select the “Instance” option and then select the Netris Softgate EC2 instance you previously created. This will ensure that traffic for those subnets is directed to the Softgate instance.

+../_images/aws-vpc-routes-created.png +
+
+

EC2 Source / destination check

+

To allow Netris SoftGate to work properly, it is necessary to disable the ‘Source / destination check’ for the SoftGate’s EC2 Instance. To do so, follow the steps below:

+
    +
  1. Go to the AWS Console and navigate to the EC2 service page.

  2. +
  3. Select the SoftGate EC2 instance.

  4. +
  5. Click on the “Actions” button in the “Networking” section.

  6. +
  7. Select “Change Source / destination check”.

  8. +
  9. Mark the “Stop” checkbox to disable the feature.

  10. +
  11. Click “Save” to apply the changes.

  12. +
+../_images/aws-ec2-stop-fwd-check.png +

By disabling the “Source / destination check”, the SoftGate EC2 instance can receive and forward traffic between the Netris other Sites and AWS VPC subnets.

+
+
+
+

Enabling Site Mesh

+

To enable Site Mesh, in Netris Controller, navigate to the “Net” tab and select “Sites”. Next, click on the three vertical dots (⋮) on the right side of the AWS site and select “Edit”, and then from the “Site Mesh” dropdown menu, select “Hub”. Save the changes. Repeat these steps for all sites that need to have meshed.

+../_images/aws-netris-enable-site-mesh.png +

The Site Mesh status can be viewed by navigating to the “Site Mesh” section under the “Net” tab. This will display the current status of Site Mesh for all Sites.

+../_images/aws-netris-site-mesh-status.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/connecting-fabric-to-isp.html b/en/latest/tutorials/connecting-fabric-to-isp.html new file mode 100644 index 0000000000..74dcca84ab --- /dev/null +++ b/en/latest/tutorials/connecting-fabric-to-isp.html @@ -0,0 +1,862 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Connecting Netris managed fabric to an ISP — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Connecting Netris managed fabric to an ISP

+

BGP is used to connect the Netris managed fabric to an ISP for internet access. To do this, connect a cable from the ISP to your switch port. Then, use the information provided by your ISP to configure a BGP session within the Netris Controller.

+

To create a BGP session go to Network → E-BGP → +Add

+../_images/create_bgp.png +

If everything is correct, State, port and BGP will get green status.

+../_images/bgp_status.png +

Check out advanced BGP configuration here, if you require additional features.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/connecting-servers-fabric.html b/en/latest/tutorials/connecting-servers-fabric.html new file mode 100644 index 0000000000..7460295dae --- /dev/null +++ b/en/latest/tutorials/connecting-servers-fabric.html @@ -0,0 +1,860 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Connecting servers to the Netris managed fabric — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Connecting servers to the Netris managed fabric

+

To connect servers or other endpoints to the switch fabric, you should use V-Nets. V-Net is a virtual networking service that provides Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports throughout the switch fabric. Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and manages the high availability for the default gateway behind the scenes. Simply add the necessary switch ports to a V-Net, and Netris will handle the rest. +Navigate to Services → V-Net → +Add

+../_images/vnet.png +

If you are planning to use Link Aggregation in your setup, please take a look at the options provided by Netris here.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/create-interconnection-to-fabric.html b/en/latest/tutorials/create-interconnection-to-fabric.html new file mode 100644 index 0000000000..bbf9593ddd --- /dev/null +++ b/en/latest/tutorials/create-interconnection-to-fabric.html @@ -0,0 +1,899 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Creating an Interconnection to Equinix Fabric — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Creating an Interconnection to Equinix Fabric
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Creating an Interconnection to Equinix Fabric

+

Once you have successfully set up Netris on Equinix Metal, you may want to use it to connect your private networks from Equinix Fabric back to Metal.

+

Netris can easily manage the routing and other required VPC services for this.

+

Create VLANs in Metal

+

To be able to create an interconnect into Equinix Fabric, first you must create a pair of VLANs for the Primary and Secondary connections.

+

To do this, go to Equinix Console -> IP&Networks -> Layer 2. +Click “Add Vlan”. +Choose the location of your Netris Softgates, then use a couple of VLAN IDs that are unused.

+

In the example below, VLAN 1000 is used for Primary, and VLAN 1001 is used for the Secondary.

+../_images/add-new-vlan-equinix.png +

Request Interconnection

+

Now we are ready to request the Equinix Interconnection on the Equinix Metal side.

+

Go to Equinix Console -> IP&Networks -> Interconnections.

+

Click “Request New Interconnection”.

+

From here, under Fabric VC select “Fabric Billed”.

+

Choose the Location where your Netris Softgates are located.

+

Add an Interconnection name. This can be any name.

+

Choose “Redundant interconnection”.

+

Under “Primary Metal VLAN” select the VLAN created in the previous step, in this example it is VLAN 1000.

+

Under “Secondary Metal VLAN” select the VLAN created in the previous step, in this example it is VLAN 1001.

+

Then, finalize the request by clicking “Submit Interconnection Request”.

+../_images/add-interconnection-request.png +

Once this is accomplished, use the token generated to connect your virtual Network Edge device to Equinix Metal. Setting up a connection on the Fabric side is outside the scope of this document. Once that is complete, the Interconnection will show as Active, and you may move onto creating BGP connections with the Network Edge Virtual Device.

+

Primary BGP Connection to Fabric Network Edge Device

+

Up to this point, a redundant pair of layer 2 connections have been created and connected back to an Equinix Fabric Network Edge device (or some other port with a router on the other end). To be able to utilize the redundancy of both connections, a BGP peering must be established over the two layer 2 VLANs.

+

On the Netris Controller, go to Net -> E-BGP, then click Add.

+

Fill in the following information:

+
Name: Choose any name
+Site: Choose the site of your Equinix Project
+V-Net: None
+BGP Router: Choose the primary Softgate
+VLAN ID: Choose the Primary VLAN ID created when creating the Interconnection in the previous steps
+Neighbor AS: Choose the ASN configured on the Network Edge router
+IP Family: IPv4
+Local IP: Any /30 can be used.  Using a Private IP is recommended.
+Remote IP: Use the IP of the remote side.
+
+
+../_images/add-equinix-bgp-primary.png +

Secondary BGP Connection to Fabric Network Edge Device

+

Now to create the backup/secondary connection.

+

On the Netris Controller, go to Net -> E-BGP, then click Add.

+

Fill in the following information:

+
Name: Choose any name
+Site: Choose the site of your Equinix Project
+V-Net: None
+BGP Router: Choose the secondary Softgate
+VLAN ID: Choose the Secondary VLAN ID created when creating the Interconnection in the previous steps
+Neighbor AS: Choose the ASN configured on the Network Edge router
+IP Family: IPv4
+Local IP: Any /30 can be used.  Using a Private IP is recommended.
+Remote IP: Use the IP of the remote side.
+
+
+

Once completed, the EBGP status shoudl look something like this:

+../_images/equinix-ebgp-links-up.png +
+

Note

+

At this point, you should have both BGP sessions up, but the link may show yellow if the Network Edge device is not advertising any routes back to the Netris Softgates. Once the Network Edge router begins sending routes from other connections, the status should turn green.

+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/enable-services-on-equinix-metal-project.html b/en/latest/tutorials/enable-services-on-equinix-metal-project.html new file mode 100644 index 0000000000..7ed52aaa3a --- /dev/null +++ b/en/latest/tutorials/enable-services-on-equinix-metal-project.html @@ -0,0 +1,890 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling services (NAT, V-Net, Load Balancer, IP pools) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Enabling services (NAT, V-Net, Load Balancer, IP pools)

+

Although bare metal servers in Equinix Metal Project get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities.

+

Both NAT and on-demand Load Balancer services need public IP addresses.

+
+

1) Requesting new Public IP address block

+

Go to Equinix Metal web console and click on Networking → IPs (see the screenshot below)

+

In this example, I’m requesting two IP address blocks, one /30 (4 IPs) for NAT and one /28 (16 IPs) for Load Balancer.

+

It’s important to tag IP blocks as “netris”. This is a signal for Netris Controller that this block is intended for Netris VPC.

+

You can always request more IP address blocks in the future. Also it is possible to request a large block and then use Netris IPAM for crushing it into smaller blocks. You can read more about Netris IPAM in Netris docs.

+../_images/equinix-metal-request-ip-block.png +

Once IP address blocks are provisioned on Equinix Metal Project you should be able to find them automatically replicated in Netris web console under Net → IPAM

+../_images/equinix-metal-netris-ipam-synced.png +

You don’t need to worry about advertising them over BGP, Netris will handle that automatically when that makes sense (associated with any service).

+
+
+

2) Enable on-demand (elastic) Load Balancer

+

To Enable on-demand (elastic) Load Balancer you only need to change the “purpose” field of appropriate IP address block from “common” into “load-balancer”

+

Click on the 3 dots menu (in this example of /28 IP address block), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field.

+../_images/netris-enable-elb.png +

Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform.

+

Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it.

+
+
+

3) Enable V-Net

+

V-Net is a service for virtual private networks. You need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources.

+../_images/netris-create-common-subnets.png +
+
+

4) Enable NAT

+

To enable NAT, you need to repurpose a block of IP addresses for NAT. In the below example I’m repurposing the newly requested /30 subnet for NAT.

+../_images/netris-ipam-nat.png +

Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT. In this example I’m enabling SNAT for the entire 10.0.0.0/8 private network, so basically I just want to ensure that VMs that will get IPs from private networks will get outbound Internet access through NAT. You can always have more granular control either through NAT rule or using Services → ACLs.

+../_images/netris-create-nat-rule.png +

At this point the minimal configuration of Netris VPC networking is done, next chapters will describe how to consume the VPC, how to request resources and services.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/enabling-load-balancing-services.html b/en/latest/tutorials/enabling-load-balancing-services.html new file mode 100644 index 0000000000..86ed183900 --- /dev/null +++ b/en/latest/tutorials/enabling-load-balancing-services.html @@ -0,0 +1,862 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling Load-balancing services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Enabling Load-balancing services

+

L4 Load Balancer is an on-demand (elastic) L4 Load Balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Kubernetes integration

+

Terraform integration

+

Below is a screenshot of requesting a Load Balancer service through the Netris web console. Navigate to Services → L4 Load Balancer → +Add

+../_images/l4lb_create.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/enabling-nat-services.html b/en/latest/tutorials/enabling-nat-services.html new file mode 100644 index 0000000000..066bb668d6 --- /dev/null +++ b/en/latest/tutorials/enabling-nat-services.html @@ -0,0 +1,868 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling NAT services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Enabling NAT services

+

If you utilize private address space for your hosts, you may need a NAT service to enable internet access. +Netris Softgates support SNAT, DNAT and Masquerade features.

+
+

Note

+

Softgate PRO will support Masquerade in the future releases.

+
+

Navigate to Network → NAT → +Add

+

Create a SNAT service to allow connections from your hosts to the Internet.

+../_images/snat_add.png +

Selecting a SNAT pool will allocate the entire pool for this service, preventing the use of IP addresses from the pool for DNAT or other SNAT purposes.

+

Create a DNAT service to allow connections from the Internet to your internal hosts with private IP.

+../_images/dnat_add.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/equinix-metal-api-integration-enablement.html b/en/latest/tutorials/equinix-metal-api-integration-enablement.html new file mode 100644 index 0000000000..7896c4e08a --- /dev/null +++ b/en/latest/tutorials/equinix-metal-api-integration-enablement.html @@ -0,0 +1,894 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Enable Equinix Metal API integration — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Enable Equinix Metal API integration

+

For each Equinix Metal Project+location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “Equinix Metal” from the dropdown menu.

Name

Type a descriptive name for your Equinix Metal Project+location.

Equinix Project ID

Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID.

Equinix Project API key

Create a new Read/Write API key in Equinix Metal portal under Project Settings → API keys → + Add an API Key. Then copy/paste here.

Equinix Location

Select your equinix location from the dropdown menu.

+

Equinix Metal Project ID

+../_images/equinix-metal-project-id.png +

Equinix Metal Project API key

+../_images/equinix-metal-project-api-keys.png +

Netris Create New Site

+../_images/netris-create-equinix-metal-site.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/gcp-concept.html b/en/latest/tutorials/gcp-concept.html new file mode 100644 index 0000000000..0326366b50 --- /dev/null +++ b/en/latest/tutorials/gcp-concept.html @@ -0,0 +1,866 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Site Mesh with GCP Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Site Mesh with GCP Overview

+
+

Introduction

+

This guide provides a step-by-step process to set up and configure Netris Softgate in GCP for establishing a site mesh network between the user’s on-premises, GCP, and other cloud environments.

+
+
+

Concept

+

Netris Softgate in GCP is an VM instance that runs the Netris software. Therefore, you’ll first need to create an VM instance for Netris Softgate and install the Netris software on it. Once that’s done, you’ll need to configure the routes in your GCP VPC for all destination IP subnets that exist in your other environments, such as on-premises or other clouds. This will allow your GCP VPC to access those destinations through the Netris Softgate VM instance.

+../_images/gcp-concept-traffic-flows.png +

Once the routes are configured, you can enable a site mesh between Netris Softgate instances in different environments. Enabling the site mesh allows for secure communication between different environments and enables you to route traffic between the different subnets in a secure and efficient way.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/gcp-deploy-softgate.html b/en/latest/tutorials/gcp-deploy-softgate.html new file mode 100644 index 0000000000..62535209b9 --- /dev/null +++ b/en/latest/tutorials/gcp-deploy-softgate.html @@ -0,0 +1,1011 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Deploy a Softgate in GCP — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Deploy a Softgate in GCP

+

As stated in the previous section, the following sequence of actions must be taken in order to proceed: create a VM instance, add Softgate into the Netris Controller, install Netris Softgate software on the VM instance, and configure routes in GCP VPC. Let’s commence with these steps in the specified order.

+
+

Create a VM instance

+

Due to Netris Softgate is a network device capable of supporting numerous network services and being equipped with its own firewall, it is advisable to open all ports for the associated VM instance. To achieve this, create a VPC firewall rule with the following parameters:

+
    +
  • Name: Any name

  • +
  • Direction of traffic: “Ingress”

  • +
  • Action on match: “Allow”

  • +
  • Targets: “Specified target tags”

  • +
  • Target tags: allow-all

  • +
  • Source filter: “IPv4 ranges”

  • +
  • Source IPv4 ranges: 0.0.0.0/0

  • +
  • Protocols and ports: “Allow all”

  • +
+

Afterward, a VM instance can be created using the network tag above.

+../_images/gcp-firewall-rule.png +

To enable connectivity with other Netris sites, creating the VM instance in the desired VPC is essential. Therefore, deploy a new VM instance with the Ubuntu 22.04 operating system installed, utilizing an instance type that meets the minimum hardware requirements of 2 virtual CPUs, 4 GB of RAM, and 30 GB of drive space, such as e2-medium or any other type that satisfies these specifications. From the Advanced Networking options enable IP forwarding and specify the previously created allow-all network tag.

+../_images/gcp-softgate-deployed.png +

After successfully deploying the VM instance, it is crucial to take note of its External IP. This address will be required in the upcoming step.

+
+
+

Configure Netris Controller

+

Prior to defining a Softgate in the Netris Controller, certain pre-requisite steps must be completed.

+
+

Pre-Requisite Steps

+

In the Netris Controller, the initial step involves creating a new site. To create it, follow the steps below:

+
    +
  1. Open the Netris Web Console.

  2. +
  3. Navigate to “Net” and select “Sites”.

  4. +
  5. Click on the “+ Add” button.

  6. +
  7. Select “Dot1q Trunk” as the “Switch Fabric”.

  8. +
  9. Input a descriptive name for the site.

  10. +
  11. Specify 65500 in the “Public ASN” field.

  12. +
  13. Click “Add” to create the new site.

  14. +
+../_images/gcp-netris-site-create.png +

Subsequently, it is necessary to create a subnet with the “Purpose” of loopback within Netris IPAM. The subnet’s “Prefix” will match the External IP of the GCP VM instance, with a netmask length of /32. For instance, if the VM instance’s External IP address is 34.85.167.128, then the “Prefix” for the loopback subnet will be 34.85.167.128/32. It’s important to note that an allocation for this “Prefix” must be created prior to creating the loopback subnet. To achieve this, follow the steps below:

+
    +
  1. Go to the “IPAM” section under the “Net” tab.

  2. +
  3. Click on the “+ Add” button located at the top-right corner.

  4. +
  5. Enter the “Prefix” for the new allocation, such as 34.85.167.128/32.

  6. +
  7. Type a descriptive “Name” for the allocation.

  8. +
  9. Select the desired tenant name from the “Tenant” dropdown menu.

  10. +
  11. Click on the “Add” button to create the allocation.

  12. +
+

Once the allocation is created, proceed with the creation of the subnet as follows:

+
    +
  1. Click on the “+ Add” button located at the top-right corner.

  2. +
  3. Enter the “Prefix” for the new subnet, such as 34.85.167.128/32.

  4. +
  5. Type a descriptive “Name” for the subnet.

  6. +
  7. Select the desired tenant name from the “Tenant” dropdown menu.

  8. +
  9. From the “Type” dropdown menu, select “Subnet”.

  10. +
  11. Select “loopback” from the “Purpose” dropdown menu.

  12. +
  13. Choose the appropriate site from the “Sites” dropdown menu.

  14. +
  15. Click on the “Add” button to create the subnet.

  16. +
+../_images/gcp-netris-ipam-lo.png +
+
+

Create the Softgate in the Netris Controller

+

After completing the pre-required steps, the next step is to create a Softgate in the Netris Controller. Here is a step-by-step guide:

+
    +
  1. Ensure that you have completed all the pre-required steps.

  2. +
  3. Navigate to the “Net” tab in the Netris Controller and select the “Inventory” section.

  4. +
  5. Click on the “+Add” button to create a new Softgate.

  6. +
  7. Provide a descriptive name for the Softgate in the “Name” field.

  8. +
  9. From the “Tenant” dropdown menu, select the same tenant name used in the pre-required steps when creating the subnets.

  10. +
  11. From the “Type” dropdown menu, select “SoftGate”.

  12. +
  13. Choose the appropriate site from the “Site” dropdown menu.

  14. +
  15. For the “Main IP address” field select “Assign automatically”.

  16. +
  17. For the “Management IP Address” field select “None”.

  18. +
  19. In the “Description” field, add int=ens4 to specify that Netris should use the softgate’s ens4 interface instead of the default bond0 interface that Netris Softgate Agent looks for.

  20. +
  21. Finally, click on the “Add” button to create the Softgate.

  22. +
+../_images/gcp-netris-create-sg.png +
+
+

Add GCP VPC Subnet(s) into the Netris Controller

+

The next step involves registering into Netris IPAM the subnet of the region where the SoftGate node has been created. Follow the steps below to accomplish this:

+
    +
  1. Access the GCP Console and navigate to your VPC Subnets.

  2. +
  3. Take note of the Internal IP range associated with the relevant region.

  4. +
  5. In Netris Controller, go to the “IPAM” section under the “Net” tab.

  6. +
  7. Click the “+ Add” button located at the top-right corner.

  8. +
  9. Enter the region’s Internal IP range into the “Prefix” field for the new subnet. For example, if your Internal IP range is “10.150.0.0/20”, enter that value.

  10. +
  11. Type a descriptive name for the subnet.

  12. +
  13. From the “Tenant” dropdown menu, select the desired tenant name.

  14. +
  15. From the “Type” dropdown menu, select “Subnet”.

  16. +
  17. Select “inactive” from the “Purpose” dropdown menu.

  18. +
  19. Choose the appropriate site from the “Sites” dropdown menu.

  20. +
  21. Click the “Add” button.

  22. +
+

Once the subnet of the region where the SoftGate node was created has been successfully registered in Netris IPAM, you can proceed to create subnets for other necessary regions in a similar manner.

+../_images/gcp-vpc-subnet-to-netris.png +
+
+

Static route for GCP VPC Subnet(s) in Netris Controller

+

We need to create route entries in Netris. The prefix for the route will be the GCP VPC Subnet(s), and the next-hop will be the default gateway of Netris Softgate VM Instance.

+

Here are the steps to create the static route:

+
    +
  1. Securely log in to the Netris Softgate’s VM instance using SSH.

  2. +
  3. Retrieve the default gateway address by typing the command ip route show default.

  4. +
  5. In Netris Controller, go to the “Routes” section under the “Net” tab.

  6. +
  7. Click on the “+ Add” button located at the top-right corner of the screen.

  8. +
  9. Enter the subnet of the GCP region in the “Prefix” field.

  10. +
  11. Enter the default gateway of the Netris Softgate VM instance in the “Next-Hop” field.

  12. +
  13. Choose the appropriate site from the “Site” dropdown menu.

  14. +
  15. From the “Apply to” dropdown menu, select the SoftGate.

  16. +
  17. Finally, click on the “Add” button to create the static route.

  18. +
+

Repeat these steps for each GCP VPC Subnet that has been registered in Netris IPAM.

+

Note: Regardless of the GCP region’s subnet, the “Next-Hop” field should always contain the default gateway of the Netris SoftGate VM instance.

+../_images/gcp-netris-static-route.png +
+
+

Netris SoftGate node provisioning

+

After creating a softgate resource in Netris Controller and defining all necessary routes, the next step is to install the softgate agent. This can be done by clicking the three vertical dots (⋮) on the right side of the created SoftGate node and selecting “Install Agent”. Copy the one-line installer command to your clipboard and connect to your VM instance via SSH. Paste the copied command into the terminal and wait for the provisioning to finish. Once it’s done, reboot the server.

+../_images/gcp-netris-provision-sg.png +
+
+
+

Configure routing in GCP VPC

+

To ensure that specific traffic is directed to the Netris Softgate VM instance within your GCP VPC, it is necessary to configure new route entries in VPC routes. Follow the step-by-step guide below to add GCP VPC routes:

+
    +
  1. Access the Google Cloud Platform (GCP) Console.

  2. +
  3. Navigate to the VPC Network page.

  4. +
  5. Click on the “Routes” then switch to the “ROUTE MANAGEMENT” tab to view the existing routes.

  6. +
  7. Click the “+ Create Route” button to create a new route.

  8. +
  9. Provide the following details for the new route:

  10. +
+
+
    +
  • Name: Assign a descriptive name to the route.

  • +
  • Network: Select the appropriate network for the route.

  • +
  • Destination IP Range: Specify the subnets of other Netris sites that you wish to access from this VPC.

  • +
  • Next hop: From the dropdown menu, select the “Specify an instance”

  • +
  • Next hop instance: From the dropdown menu, select the SoftGate VM Instance

  • +
+
+
    +
  1. Review the route configuration and ensure all the details are accurate.

  2. +
  3. Click the “Create” button to add the route to the GCP VPC network.

  4. +
  5. Verify that the new route appears in the list of routes for the selected VPC network.

  6. +
+

Repeat these steps for any additional routes you need to add.

+../_images/gcp-vpc-routes-created.png +
+
+

Enabling Site Mesh

+

To enable Site Mesh, in Netris Controller, navigate to the “Net” tab and select “Sites”. Next, click on the three vertical dots (⋮) on the right side of the GCP site and select “Edit”, and then from the “Site Mesh” dropdown menu, select “Hub”. Save the changes. Repeat these steps for all sites that need to have meshed.

+../_images/gcp-netris-enable-site-mesh.png +

The Site Mesh status can be viewed by navigating to the “Site Mesh” section under the “Net” tab. This will display the current status of Site Mesh for all Sites.

+../_images/gcp-netris-site-mesh-status.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/getting-started-for-equinix-metal.html b/en/latest/tutorials/getting-started-for-equinix-metal.html new file mode 100644 index 0000000000..6f9746f240 --- /dev/null +++ b/en/latest/tutorials/getting-started-for-equinix-metal.html @@ -0,0 +1,904 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Equinix Metal API integration enablement — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Equinix Metal API integration enablement
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Equinix Metal API integration enablement

+

For each Equinix Metal Project+location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “Equinix Metal” from the dropdown menu.

Name

Type a descriptive name for your Equinix Metal Project+location.

Equinix Project ID

Copy/Paste the Project ID from Equinix Metal portal under Project Settings → General → Project ID.

Equinix Project API key

Create a new Read/Write API key in Equinix Metal portal under Project Settings → Project API keys → + Add New Key. Then copy/paste here.

Equinix Location

Select your equinix location from the dropdown menu.

+

Equinix Metal Project ID

+../_images/equinix-metal-project-id.png +

Equinix Metal Project API key

+../_images/equinix-metal-project-api-keys.png +

Netris Create New Site

+../_images/netris-create-equinix-metal-site.png +
+

Adding Netris SoftGate nodes

+

For SoftGate nodes you can start with two servers of the smallest flavor. In the future if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration) you can upgrade the servers one-by-one.

+

Request two servers from Equinix Metal with Ubuntu 22.04 OS and wait until provisioned.

+
    +
  1. At this point you should see Netris Controller listing newly created servers as “Equinix Metal Server” under Netris Web Console → Net → Inventory

  2. +
+../_images/softgate-nodes-created-in-equinix.png +
    +
  1. When Equinix finishes provisioning of the servers, click on each server name, then click tag, and add a tag “netris-softgate”.

    +
    +

    Tag “netris-softgate” will signal Netris Controller that these two servers are going to be used as Netris SoftGate nodes in this particular site (Project+Location).

    +
    +
  2. +
+

Then you should see in Netris web console that description changes from “Equinix Metal Server” into “Softgate Softgate1(2)”. You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+../_images/softgate-nodes-recognized-in-netris.png +
    +
  1. Provision SoftGate nodes.

  2. +
+

Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server as a root user and paste the one-liner there.

+../_images/softgate-one-liner-provisioning.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/softgate-green.png +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/index-equinix.html b/en/latest/tutorials/index-equinix.html new file mode 100644 index 0000000000..e37243ed70 --- /dev/null +++ b/en/latest/tutorials/index-equinix.html @@ -0,0 +1,868 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris VPC for Equinix Metal Getting Started Guide — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris VPC for Equinix Metal Getting Started Guide
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Netris VPC for Equinix Metal Getting Started Guide

+

You are welcome to join our Slack channel to get additional support from our engineers and community.

+

This video walks through Netris VPC setup steps described below.

+ + +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/index-vpc.html b/en/latest/tutorials/index-vpc.html new file mode 100644 index 0000000000..ec5f9557bf --- /dev/null +++ b/en/latest/tutorials/index-vpc.html @@ -0,0 +1,859 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + VPC for Anywhere Getting Started Guide — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • VPC for Anywhere Getting Started Guide
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

VPC for Anywhere Getting Started Guide

+

You are welcome to join our Slack channel to get additional support from our engineers and community.

+ +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/index.html b/en/latest/tutorials/index.html new file mode 100644 index 0000000000..8514ef8903 --- /dev/null +++ b/en/latest/tutorials/index.html @@ -0,0 +1,868 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Generic Tutorials — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Netris Generic Tutorials
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Netris Generic Tutorials

+

You are welcome to join our Slack channel to get additional support from our engineers and community.

+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/install-netris-controller-in-equinix-metal.html b/en/latest/tutorials/install-netris-controller-in-equinix-metal.html new file mode 100644 index 0000000000..20d380f450 --- /dev/null +++ b/en/latest/tutorials/install-netris-controller-in-equinix-metal.html @@ -0,0 +1,899 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller on Equinix Metal on-demand server — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Installing a Netris Controller on Equinix Metal on-demand server

+
+

Installation Steps

+

You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions).

+

Request a server

+

The smallest flavor, c3.small.x86, is enough for most users.

+../_images/equinix-request-c3-small-server.png +

if you decide to use a VM, please see below the minimal VM requirements.

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

DNS record

+

In my example my host got a public IP address 139.178.89.255. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is example using Cloudflare DNS service. (same idea with any DNS software or service)

+../_images/dns-cloudflare-equinix-ip.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 139.178.89.255
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/installing-netris-controller.html b/en/latest/tutorials/installing-netris-controller.html new file mode 100644 index 0000000000..fe7ae2b983 --- /dev/null +++ b/en/latest/tutorials/installing-netris-controller.html @@ -0,0 +1,926 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Installing a Netris Controller

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where you host the Netris controller. What matters is that 1) the Netris controller needs to be accessible over the Internet. 2) You can access the web console. 3) Nodes that are going to be managed by Netris have access to the Netris controller through their management network interface.

+

Linux host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

DNS record

+

In the example below, the host has a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, it is recommended to use a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is an example using Cloudflare DNS service (may use any DNS service).

+../_images/dns-record-netrisctl.png +

Ensure that newly created domain name resolves to the right IP address of the machine that will host the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 54.183.23.201
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

The Netris Controller installer will create a K3s cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once the installation process is finished, you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +

Restrict incoming TCP requests to the list below:

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + +

TCP Port

Service

22

SSH

80

HTTP

443

Netris Web Console

2003

Streaming Telemetry (Collectd)

3033

Netris Monitoring (Telescope)

50051

Netris Agent (gRPC)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/inventory-setup.html b/en/latest/tutorials/inventory-setup.html new file mode 100644 index 0000000000..281fc805d2 --- /dev/null +++ b/en/latest/tutorials/inventory-setup.html @@ -0,0 +1,884 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Inventory setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Inventory setup

+

The Inventory section includes Netris-managed devices and allows you to add, edit, or delete network switches and SoftGates. +The initial setup of the Netris managed fabric consists of a three-step process:

+
    +
  • Create Inventory Profiles

  • +
  • Add Switches

  • +
  • Add Softgates

  • +
+
+

Note

+

You can also add new devices in the Topology view.

+
+

Inventory profiles

+

Inventory profiles enable security hardening for inventory devices. By default, all traffic flow destined for a switch or SoftGate is allowed. However, once an inventory profile is attached to a device, it denies all traffic destined for the device except for Netris-defined and user-defined custom flows. Generated rules include:

+
    +
  • SSH from user-defined subnets

  • +
  • NTP from user-defined NTP services

  • +
  • DNS from user-defined DNS servers

  • +
  • Custom user-defined rules

  • +
+

The Netris Controller includes a preconfigured Inventory profile named “default-inventory-profile.” You can either edit this profile or create your own.

+

SoftGate creation

+

Each SoftGate node needs to be added to the Netris Controller inventory. +Network → Inventory → +Add

+../_images/inventory_softgate.png +

Switch creation

+

Each Switch node needs to be added to the Netris Controller inventory. +Network → Inventory → +Add

+../_images/inventory_switch.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/ipam-setup.html b/en/latest/tutorials/ipam-setup.html new file mode 100644 index 0000000000..5aa63e24d7 --- /dev/null +++ b/en/latest/tutorials/ipam-setup.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + IPAM setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

IPAM setup

+

Netris IPAM enables users to manage their IP addresses and monitor pool usage effectively. It features a hierarchical view to facilitate various subnetting tasks. +Users must first assign specific roles (purposes) to each subnet or address before they can utilize these subnets in services such as V-Net, NAT, Load Balancing, etc.. +Each VPC has its own IPAM table.

+

Create allocations

+

There are two primary types of IP prefixes: allocations and subnets. Allocations consist of IP ranges assigned to an organization through RIR/LIR or private IP ranges intended for network use. Subnets, on the other hand, are prefixes that will be utilized in various services. Subnets always fall under allocations, while allocations do not have parent subnets.

+

In addition to the predefined subnets, the Netris Controller also includes predefined allocations, consisting of private IP addresses defined in RFC 1918 - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. If you intend to create subnets that fall outside of these predefined allocations, you should first create allocations that encompass those subnets.

+../_images/ipam_allocation.png +

Create subnets for devices

+

You will require two subnets for your devices: one for loopback IP addresses and another for the management network. Note that device subnets must reside in the System VPC.

+../_images/ipam_mgmt_subnet.png +../_images/ipam_loopback_subnet.png +

Create subnets for V-Nets

+

Create at least one subnet with the Common purpose to use it for a new V-Net. IP addresses from this subnet will be assigned to your servers.

+../_images/ipam_common_subnet.png +

Create subnets for Load-balancing service

+

If you plan to use load-balancing services, you should first define subnet(s) from which IP addresses will be assigned for Virtual IP (frontend).

+../_images/ipam_l4lb_subnet.png +

Create subnets for NAT service

+

If you plan to perform network address translation (NAT), you must first create subnets for this purpose.

+../_images/ipam_nat_subnet.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/more-features.html b/en/latest/tutorials/more-features.html new file mode 100644 index 0000000000..e5e085f717 --- /dev/null +++ b/en/latest/tutorials/more-features.html @@ -0,0 +1,858 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + More features — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

More features

+

Access lists

+

Static routing, Site Mesh (VPN) and Routing on the host features can be found under Network Policies.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/netris-controller-installation.html b/en/latest/tutorials/netris-controller-installation.html new file mode 100644 index 0000000000..91285a38b9 --- /dev/null +++ b/en/latest/tutorials/netris-controller-installation.html @@ -0,0 +1,929 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Install a Netris Controller — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Install a Netris Controller

+
+

Requirements and Installation steps

+

You can install the Netris controller on almost any 64-bit Linux host. The Netris Controller may or may not be on the same network as the managed network nodes. Multi-region, multi-site deployments can use a single Netris Controller (+backup).

+

The location of the Netris controller host is not critical. However, it is important that the Netris controller is accessible over the network. This is required for two reasons: 1) So you can access the web console, and 2) Nodes that will be managed by Netris need access to the Netris controller through their management network interface.

+

If you choose to set up the Netris Controller within your Netris managed network, make sure to provide access to the controller using an out-of-band network connection.

+

Linux host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

DNS record

+

In this example, the controller host has a public IP address 54.183.23.201. While it is acceptable for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record. This approach will make it easier to potentially move the Netris Controller to a different IP address in the future.

+

Below is an example using the Cloudflare DNS service. The same concept applies to any DNS software or service.

+../_images/dns-record-netrisctl.png +

Ensure that the newly created domain name indeed resolves to the correct IP address of the machine on which you plan to install the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 54.183.23.201
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

The Netris Controller installer will create a K3s cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once the installation process is finished, you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +

Restrict incoming TCP requests to the list below:

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + +

TCP Port

Service

22

SSH

80

HTTP

443

Netris Web Console

2003

Streaming Telemetry (Collectd)

3033

Netris Monitoring (Telescope)

50051

Netris Agent (gRPC)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/netris-managed-fabric-overview.html b/en/latest/tutorials/netris-managed-fabric-overview.html new file mode 100644 index 0000000000..b62f9a0e23 --- /dev/null +++ b/en/latest/tutorials/netris-managed-fabric-overview.html @@ -0,0 +1,866 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris managed fabric Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris managed fabric Overview

+
+

Introduction

+

VPC gateways (SoftGate) with Netris managed fabric is a comprehensive solution for operating physical networks as if they were cloud-based. By abstracting the complexities of granular network configuration, Netris enables you to operate your physical network using a top-down approach, similar to cloud management, instead of the traditional box-by-box operation.

+
+
+

Concept

+

The Netris managed fabric is a switch fabric that employs EVPN technology for enhanced configuration and performance. The Netris Switch Agent should be installed and operational within the user space of the switch’s network operating system (NOS). The agent is responsible for automatically generating specific switch configurations based on service requirements and policies defined in the Netris Controller. The Netris managed fabric delivers high capacity, redundancy, resilience, and high availability, along with traffic filtering based on access lists for optimal network performance.

+

Netris SoftGate, known as VPC gateway, is a software for enabling border routing, Layer-4 Load Balancing, Network Address Translation (NAT), DHCP, and site-to-site VPN function on a regular x86 server with a SmartNIC card or without it.

+../_images/switch_fabric_vpc.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/netris-switch-agent-installation.html b/en/latest/tutorials/netris-switch-agent-installation.html new file mode 100644 index 0000000000..1806877fb6 --- /dev/null +++ b/en/latest/tutorials/netris-switch-agent-installation.html @@ -0,0 +1,857 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris Switch Agent Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris Switch Agent Installation

+

The Netris Switch Agent is compatible with several Network Operating Systems. Here is a list of these systems, along with installation guides for each one. Install the Netris Agent and come back to this tutorial.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/netris-vpc-for-aws.html b/en/latest/tutorials/netris-vpc-for-aws.html new file mode 100644 index 0000000000..bc4fb53b47 --- /dev/null +++ b/en/latest/tutorials/netris-vpc-for-aws.html @@ -0,0 +1,875 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling Site Mesh with AWS — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Enabling Site Mesh with AWS
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/netris-vpc-for-equinix-metal.html b/en/latest/tutorials/netris-vpc-for-equinix-metal.html new file mode 100644 index 0000000000..54466fd995 --- /dev/null +++ b/en/latest/tutorials/netris-vpc-for-equinix-metal.html @@ -0,0 +1,881 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Getting Started with Equinix Metal VPC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/netris-vpc-for-gcp.html b/en/latest/tutorials/netris-vpc-for-gcp.html new file mode 100644 index 0000000000..d2cea7be89 --- /dev/null +++ b/en/latest/tutorials/netris-vpc-for-gcp.html @@ -0,0 +1,875 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Enabling Site Mesh with GCP — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Enabling Site Mesh with GCP
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/netris-vpc-for-phoenixnap-bmc.html b/en/latest/tutorials/netris-vpc-for-phoenixnap-bmc.html new file mode 100644 index 0000000000..eb17f2ab3c --- /dev/null +++ b/en/latest/tutorials/netris-vpc-for-phoenixnap-bmc.html @@ -0,0 +1,892 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Getting Started with PhoenixNAP VPC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Getting Started with PhoenixNAP VPC
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/new-site-setup.html b/en/latest/tutorials/new-site-setup.html new file mode 100644 index 0000000000..23e929b8c3 --- /dev/null +++ b/en/latest/tutorials/new-site-setup.html @@ -0,0 +1,867 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + New Site setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

New Site setup

+

For each individual deployment, region, location, data center, etc. you should define it as a Site. All network components and resources should be associated with their respective Site and VPC.

+

Add a new Site. +Network → Sites → +Add

+
    +
  • Enter the Site name

  • +
  • Enter your public Autonomous System Number (ASN) if you have one. If not, use a private ASN within the range of 64512 to 65534.

  • +
  • Choose Switch fabric type “Netris”.

  • +
+

If you’re implementing the Zero Trust security model, you may want to select the ACL Default Policy “Deny.” More details can be found +here.

+../_images/site_setup.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.html b/en/latest/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.html new file mode 100644 index 0000000000..f436df2ad9 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-adding-netris-softgate-nodes.html @@ -0,0 +1,856 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Provisioning Netris SoftGate nodes in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Provisioning Netris SoftGate nodes in phoenixNAP BMC
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Provisioning Netris SoftGate nodes in phoenixNAP BMC

+

For SoftGate nodes you can start with two s2.c1.small or larger servers. In the future, if you happen to need to upgrade to high-performance SoftGate PRO (with DPDK acceleration), you can upgrade the servers one-by-one.

+

1) Request two servers (s2.c1.small) from phoenixNAP BMC with Ubuntu Jammy OS, type netris-softgate in the “Server Description” field, +choose a /31 Public IP Allocation and wait until provisioned.

+
+

Note

+

It’s required to type the netris-softgate in the description. This signals Netris Controller that those are not regular bare-metal servers, and they should be synced with the type of SoftGate.

+
+../_images/phoenixnap-softgate-nodes-creation.png +

2) At this point you should see Netris Controller listing newly created servers as “Softgate Softgate1(2)” under Netris Web Console → Net → Inventory. +You will also notice IP addresses listed per SoftGate, and Heartbeat: CRIT. We will bring Heartbeat to OK in next step.

+

3) Provision SoftGate nodes. Netris Controller provides a one-liner command for provisioning SoftGate nodes. +Go to Netris web console → Net → Inventory, click on the 3 dots menu, and click “Install Agent”, and copy the one-liner command.

+

Then SSH to the corresponding SoftGate server with the ubuntu user and paste the one-liner there.

+../_images/phoenixnap-softgate-nodes-created.png +

When provisioning is done, reboot the server. In a few minutes Netris Controller should sense the heart beats from SoftGate as in the below screenshot. Repeat for the second SoftGate too.

+../_images/phoenixnap-softgate-nodes-green.png +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-api-integration-enablement.html b/en/latest/tutorials/phoenixnap-bmc-api-integration-enablement.html new file mode 100644 index 0000000000..07775270bf --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-api-integration-enablement.html @@ -0,0 +1,875 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Enable phoenixNAP BMC API integration — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Enable phoenixNAP BMC API integration
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Enable phoenixNAP BMC API integration

+

For each phoenixNAP BMC location you need to define an individual Site in Netris Controller.

+

Go to Netris Web Console → Net → Sites and click +Add.

+

You only need to deal with the below 5 fields. Leave the rest to default values for now.

+ ++++ + + + + + + + + + + + + + + + + + + + + + + +

Netris Parameter

What to do:

Switch Fabric

Select “PhoenixNAP BMC” from the dropdown menu.

Name

Type a descriptive name for your phoenixNAP BMC location.

PhoenixNAP Client ID

Create a new API Credential with “bmc” scope in phoenixNAP BMC portal under API Credentials → + Create Credentials. Then copy/paste Client ID.

PhoenixNAP Client Secret

Copy/Paste the Client Secret from the already created API Credential.

PhoenixNAP Location

Select your phoenixNAP BMC location from the dropdown menu.

+

phoenixNAP BMC API Credential

+../_images/phoenixnap-api-credential.png +

Netris Create New Site

+../_images/phoenixnap-site-create.png +
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-concept.html b/en/latest/tutorials/phoenixnap-bmc-concept.html new file mode 100644 index 0000000000..6f3fbd1336 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-concept.html @@ -0,0 +1,880 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Netris VPC for phoenixNAP BMC Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris VPC for phoenixNAP BMC Overview

+
+

Introduction

+

Netris VPC for phoenixNAP BMC is a solution that enables VPC network functionality in phoenixNAP’s Bare Metal Cloud. Netris VPC can be used for all or part of the servers/network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures.

+
+
+

Concept

+

There is an agent inside the Netris VPC controller for phoenixNAP BMC that communicates to phoenixNAP API once the integration has been enabled. That agent is responsible for metadata synchronization between phoenixNAP BMC and Netris VPC controller. Also, it configures the BMC network services based on the services defined in the Netris VPC controller.

+

SoftGate is a highly optimized automatic Linux gateway, which in turn communicates with the Netris VPC controller via an encrypted protocol. A SoftGate node (typically two of them for redundancy) is a regular BMC server that should be deployed on desired phoenixNAP location. Once the server has been deployed, it starts consuming the BMC public and private networks with various VLANS.

+../_images/phoenixnap-concept-solution-traffic-flows.png +

Due to phoenixNAP BMC creating a new VLAN for each server’s Public IP Allocation and Public Network/Private Network, we decided to use the upper VLANS Range - 3000-4094 (if you need to change the default range, that can be done in the Netris Site settings). Netris will consume VLANS only from that range. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the BMC network.

+

SoftGate node becomes the default gateway for the workloads consuming the VPC network. PhoenixNAP BMC Private Network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers. Netris also creates a Public Network in the phoenixNAP BMC when the IP Addresses from Public IP Allocations (passed to Netris VPC) have been used in any Netris VPC’s services. Besides Public/Private Networks creation Netris also attaches all necessary servers to that networks. Therefore, you don’t need to worry about assigning servers to the networks on the phoenixNAP side, Netris will handle that automatically when that makes sense.

+../_images/phoenixnap-concept-public-network.png +

What’s next

+ +

Usage

+ +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-install-netris-controller.html b/en/latest/tutorials/phoenixnap-bmc-install-netris-controller.html new file mode 100644 index 0000000000..27b1de3c87 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-install-netris-controller.html @@ -0,0 +1,882 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing a Netris Controller on phoenixNAP BMC server — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Installing a Netris Controller on phoenixNAP BMC server
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ +
+

Installing a Netris Controller on phoenixNAP BMC server

+
+

Installation Steps

+

You can install the Netris controller almost on any 64-bit Linux host. You can use a single Netris controller for operating multiple sites (regions).

+

Minimum Hardware Requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

Deploy New Server

+

Deploy a new server in phoenixNAP BMC: select the Location, choose the server flavor (the smallest flavor, s0.d1.small, is enough for most users), type some name for Controller Server, request a new /31 public allocation for this instance.

+../_images/phoenixnap-request-ctl-server.png +

DNS record

+

In my example my host got a public IP address 131.153.154.61. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is example using Cloudflare DNS service. (same idea with any DNS software or service)

+../_images/phoenixnap-dns-cloudflare.png +

Ensure that newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 131.153.154.61
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netrisctl.netris.dev --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-ipam-setup.html b/en/latest/tutorials/phoenixnap-bmc-ipam-setup.html new file mode 100644 index 0000000000..0c3c353c90 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-ipam-setup.html @@ -0,0 +1,914 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + IPAM Setup for Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

IPAM Setup for Services

+

Although bare metal servers in phoenixNAP BMC may get a public IP address and can access the Internet, it’s not the case with the VMs. When you use server virtualization, something needs to be the gateway for your virtual networks. That gateway will provide packet forwarding with access control between your virtual private networks, will provide NAT, and on-demand (Elastic) Load Balancer services. Physically that gateway is Netris SoftGate. And it operates automatically, providing you with the VPC-like networking capabilities.

+

Both NAT and on-demand Load Balancer services need public IP addresses.

+

Go to phoenixNAP BMC web console and click on Networking → Public IP Allocations (see the screenshot below)

+
+

Option 1 - Separated allocation for each purpose

+

In this example, I’m requesting two /29 (5 assignable IPs) IP Allocations, one for NAT and one for Load Balancer.

+

It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC.

+

You can always request more IP address allocations in the future.

+../_images/phoenixnap-request-ip-allocation.png +

Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM

+../_images/phoenixnap-netris-ipam-synced.png +

You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service).

+
+

1. Enable on-demand (elastic) Load Balancer

+

To Enable an on-demand (elastic) Load Balancer, you only need to change the “purpose” field of appropriate IP subnet from “inactive” into “load-balancer”

+

Click on the 3 dots menu (in this example of first /29 subnet), click edit, and select “load-balancer” from the dropdown menu next to the “purpose” field.

+../_images/phoenixnap-netris-ipam-lb-purpose.png +

Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform.

+

Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it.

+
+
+

2. Enable NAT

+

To enable NAT, you need to repurpose a subnet for NAT. In the below example, I’m repurposing the second of the newly requested /29 subnets for NAT.

+../_images/phoenixnap-netris-ipam-nat-purpose.png +

Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT.

+
+
+
+

Option 2 - Splitting single allocation into different subnets

+

In this example, I’m requesting a single /28 (13 assignable IPs) Public IP Allocation, then splitting it into two /29 subnets, one for NAT and one for Load Balancer.

+

It’s important to type the “netris” word in the description. This is a signal for Netris Controller that this allocation is intended for Netris VPC.

+

You can always request more IP address allocations in the future.

+../_images/phoenixnap-request-ip-allocation-slash-28.png +

Once IP address allocations are provisioned on phoenixNAP BMC you should be able to find them automatically replicated in Netris web console under Net → IPAM

+../_images/phoenixnap-netris-ipam-synced-slash-28.png +

You don’t need to worry about assigning them to a Public Network on the phoenixNAP side, Netris will handle that automatically when that makes sense (associated with any service).

+
+

1. Enable on-demand (elastic) Load Balancer

+

In this scenario, to Enable an on-demand (elastic) Load Balancer, you need to create a smaller subnet with the purpose of “load-balancer”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/phoenixnap-netris-ipam-lb-purpose-slash-28.png +

Now on-demand (elastic) load balancer service is enabled and can be consumed either from web console, or from Kubernetes using service of the type load-balancer, or with Terraform.

+

Please note that in this example we left the field Tenant set to Admin. Tenancy is used for role based access control and resource delegation. In other words you may want to create a user role and tenant for your colleagues that are supposed to consume Netris VPC services, but not administer it.

+
+
+

2. Enable NAT

+

To enable NAT, you need to create a smaller subnet with the purpose of “nat”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/phoenixnap-netris-ipam-nat-purpose-slash-28.png +

Then You need to create a NAT rule in the Net → NAT section of Netris web console. Netris supports most of the standard rules for SNAT and DNAT.

+
+
+
+

Note*

+

It doesn’t matter with what Option (1,2) you will go. Once the Public IP Allocation has been replicated in the Netris IPAM, Netris will automatically reserve the network, first usable, and broadcast IP addresses because they are unusable in this (phoenixNAP BMC) scenario.

+../_images/phoenixnap-reserved-ips.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-link-to-installation.html b/en/latest/tutorials/phoenixnap-bmc-link-to-installation.html new file mode 100644 index 0000000000..333eb67240 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-link-to-installation.html @@ -0,0 +1,859 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Installing Netris on phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Installing Netris on phoenixNAP BMC

+

To deploy Netris on the phoenixNAP Bare Metal Cloud, please follow the official guide provided by phoenixNAP. This guide will walk you through the step-by-step installation process.

+

Netris for Bare Metal Cloud

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-using-l4lb.html b/en/latest/tutorials/phoenixnap-bmc-using-l4lb.html new file mode 100644 index 0000000000..03533be153 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-using-l4lb.html @@ -0,0 +1,861 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using on-demand (elastic) L4 Load Balancer service in phoenixNAP BMC

+

Services –> L4 Load Balancer is an on-demand (elastic) load balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console.

+../_images/phoenixnap-l4lb.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-using-nat.html b/en/latest/tutorials/phoenixnap-bmc-using-nat.html new file mode 100644 index 0000000000..76103c6cb3 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-using-nat.html @@ -0,0 +1,875 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using NAT services in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using NAT services in phoenixNAP BMC

+

NAT, under Net –> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 10.0.0.0/8 to have access to services outside their VPC. The goal is to provide bare-metal servers/VMs with the capability to connect to the Internet through NAT for outbound access.

+
+

Option 1 - MASQUERADE

+

The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn’t require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT).

+../_images/phoenixnap-nat-masquerade.png +
+
+

Option 2 - SNAT

+

Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a NAT purpose. SNAT replaces the source IP address of the instances with the IP address of the “SNAT to IP”. Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity.

+../_images/phoenixnap-nat-snat.png +

You can always have more granular control either through NAT rule or using Services → ACLs.

+
+
+

DNAT

+

A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a “DNAT to IP/Port”. This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database.

+

I’m creating a DNAT rule for the ssh port in the example below. It forwards the public IP’s 55022 port to the local IP’s 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server.

+../_images/phoenixnap-nat-dnat.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/phoenixnap-bmc-using-vnet.html b/en/latest/tutorials/phoenixnap-bmc-using-vnet.html new file mode 100644 index 0000000000..1775269371 --- /dev/null +++ b/en/latest/tutorials/phoenixnap-bmc-using-vnet.html @@ -0,0 +1,931 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using V-Net (isolated virtual network) services in phoenixNAP BMC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using V-Net (isolated virtual network) services in phoenixNAP BMC

+

V-Net, under Services –> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from phoenixNAP BMC into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network.

+

Netris V-Net (in phoenixNAP BMC scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Private Network using phoenixNAP BMC API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Private Network. SoftGate nodes are the default gateway for the V-Net services.

+
+

Adding Subnets for V-Net

+

Before starting to use the V-NETs, you need to create a few subnets of private IP address blocks to be used by you and your colleagues later on for creating V-Nets. For now you can set the Tenant field to Admin, but in the future if you need to create a separate user role that should be able to consume Netris VPC but not administer it, you will need to create a separate Tenant and give that Tenant IP resources.

+../_images/phoenixnap-netris-create-common-subnets.png +
+
+

Creating a V-Net

+

You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, phoenixNAP BMC, and your Compute.

+../_images/phoenixnap-netris-creating-vnet.png +

In this example, the new V-NET has VLAN ID 3000, subnet 10.128.1.0/24, and gateway 10.128.1.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 3000, and they should use IP addresses from 10.128.1.2-254 pointing to 10.128.1.1 as the default gateway or use DHCP. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 3000 will have Internet access over the NAT.

+../_images/phoenixnap-netris-vnet-ready.png +
+

Deploy a new server w/ dynamic IP into an existing V-Net

+

Deploying a new server is significantly simpler when using Netris V-NET compared to the traditional process of setting up a server with a public IP.

+

The recommended steps:

+
    +
  1. Request a Server: When ordering a server from phoenixNAP BMC, opt for no public IP address.

  2. +
  3. Select Netris V-Net: In the phoenixNAP web console, select a Netris V-Net (previously created in Netris) as your private network.

  4. +
  5. DHCP: In the phoenixNAP web console, check the option “Use your own privately managed DHCP server (Obtain IP address automatically)”. Make sure that DHCP is enabled for the given V-Net through the Netris web console.

  6. +
+

Netris will automatically include the server into the V-Net and will assign an IP configuration through Netris DHCP service.

+../_images/phoenixnap-vnet-import-a-new-server.png +

As a result, you will have bare-metal servers in a private subnet that can connect to services outside your VPC (since we have enabled NAT globally in previous chapters), but external services cannot initiate a connection with those servers. Once the servers have been provisioned, they will get a private IP from Netris DHCP, and you can find those IPs by pressing the “IP/MAC Info” button on the V-NET.

+../_images/phoenixnap-vnet-imported-new-server.png +

In order to connect via SSH to the newly deployed server, you can either create a DNAT rule and connect via Public IP, or if you don’t need permanent ssh access to that server, you can simply connect using Softgate as a JumpHost.

+
ssh -o ProxyCommand="ssh -W %h:%p ubuntu@<SoftGate Main IP>" ubuntu@<Server Private IP>
+
+
+../_images/phoenixnap-vnet-ssh-to-server.png +
+
+

Deploy a new server w/ static IP into an existing V-Net

+

Deploying a new server is significantly simpler when using Netris V-NET compared to the traditional process of setting up a server with a public IP.

+

The recommended steps:

+
    +
  1. Request a Server: When ordering a server from phoenixNAP BMC, opt for no public IP address.

  2. +
  3. Select Netris V-Net: In the phoenixNAP web console, select a Netris V-Net (previously created in Netris) as your private network.

  4. +
  5. IP Configuration: In the phoenixNAP web console, enter a valid IP address in the ‘IP Addresses’ text field and specify the Netris V-Net’s gateway IP address in the ‘Default Gateway’ field.

  6. +
+

Netris will automatically include the server into the V-Net.

+../_images/phoenixnap-vnet-import-a-new-server-with-ip.png +

As a result, you will have bare-metal servers in a private subnet that can connect to services outside your VPC (since we have enabled NAT globally in previous chapters), but external services cannot initiate a connection with those servers.

+

In order to connect via SSH to the newly deployed server, you can either create a DNAT rule and connect via Public IP, or if you don’t need permanent ssh access to that server, you can simply connect using Softgate as a JumpHost.

+
ssh -o ProxyCommand="ssh -W %h:%p ubuntu@<SoftGate Main IP>" ubuntu@<Server Private IP>
+
+
+../_images/phoenixnap-vnet-ssh-to-server.png +
+
+

Tags

+

Tags are used to associate bare-metal servers with V-NET dynamically. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers from the Private Network based on that tag. Thus, you can make flexible V-NETs, and there is no need to include every new server in the V-NET.

+../_images/phoenixnap-vnet-with-tag.png +

This feature is even more efficient when you build your infrastructure via Terraform. For example, let’s say you’ve created a V-NET with a tag using Netris Terraform Provider, then order several servers with the same tag using phoenixNAP Terraform Provider. And that’s it, when the servers are ready, Netris will detect them and make them part of the V-NET.

+../_images/phoenixnap-vnet-with-tag-terraform.png +
+
+

Unmanaged

+

Another option is turning the existing private network into Netris V-Net. All private networks from the allowed VLAN IDs range and in the proper location that Netris has not created are visible as “unmanaged” in the V-Net section.

+../_images/phoenixnap-vnet-unmanaged-vnet.png +

The “manage” button will open a dialogue window where it’s also possible to add a default gateway for the appropriate VLAN.

+
+

Warning

+

Once the private network is being converted into V-Net, it will be managed by Netris and no longer manageable through phoenixNAP BMC console.

+
+../_images/phoenixnap-vnet-managed-vnet.png +

Note that you can use Services –> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites)

+
+

Note

+

Netris creates a private network in phoenixNAP BMC based on declared V-Nets. Besides creation, Netris continuously monitors that private networks. As a result of this continuous monitoring, you can’t edit private networks created by Netris from the phoenixNAP BMC console. However, if any modifications are made, Netris will automatically roll everything back to its state.

+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/softgate-software-provisioning.html b/en/latest/tutorials/softgate-software-provisioning.html new file mode 100644 index 0000000000..b4cd6f288d --- /dev/null +++ b/en/latest/tutorials/softgate-software-provisioning.html @@ -0,0 +1,958 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + SoftGate software provisioning — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

SoftGate software provisioning

+
+

SoftGate PRO (100G) software provisioning

+

SoftGate PRO is the high performance version that takes advantage of DPDK and SmartNIC technologies.

+

Check hardware requirements and BIOS configuration here.

+

Install the Netris Agent

+

A requirement for this setup is a newly installed Ubuntu Linux 20.04 LTS, along with internet connectivity through the management port configured using netplan.

+
    +
  1. Navigate to Network → Inventory.

  2. +
+../_images/sg_agent_install.png +

Copy the one-liner displayed in the window and execute it as a regular user on the SoftGate.

+
    +
  1. Upon completing the installation, examine the ifupdown configuration file and ensure that the displayed configuration aligns with what you set up during the OS installation process (the file is generated based on your initial netplan configuration).

  2. +
+
+

Note

+

If the Netris Controller is not in the same OOB network, then add a route to the Netris Controller. No default route or other IP addresses should be configured.”

+
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node.
+    up ip route add <Controller address> via <Management network gateway>
+    gateway <Gateway IP address>
+
+ source /etc/network/interfaces.d/*
+
+
+

If the Netris agent is able to reach the controller, please remove/comment the Gateway line and save the file.

+
+

Note

+

Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent.

+
+
    +
  1. Reboot the SoftGate.

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up you should see its heartbeat going from Critical to OK in Network → Inventory.

+
+
+

SoftGate (1G, 10G) software provisioning

+

Use SoftGate non-pro if you are not using SmartNIC cards. +Check hardware requirements here.

+

Install the Netris Agent

+

A requirement for this setup is a newly installed Ubuntu Linux 22.04 LTS, along with internet connectivity through the management port configured using netplan.

+
    +
  1. Navigate to Network → Inventory.

  2. +
+../_images/sg_agent_install.png +

Copy the one-liner displayed in the window and execute it as a regular user on the SoftGate.

+
    +
  1. Upon completing the installation, examine the ifupdown configuration file and ensure that the displayed configuration aligns with what you set up during the OS installation process (the file is generated based on your initial netplan configuration).

  2. +
+
+

Note

+

If the Netris Controller is not in the same OOB network, then add a route to the Netris Controller. No default route or other IP addresses should be configured.

+
+
user@host:~$ sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# Physical port on SoftGate node connected to a TRUNK port of your network
+auto ens<X>
+iface ens<x> inet static
+    address 0.0.0.0/0
+
+
+# Bond interface
+auto bond0
+iface bond0 inet static
+    address 0.0.0.0/0
+    # Please replace the ensX/Y with the actual interface name(s) below to one(s) present in the OS.
+    bond-slaves ens<X>
+source /etc/network/interfaces.d/*
+
+
+

If the Netris agent is able to reach the controller, please remove/comment the Gateway line and save the file.

+
+

Note

+

Please do not configure any additional IP addresses other than those described in the example above. The further configuration will be performed by the Netris agent.

+
+
    +
  1. Reboot the SoftGate.

  2. +
+
user@host:~$ sudo reboot
+
+
+

Once the server boots up you should see its heartbeat going from Critical to OK in Network → Inventory.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/topology-setup.html b/en/latest/tutorials/topology-setup.html new file mode 100644 index 0000000000..9961f90693 --- /dev/null +++ b/en/latest/tutorials/topology-setup.html @@ -0,0 +1,862 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Topology setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Topology setup

+

The topology manager is used for describing and monitoring the desired network topology. Netris software will configure the underlying network devices according to this topology dynamically and will watch against potential failures. Wire your switches in accordance with the topology view.

+

Right-click on the desired device and choose “Create Link”. Select switch-ports for current and remote nodes.

+../_images/topology.png +../_images/topology_create_link.png +../_images/topology_completed.png +

Once the links have been defined, the network is automatically configured as long as physical connectivity is in place and Netris Agents can communicate with the Netris Controller.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/upgrading-netris.html b/en/latest/tutorials/upgrading-netris.html new file mode 100644 index 0000000000..42ee63f377 --- /dev/null +++ b/en/latest/tutorials/upgrading-netris.html @@ -0,0 +1,1028 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Upgrade and Rollback — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ + +
+

Upgrade and Rollback

+
+

Netris upgrade Procedure

+
+

Backup current database

+

Always have a backup, just in case anything hypothetically goes wrong. SSH to the host running the Netris Controller and execute below command.

+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysqldump -u $MARIADB_USER -p${MARIADB_PASSWORD} $MARIADB_DATABASE' > db-snapshot.sql
+
+
+

Ensure that SQL file db-snapshot.sql is generated and present in the current directory.

+
+

Note

+

An SQL dump is enough for this basic upgrade scenario, however detailed backup & restore procedure is described in here.

+
+
+
+

Stop Netris Agents

+

Stop Netris agents on switches and SoftGate nodes.

+

For Switches:

+

SSH to the switch and run the following command:

+
sudo systemctl stop netris-sw
+
+
+

For SoftGate nodes:

+

SSH to the SoftGate and run the following command:

+
sudo systemctl stop netris-sg
+
+
+

Make sure that all devices in the Network → Inventory section are “red” with the “check_agent” status being “Agent is unavailable”.

+
+

Note

+

A stopped Netris agent has no impact on production traffic through the device.

+
+
+
+

Check your current version

+

Before upgrading the Netris Controller, take a note of the “Netris Version” by navigating to Settings → General in the Controller web interface. The current version number may be used in case of the hypothetical need to perform a rollback procedure.

+Netris Version Example +
+
+

Upgrade the Controller

+

SSH to the Controller host and execute the below command.

+
curl -sfL https://get.netris.io | sh -
+
+
+
+

Note

+

This process can take up to 5 minutes

+
+

Afterwards, make sure that all pods have either “Running” or “Completed” status by executing the following command:

+
kubectl -n netris-controller get pods
+
+
+

The output is similar to this:

+
NAME                                                      READY   STATUS      RESTARTS    AGE
+svclb-netris-controller-haproxy-6tkgj                     4/4     Running     0           38d
+netris-controller-haproxy-bcb944b7c-qcbf8                 1/1     Running     0           13d
+netris-controller-squid-7f6fdc6cf9-7fdx8                  1/1     Running     0           38d
+svclb-netris-controller-squid-58rnp                       1/1     Running     0           38d
+netris-controller-graphite-0                              1/1     Running     0           38d
+netris-controller-mongodb-0                               1/1     Running     0           38d
+netris-controller-redis-master-0                          1/1     Running     0           38d
+netris-controller-smtp-76778cf85f-lw5v5                   1/1     Running     0           10d
+netris-controller-mariadb-0                               1/1     Running     0           10d
+netris-controller-web-session-generator-8b9dbbcd8-8snhd   1/1     Running     0           10d
+netris-controller-telescope-notifier-647975848f-fs5dn     1/1     Running     0           10d
+netris-controller-app-b9b8d8f8d-4ssqb                     1/1     Running     0           10d
+netris-controller-grpc-987669fb9-jjskp                    1/1     Running     0           10d
+netris-controller-telescope-777c98c5d9-mqwl6              1/1     Running     0           10d
+helm-install-netris-controller-lqmq7                      0/1     Completed   0           20h
+
+
+
+

Warning

+

If, after 5 minutes, you see pods with a status other than “Running” or “Completed”, please reach out to us via Slack.

+
+
+
+

Check the upgraded version

+

Make sure that the “Netris Version” reflects the version change by navigating to Settings → General in the Controller web interface.

+
+
+

Upgrade Switches and SoftGate nodes

+

Once you have verified that the Netris controller is up-to-date, it is time to update the switch and SoftGate agents.

+

Upgrade the switch & SoftGate agents by copying the one-liner from the “Install Agent” option of the device’s 3-dot menu found under the Network → Inventory section and pasting it into appropriate devices by SSHing to the corresponding device.

+
+

Note

+

These one-liners include a unique identifier for binding the physical device with the virtual object in the Controller. Please make sure +to copy/paste into the right devices.

+
+Install Agent +

After all the agents have finished the upgrade process, make sure all devices in the Network → Inventory section have a “green” status and the Netris version for each device reflects the version change.

+

In the event the “check_agent” status is “Agent is unavailable” after the agent upgrade has finished, perform agent restart on the affected device(s).

+

For Switches:

+

SSH to the switch and run the following command:

+
sudo systemctl restart netris-sw
+
+
+

For SoftGate nodes:

+

SSH to the SoftGate and run the following command:

+
sudo systemctl restart netris-sg
+
+
+
+
+
+

Rollback Procedure

+

A rollback procedure can be executed in the event the upgrade introduces any adverse impact on the production traffic.

+
+

Stop Netris Agents

+

Stop all Netris agents on the devices managed by the controller (switch & SoftGate).

+

For Switches:

+

SSH to the switch and run the following command:

+
sudo systemctl stop netris-sw
+
+
+

For SoftGate nodes:

+

SSH to the SoftGate and run the following command:

+
sudo systemctl stop netris-sg
+
+
+
+
+

Restore The Database

+

Restore the database from the previously taken snapshot.

+

Drop the current database and create a new one by running the following command after SSHing to the Controller:

+
kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "DROP DATABASE $MARIADB_DATABASE"'
+kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} -e "CREATE DATABASE $MARIADB_DATABASE"'
+
+
+

While still connected to the Controller, copy the backup file from the controller host system to the MariaDB container and restore the database:

+
kubectl -n netris-controller cp db-snapshot.sql netris-controller-mariadb-0:/opt/db-snapshot.sql
+kubectl -n netris-controller exec -it netris-controller-mariadb-0 -- bash -c 'mysql -u root -p${MARIADB_ROOT_PASSWORD} $MARIADB_DATABASE < /opt/db-snapshot.sql'
+
+
+
+
+

Downgrade the Controller Software

+

Downgrade Netris Controller application with the following command.

+
+

Note

+

For the version number, use the number collected from step #3 during the upgrade procedure.

+
+

Example:

+
curl -sfL https://get.netris.io | sh -s -- --ctl-version 3.0.10-031
+
+
+

Afterwards, verify that the version of the “Netris Version” reflects the downgraded version by navigating to Settings → General in the Netris Controller.

+
+
+

Downgrade Netris Agent Software

+

Once you have verified that the Netris controller has been downgraded to the correct version, it is time to downgrade the switch and SoftGate agents.

+

Install the appropriate version of switch & SoftGate agents by copying the one-liner from the “Install Agent” option of the device’s 3-dot menu found under the Network → Inventory section and pasting it into appropriate devices by SSHing to the corresponding device.

+
+

Note

+

One-liners include a unique identifier for binding the physical device with the virtual object in the Controller. Please make sure to copy/paste into the right devices.

+
+

After all the switches and SoftGates have been successfully downgraded, make sure all the devices in the Network → Inventory section have a “green” status and the Netris version for each device reflects the version downgrade.

+

In case the “check_agent” status is “Agent is unavailable” after agent downgrade, perform agent restart.

+

For Switches:

+

SSH to the switch and run the following command:

+
sudo systemctl restart netris-sw
+
+
+

For SoftGate nodes:

+

SSH to the SoftGate and run the following command:

+
sudo systemctl restart netris-sg
+
+
+
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/upgrading-sonic-os.html b/en/latest/tutorials/upgrading-sonic-os.html new file mode 100644 index 0000000000..97dfc3ef3b --- /dev/null +++ b/en/latest/tutorials/upgrading-sonic-os.html @@ -0,0 +1,887 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Upgrading SONiC Operating System — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Upgrading SONiC Operating System
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + +
+
+
+
+ + +
+

Upgrading SONiC Operating System

+
+

Upgrade Procedure

+

The SONiC OS image is being distributed by the Vendor of the switch, you need to obtain one via the support page of the vendor. +The image should be placed on a web server to which switch(es) have access. +In this example we will use a server with an IP address of 10.0.0.36, the image is located in the /var/www/html/ folder, which is the root of the web server.

+../_images/upgrading_sonic_folder_listing.png +
+

Hint

+

To make sure that the image is available as expected, please try to download the image with browser.

+
+

The upgrade process consists of the following steps:

+
    +
  1. Login to the switch and execute

  2. +
+
sudo sonic-installer install -y http://10.0.0.36/Edgecore-SONiC_20220929_052156_ec202012_420.bin
+
+
+

The process will take some time, after installation complete, reboot is mandatory.

+
+

Warning

+

The installation will wipe all the data on the switch, including the admin user password, the authorized keys and ssh identity. After the reboot, a new ssh identity is generated. Because of that, you will be prompted with an identity mismatch message on the first login attempt. Please use your OS-specific procedure to remove the old key.

+
+
    +
  1. Login to the switch and verify that the OS version is updated

  2. +
+
admin@switch15:~$ show version
+
+SONiC Software Version: SONiC.Edgecore-SONiC_20220929_052156_ec202012_420
+Distribution: Debian 10.13
+Kernel: 4.19.0-12-2-amd64
+Build commit: 895d178f6
+Build date: Thu Sep 29 05:49:16 UTC 2022
+Built by: ubuntu@ip-10-5-1-225
+
+Platform: x86_64-accton_as7326_56x-r0
+HwSKU: Accton-AS7326-56X
+ASIC: broadcom
+ASIC Count: 1
+Serial Number: REDACTED
+Uptime: 00:06:00,  1 user,  load average: 1.39, 2.08, 2.29
+
+
+
    +
  1. Perform Netris agent installation step 4 from “Install the Netris Agent” tutorial.

  2. +
+
+
+ + +
+ +
+
+ +
+ +
+

+ © Copyright 2024, Netris. + +

+
+ + + + Built with Sphinx using a + + theme + + provided by Read the Docs. + +
+
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/using-l4-load-balancer.html b/en/latest/tutorials/using-l4-load-balancer.html new file mode 100644 index 0000000000..b2901590b5 --- /dev/null +++ b/en/latest/tutorials/using-l4-load-balancer.html @@ -0,0 +1,860 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Using on-demand (elastic) L4 Load Balancer service — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using on-demand (elastic) L4 Load Balancer service

+

Services –> L4 Load Balancer is an on-demand (elastic) server load balancer. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console.

+../_images/netris-l4-load-balancer.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/using-vnet-in-equinix-metal-project.html b/en/latest/tutorials/using-vnet-in-equinix-metal-project.html new file mode 100644 index 0000000000..9de805fa34 --- /dev/null +++ b/en/latest/tutorials/using-vnet-in-equinix-metal-project.html @@ -0,0 +1,882 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Using V-Net (isolated virtual network) services in Equinix Metal Project — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using V-Net (isolated virtual network) services in Equinix Metal Project

+

V-Net, under Services –> V-Net, is an isolated virtual network service. Netris loads your bare metal server metadata from Equinix Metal Project into the Netris database. So when you create a V-Net service (a virtual network), you list the bare-metal servers you need to get on that virtual network.

+

Netris V-Net (in Equinix Metal scenario) has a global VLAN ID. Per every V-Net, Netris will provision a Layer-2 network using Equinix Metal API. Then Netris will include the listed bare metal servers + SoftGate nodes into that newly created Layer-2 network. SoftGate nodes are the default gateway for the V-Net services.

+

You should create a corresponding V-Net for each virtual network if you use Vmware, KVM, or any other server virtualization platform or VLANs in any way. VLAN ID will be the unique identifier between Netris, Equinix, and your Compute.

+../_images/netris-creating-vnet-for-equinix-metal.png +

In this example, the new V-NET has VLAN ID 2, subnet 10.0.0.0/24, and gateway 10.0.0.1. That means three servers (server-01, server-02, server-03) can launch VMs (or subinterfaces) into a virtual network with VLAN ID 2, and they should use IP addresses from 10.0.0.2-254 pointing to 10.0.0.1 as the default gateway. Netris SoftGate will serve that traffic, and since we have enabled NAT globally in previous chapters, hosts living in VLAN 2 will have Internet access over the NAT.

+../_images/netris-vnet-ready-in-equinix-metal.png +
+

Tags

+

Starting from Netris v3.3.0, it is possible to dynamically associate bare-metal servers with V-NET. For that, set any tag to the V-NET and add the same tag to the metal server(s). Then, Netris will include and exclude metal servers to the Layer-2 network based on that tag. Due to this, you can make flexible V-NETs, and there is no need to include every new server in the V-NET.

+../_images/equinix-metal-vnet-with-tag.png +

This feature is even more efficient when you build your infrastructure via Terraform. For example, let’s say you’ve created a V-NET with a tag using Netris Terraform Provider, then order several metal servers with the same tag using Equinix Metal Terraform Provider. And that’s it, when the servers are ready, Netris will detect them and make them part of the V-NET.

+../_images/equinix-metal-vnet-with-tag-terraform.png +
+
+

Unmanaged

+

Another option is turning the existing Layer-2 network (VLAN) into Netris V-Net. All VLANs in the particular project that aren’t used in other services, like an E-BGP, are visible as “unmanaged” in the V-Net section.

+../_images/unmanaged-vlan-equinix.png +../_images/unmanaged-vnet.png +

The “manage” button will open a dialogue window where it’s also possible to add a default gateway for the appropriate VLAN.

+
+

Warning

+

Once the VLAN is being converted into V-Net, it will be managed by Netris and no longer manageable through Equinix Metal console.

+
+../_images/manage-vnet.gif +

Note that you can use Services –> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-check-default-site.html b/en/latest/tutorials/vpc-anywhere-check-default-site.html new file mode 100644 index 0000000000..4961886325 --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-check-default-site.html @@ -0,0 +1,860 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Check Default Site Settings — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Check Default Site Settings

+

The VLAN ID range (under Net–>Site–>Default) defines what VLAN IDs Netris can use. You may want to ensure that this range does not overlap with other VLAN IDs you may have in your network.

+../_images/vpc-anywhere-check-site-default.png +

The default value is 700-900. Feel free to change to any range your network administrators are comfortable.

+../_images/vpc-anywhere-edit-vlan-range-default-site.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-concept.html b/en/latest/tutorials/vpc-anywhere-concept.html new file mode 100644 index 0000000000..4b5c991e7e --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-concept.html @@ -0,0 +1,868 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + VPC Anywhere Overview — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

VPC Anywhere Overview

+
+

Introduction

+

VPC Anywhere is a solution designed by Netris that allows the integration of VPC network functionality into any network. Netris VPC can be used for all or part of the network traffic. Therefore it is easy to integrate Netris VPC even into existing production Networks without introducing disruptive procedures.

+
+
+

Concept

+

SoftGate is a highly optimized automatic Linux gateway that communicates with Netris VPC controller through an encrypted management link.

+

A SoftGate node (typically two of them for redundancy) connects to the physical network on an 802.1q trunk port (similar to your Vmware or KVM hypervisors).

+

Netris needs to know what VLAN IDs it is allowed to utilize. We recommend reserving a range of VLAN IDs dedicated to Netris VPC. Thus safely isolating VPC traffic flows from all other traffic flows that may exist on the network.

+

SoftGate node becomes the default gateway for the workloads consuming the VPC network. Your existing switch network becomes a transport network for moving 802.1q tagged packets between SoftGate nodes and workload servers.

+../_images/vpc-anywhere-solution-traffic-flows.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-controller-installation.html b/en/latest/tutorials/vpc-anywhere-controller-installation.html new file mode 100644 index 0000000000..510e50302a --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-controller-installation.html @@ -0,0 +1,929 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Install a Netris Controller — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Install a Netris Controller

+
+

Requirements and Installation steps

+

You can install the Netris controller almost on any 64-bit Linux host. Netris Controller may or may not be on the same network as the managed network nodes are. In fact, if there are multiple Netris managed deployments there’s no need for an individual controller for each deployment.

+

It doesn’t matter where to host the Netris controller. What does matter is that the Netris controller needs to be accessible over the Internet. 1) So you can access the web console. 2) Nodes that are going to be managed by Netris need to have access to the Netris controller through their management network interface.

+

Linux Host requirements

+
    +
  • RAM: 8 GB

  • +
  • CPU: 4 Cores

  • +
  • Disk: 50GB

  • +
  • OS: Linux 64-bit

  • +
+

DNS record

+

In my example, my host got a public IP address 54.183.23.201. While it is OK for users and nodes to refer to the Netris Controller through an IP address, we recommend using a DNS record (this way it will be easier to potentially move Netris Controller somewhere with a different IP address).

+

Below is an example using Cloudflare DNS service. (same idea with any DNS software or service)

+../_images/dns-record-netrisctl.png +

Ensure that the newly created domain name indeed resolves into the right IP address of the machine that you are going to install the Netris Controller.

+
host netrisctl.netris.dev
+netrisctl.netris.dev has address 54.183.23.201
+
+
+

Install Netris Controller software and dependencies

+
curl -sfL https://get.netris.io | sh -s -- --ctl-hostname netris.example.com --ctl-ssl-issuer letsencrypt
+
+
+
+

Note

+

Netris Controller installer will stand up a K3S cluster and then will deploy Netris Controller on top of it using Helm Chart. The “–ctl-ssl-issuer” will instruct the installer to generate a Let’s Encrypt SSL certificate and the “–ctl-hostname” will hint for what domain name the certificate must be generated. That’s why it is important to create the DNS record before this step. Detailed info here: doc.

+
+../_images/netris-controller-installed.png +

Once the installation process is finished you will be able to access your newly installed Netris Controller web console using netris/newNet0ps credentials.

+
+
+

Security Matters

+

Change the default password

+

Setting → My Account → Change Password

+../_images/change-password.png +

Add new admin user

+

Accounts → Users → +Add

+../_images/create-new-admin-user.png +

Restrict incoming TCP requests to the list below:

+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + +

TCP Port

Service

22

SSH

80

HTTP

443

Netris Web Console

2003

Streaming Telemetry (Collectd)

3033

Netris Monitoring (Telescope)

50051

Netris Agent (gRPC)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-ipam-setup.html b/en/latest/tutorials/vpc-anywhere-ipam-setup.html new file mode 100644 index 0000000000..c020d680ee --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-ipam-setup.html @@ -0,0 +1,878 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + IPAM Setup for Services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

IPAM Setup for Services

+

To take advantage of the services provided by Netris, it is imperative to define Allocations/Subnets within the Netris IPAM. The Netris Controller comes with predefined subnets that serve the purpose of “common” and these subnets can be used to create V-Nets. However, to leverage services like NAT/L4 (elastic) Load Balancer, it is essential to create new Allocations and Subnets that align with the appropriate purpose. Alternatively, if an existing predefined subnet satisfies the requirements of your network architecture, it is possible to reconfigure it with the desired purpose.

+
+

Create an allocation

+

In addition to the predefined subnets, the Netris Controller also includes predefined allocations, consisting of private IP addresses defined in RFC 1918 - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. If you intend to create subnets that fall outside of these predefined allocations, you must first create allocations that encompass those subnets.

+../_images/vpc-anywhere-ipam-allocation.png +
+
+

Enable on-demand L4 (elastic) Load Balancer

+

Once the allocation is created, you can proceed with creating a subnet with the purpose of “load-balancer”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “load-balancer” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/vpc-anywhere-ipam-l4lb-subnet.png +

The on-demand L4 (elastic) Load Balancer service has been successfully activated and can be accessed through either the web console or Kubernetes using a service of the type load-balancer, or with Terraform.

+

It is worth noting that in the example provided, the Tenant field was left at its default setting of “Admin.” Tenancy is a feature that enables role-based access control and resource delegation. To illustrate, you may want to establish a user role and tenant for your colleagues who are meant to consume Netris VPC services but are not authorized to administer them.

+
+
+

Enable Network Address Translation (NAT)

+

To enable NAT, you need to create a subnet with the purpose of “nat”.

+

Click the + Add button in the top right corner, type a “Prefix” for a new subnet, type a descriptive “Name” for it, select the desired tenant name from the dropdown menu next to the “Tenant” field, select “Subnet” from the “Type” dropdown menu, select “nat” from the “Purpose” dropdown menu, and select the appropriate site from the “Sites” dropdown menu.

+../_images/vpc-anywhere-ipam-nat-subnet.png +

The NAT service has been successfully activated. To create a NAT rule, navigate to the Net → NAT section of the Netris web console. Netris supports most standard SNAT and DNAT rules.

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-softgate-installation.html b/en/latest/tutorials/vpc-anywhere-softgate-installation.html new file mode 100644 index 0000000000..568f710d3b --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-softgate-installation.html @@ -0,0 +1,937 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + SoftGate Installation — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

SoftGate Installation

+
+

Minimum Hardware Requirements

+
    +
  • 8 CPU cores

  • +
  • 16 GB RAM

  • +
  • 300 GB HDD

  • +
+
+
+

SoftGate software provisioning

+

The SoftGate deployment needs a freshly installed Ubuntu Linux 22.04 LTS.

+
+

Note

+

Netris controller ships with two SoftGate nodes pre-defined in the Default site. (softgate1-default, softgate2-default). We recommend using these if you are new to Netris. Alternatively, you can learn how to define new SoftGate nodes here: “Adding SoftGates”.

+
+
    +
  1. Navigate to the Net–>Inventory section and click the three vertical dots (⋮) on the right side of the SoftGate node you are provisioning. Then click Install Agent and copy the one-line installer command to your clipboard.

  2. +
+../_images/softgate-install-agent.png +
    +
  1. Paste the one-line install command on your SoftGate node as an ordinary user. (keep in mind that one-line installer commands are unique for each node)

  2. +
+../_images/softgate-provisioning-cli-output.png +
+

Note

+

Please note that the Netris installation script replaces the default Netplan networking backend with regular ifupdown and attempts to migrate the configuration set during installation to /etc/network/interfaces.

+
+
    +
  1. Handoff Netris the bond0 interface for further automatic operations. Netris will automatically create necessary subinterfaces under your bond0 interface. (bond0.<xyz>). But you need to manually configure which physical interfaces should bind under the bond0 interface. Netris will only make changes to your bond0 and loopback interfaces; all other interfaces will remain as described in /etc/network/interfaces.

  2. +
+
sudo vim /etc/network/interfaces
+
+
+
# The loopback network interface
+auto lo
+iface lo inet loopback
+
+# The management network interface
+auto ensZ
+iface ensZ inet static
+    address <Management IP address/prefix length>
+    # Please delete or comment out the line below if Netris Controller is located in the same network with the SoftGate node, otherwise adjust the line
+    # according to your setup.
+    up ip route add <Controller IP address> via <Management network gateway>
+
+# Physical port on SoftGate node connected to a TRUNK port of your network
+auto ens<X>
+iface ens<x> inet static
+    address 0.0.0.0/0
+
+# Optionally you can add more physical interfaces under your bond0, uncomment as needed
+#auto ens<Y>
+#iface ens<Y> inet static
+#    address 0.0.0.0/0
+
+# Bond interface
+auto bond0
+iface bond0 inet static
+    address 0.0.0.0/0
+    # Please replace/remove the ensX/Y with actual interface name(s) below to one(s) present in the OS.
+    bond-slaves ens<X>
+    # Optional, please adjust the bonding mode below according to the desired functionality.
+    bond-mode active-backup
+
+source /etc/network/interfaces.d/*
+
+
+
+

Note

+

Ensure that the Management network interface IP address is as expected so the SoftGate node will maintain IP connectivity with Netris Controller after reboot.

+
+
    +
  1. Reboot the SoftGate

  2. +
+
sudo reboot
+
+
+

Once the server boots up, you should see its heartbeat going from Critical to OK in Net→Inventory, Telescope→Dashboard, and the SoftGate color will reflect its health in Net→Topology.

+../_images/vpc-anywhere-softgates-green.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-upstream-peering.html b/en/latest/tutorials/vpc-anywhere-upstream-peering.html new file mode 100644 index 0000000000..053e598b54 --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-upstream-peering.html @@ -0,0 +1,889 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Connecting VPC to upstream networks (use one of two options) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Connecting VPC to upstream networks (use one of two options)

+
+

Using a VLAN with public IP addresses (DMZ)

+

Netris VPC can use a traditional subnet or vlan with public IP addresses (a DMZ network) for upstream network peering.

+

logical diagram

+../_images/upstream-dmz-logical.png +

physical diagram

+../_images/upstream-dmz-physical.png +
+
+

BGP Upstream

+
+

BGP Upstream: overview

+

A BGP peering between Netris VPC and upstream routers is the most recommended option, especially for production use.

+

In this scenario, SoftGate nodes form BGP peering with upstream routers. Typically each BGP session will use an individual VLAN id and /30 subnet. Upstream BGP router should advertise the default route 0.0.0.0/0 or full Internet table if necessary. Netris SoftGate nodes are designed to handle over 1M routes in the routing table and can perform as border routers for the full-view table. +SoftGate nodes will automatically advertise public IP subnets in the current site as defined under the Net->IPAM section. Alternatively, you can optionally alter the default settings for full granular control over sent and received prefixes.

+

Logical diagram

+../_images/vpc-anywhere-upstream-bgp-router-logical.png +

Physical diagram

+../_images/vpc-anywhere-upstream-bgp-router-physical.png +
+
+

BGP Upstream: configuration

+

Your local public AS number is a site attribute located under the Net->Sites section. You can use the default private AS number or change to a real AS number depending on local network architecture needs.

+../_images/local-public-asn.png +

BGP peers can be defined under the Net->E-BGP section.

+../_images/add-new-ebgp-form.png +

Check BGP neighbor statuses under the Net->E-BGP

+../_images/bgp-listing.png +

To check/trobleshoot BGP use Net->Looking Glass

+../_images/bgp-looking-glass.png +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-using-l4lb.html b/en/latest/tutorials/vpc-anywhere-using-l4lb.html new file mode 100644 index 0000000000..1c2b88ae86 --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-using-l4lb.html @@ -0,0 +1,861 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using on-demand (elastic) L4 Load Balancer service — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using on-demand (elastic) L4 Load Balancer service

+

Services –> L4 Load Balancer is an on-demand (elastic) L4 Load Balancer service. You can natively use it for Kubernetes, as well as for any TCP/UDP service.

+

Please check our cloud-native tools section of the documentation portal for consuming Load Balancer using Kubernetes native method, Kubernetes CRDs, or Terraform.

+

Meanwhile, below is a screenshot of simply requesting a Load Balancer service through the Netris web console.

+../_images/vpc-anywhere-l4lb.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-using-multi-interface-softgate.html b/en/latest/tutorials/vpc-anywhere-using-multi-interface-softgate.html new file mode 100644 index 0000000000..a250835aab --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-using-multi-interface-softgate.html @@ -0,0 +1,872 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using Multi-interface SoftGate — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using Multi-interface SoftGate

+

By default, Netris uses the bond0 interface of SoftGates exclusively. For each V-NET, Netris creates a new subinterface (e.g., bond0.700) with the next available VLAN ID from the Site’s defined VLAN ID range. However, a proposed solution aims to modify this behavior.

+
+

Creating a V-Net

+

If you want to create a V-Net but prefer Netris to use existing interfaces instead of creating a new subinterface on the bond0 interface, you can achieve that by using the “int=<interface_name>” tag.

+

example - int=eth1

+../_images/vpc-anywhere-vnet-experimental.png +
+ +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-using-nat.html b/en/latest/tutorials/vpc-anywhere-using-nat.html new file mode 100644 index 0000000000..bc79ff49f9 --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-using-nat.html @@ -0,0 +1,876 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using NAT services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using NAT services

+

NAT, under Net –> NAT, is a Network Address Translation (NAT) service. Netris supports most of the standard rules for MASQUERADE, SNAT, and DNAT. In this scenario, I am enabling instances in a private network with a subnet of 172.24.0.0/16 to have access to services outside their VPC. The goal is to provide instances with the capability to connect to the Internet through NAT for outbound access.

+
+

MASQUERADE

+

The MASQUERADE is ideal for situations where you want to allow your instances to access the Internet for essential activities like installing/updating software packages, downloading files, and other similar tasks. The MASQUERADE doesn’t require having a subnet with the purpose of NAT. Instead, it leverages the main IP address of the active softgate to perform network address translation (NAT).

+

The Netris Controller includes a preconfigured MASQUERADE rule that facilitates instances within a private network with a subnet of 172.24.0.0/16 to have immediate access to services outside their VPC environment.

+../_images/vpc-anywhere-nat-masquerade.png +
+
+

SNAT

+

Unlike MASQUERADE, the SNAT requires having a dedicated subnet with a NAT purpose. SNAT replaces the source IP address of the instances with the IP address of the “SNAT to IP”. Thus, you can be sure that instances originating from certain source IP addresses are consistently translated through the designated IP address. Therefore, SNAT is well-suited for scenarios where your production traffic flow requires the use of dedicated IP addresses for outbound connectivity.

+../_images/vpc-anywhere-nat-snat.png +

You can always have more granular control either through NAT rule or using Services → ACLs.

+
+
+

DNAT

+

A DNAT (Destination Network Address Translation) allows incoming traffic to be redirected from a destination IP address and port to a “DNAT to IP/Port”. This type of rule is often used in scenarios where you want to forward incoming traffic to a specific server within your network, such as a web server or database.

+

I’m creating a DNAT rule for the ssh port in the example below. It forwards the public IP’s 55022 port to the local IP’s 22 port. Once the rule is applied, you can easily establish a remote SSH connection to the server.

+../_images/vpc-anywhere-nat-dnat.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere-using-vnet.html b/en/latest/tutorials/vpc-anywhere-using-vnet.html new file mode 100644 index 0000000000..7983bbef23 --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere-using-vnet.html @@ -0,0 +1,865 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Using V-Net (isolated virtual network) services — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Using V-Net (isolated virtual network) services

+

V-Net, under Services –> The V-Net service is a virtual network that operates in an isolated environment. In Netris, each V-Net takes the next available VLAN ID from the Site’s defined VLAN ID range and uses it as a global VLAN ID. The SoftGate nodes serve as the default gateway for the V-Net services.

+
+

Creating a V-Net

+

To ensure proper connectivity, it is recommended to create a corresponding V-Net for each virtual network used in Vmware, KVM, or any other server virtualization platform. The same applies to an isolated group of physical servers. VLAN ID serves as the unique identifier between Netris and the computing platform used.

+../_images/vpc-anywhere-vnet.png +

In this example, the new V-NET is assigned VLAN ID 700, subnet 172.24.0.0/20, and gateway 172.24.0.1. As a result, on your Compute side, you can launch VMs or subinterfaces on physical servers with a VLAN ID of 700 and obtain all necessary IP configurations from DHCP. Alternatively, if you prefer not to use DHCP, instances should use IP addresses from the 172.24.0.0/20 subnet and set the default gateway to 172.24.0.1. Netris SoftGate will manage this traffic, and the preconfigured MASQUERADE rule for the 172.24.0.0/16 subnet included in the Netris Controller will enable hosts in VLAN 700 to have internet access through NAT.

+

Note that you can use Services –> ACLs for granular control over traffic between multiple V-NETs as well as to/from outside (Internet or other remote sites)

+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-anywhere.html b/en/latest/tutorials/vpc-anywhere.html new file mode 100644 index 0000000000..928940579a --- /dev/null +++ b/en/latest/tutorials/vpc-anywhere.html @@ -0,0 +1,911 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + VPC Anywhere Getting Started Guide — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + + + +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-gateways-with-managed-fabric.html b/en/latest/tutorials/vpc-gateways-with-managed-fabric.html new file mode 100644 index 0000000000..e88ed62d7b --- /dev/null +++ b/en/latest/tutorials/vpc-gateways-with-managed-fabric.html @@ -0,0 +1,936 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + Getting Started with Switch-Fabric Manager & VPC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Getting Started with Switch-Fabric Manager & VPC
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+ + +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/tutorials/vpc-setup.html b/en/latest/tutorials/vpc-setup.html new file mode 100644 index 0000000000..3d643f4638 --- /dev/null +++ b/en/latest/tutorials/vpc-setup.html @@ -0,0 +1,861 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + VPC setup — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

VPC setup

+

The Netris VPC offers you the ability to operate your resources within a logically segregated virtual network. You can create, edit, and remove VPCs as needed. The VPC acts as a VRF in traditional networking, providing the ability to employ overlapping IP ranges across various VPCs while maintaining secure management and operation of resources.

+

Netris Controller is preconfigured with a default system VPC-1. Use the default VPC, and create additional VPCs as needed in the future.

+../_images/vpc_concept.png +

To create VPC go to Network → VPC → +Add

+../_images/vpc_add.png +
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/visibility.html b/en/latest/visibility.html new file mode 100644 index 0000000000..6cab72012d --- /dev/null +++ b/en/latest/visibility.html @@ -0,0 +1,912 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Visibility (Telescope) — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ +
    + +
  • »
  • + +
  • Visibility (Telescope)
  • + + +
  • + + + + Edit on GitHub + + + +
  • + +
+ + + + +
+
+
+
+ +
+

Visibility (Telescope)

+
+

Graph Boards

+

You can create custom graph boards with data sources available in different parts of the system. You can even sum multiple graphs and visualize them in a single view.

+

To start with Graph Boards, first, you need to add a new Graph Board.

+
    +
  1. Navigate to Telescope → Graph Boards, open the dropdown menu in the top left corner, then click + button.

  2. +
+_images/add_board.png +
    +
  1. Type a name and assign it to one of the tenants that you manage. Later on, you can optionally mark the Graph Board as public if you want the particular board to be visible to all users across multiple tenants.

  2. +
+_images/create_board.png +

Now you can add graphs by clicking +Add graph.

+

Description of +Add graph fields:

+
    +
  • Title - Title for the new graph.

  • +
  • Type - Type of data source.

    +
      +
    • Bps - Traffic bits per second.

    • +
    • Pps - Traffic packets per second.

    • +
    • Errors - Errors per second.

    • +
    • Optical - Optical signal statistics/history.

    • +
    • MAC Count - History of the number of MAC addresses on the port.

    • +
    +
  • +
  • Function - Currently, only summing is supported.

  • +
  • +Member - Add data sources by service (E-BGP, V-NET, etc..) or by Switch Port.

  • +
+

Example: Sum of traffic on two ISP uplinks.

+_images/add_graph.png +

Example: Sum of the traffic on all ports under the service called “PC06 V-NET”.

+_images/add_graph2.png +

Screenshot: Listing of a Graph Board with the explanation of the controls.

+_images/graph_board.png +
+
+

API Logs

+

Comprehensive logging of all API calls sent to Netris Controller with the ability to search by various attributes, sort by each column, and filter by method type.

+
+
+

Dashboard

+

Netris, besides automatic configuration, also provides automatic monitoring of the entire network without the need for configuration of the monitoring systems.

+

Telescope → Dashboard summarizes Network Health, which can also be accessed by clicking on the Netris icon in the top left corner.

+

Description of the pie charts.

+
    +
  • Hardware Health - summary of CPU, RAM, disk utilization. Statuses of power supplies, fans, temperature sensors, critical system services, and time synchronization. Statuses of switch port link, utilization, optical signal levels, and BGP sessions.

  • +
  • E-BGP - Statuses of external BGP sessions.

  • +
  • LB VIP - Statuses of Load Balancer frontend / VIP availability.

  • +
  • LB Members - Statuses of Load Balancer backend members.

  • +
+

By clicking on each title you can see the details of the checks on the right side.

+

Screenshot: Dashboard showing details of “Hardware Health.”

+_images/hardware_health.png +

Port up/down state can be set to “Save as normal.” So the system will alarm only if the actual state is different from the saved as the normal state.

+

Screenshot: “Save as normal” on selected ports.

+_images/saveasnormal.png +
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/vnet.html b/en/latest/vnet.html new file mode 100644 index 0000000000..1022ebb4b4 --- /dev/null +++ b/en/latest/vnet.html @@ -0,0 +1,888 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + V-Net — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

V-Net

+

V-Net is a virtual networking service that provides a Layer-2 (unrouted) or Layer-3 (routed) virtual network segments on switch ports anywhere on the switch fabric. V-NETs can be created and managed by a single tenant (single team) or they can be created and managed collaboratively by multiple tenants (different teams inside and/or outside the organization). +Netris automatically configures a VXLAN with an EVPN control plane over an unnumbered BGP Layer-3 underlay network and organize the high availability for the default gateway behind the scenes.

+
+

V-Net Fields

+
    +
  • Name - Unique name for the V-Net

  • +
  • Owner - Tenant, who can make any changes to current V-Net

  • +
  • V-Net state - Active/Disable state for entire V-Net

  • +
  • VLAN aware - Enable VLAN aware bridge, use only in rare cases, if otherwise is not possible

  • +
  • Guest tenants - List of tenants allowed to add/edit/remove ports to the V-Net but not manage other parameters

  • +
  • Sites - Ports from these sites will be allowed to participate in the V-Net. (Multi-site circuits would require backbone connectivity between sites).

  • +
  • IPv4 Gateway - IPv4 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site where V-Net is intended to span.

  • +
  • IPv6 Gateway - IPv6 address to be used as a default gateway in this V-Net. Should be configured under Net→IPAM as an assignment, assigned to the owner tenant, and available in the site or sites where V-Net is intended to span.

  • +
  • Port - Physical Switch Port anywhere on the network. Switch Port should be assigned to the owner or guest tenant under Net→Switch Ports.

    +
      +
    • Enabled - Enable or disable individual Switch Port under current V-Net

    • +
    • Port Name - Switch Port format: <alias>(swp<number>)@<switch name>

    • +
    • VLAN ID / Untag - Specify a VLAN ID for tagging traffic on a per-port basis or set Untag not to use tagging on a particular port. VLAN tags are only significant on each port’s ingress/egress unless VLAN aware mode is used.

    • +
    • LAG Mode - Allows for active-standby dual-homing, assuming LAG configuration on the remote end. Active/active dual homing will be enabled in future releases (dependence on SVI support by NOSes).

    • +
    +
  • +
+
+

Tip

+

Many switches can’t autodetect old 1Gbps ports. If attaching hosts with 1Gbps ports to 10Gbps switch ports, you’ll need to change the speed for a given Switch Port from Auto(default) to 1Gbps. You can edit a port in Net→Switch Ports individually or in bulk.

+
+_images/add-vnet.png +

+Example: Adding a new V-Net.

_images/list-vnet.png +

+Example: Listing of V-Nets.

_images/list-vnet-expanded.png +

+Example: Expanded view of a V-Net listing.

+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/en/latest/vpc.html b/en/latest/vpc.html new file mode 100644 index 0000000000..b0c9082733 --- /dev/null +++ b/en/latest/vpc.html @@ -0,0 +1,909 @@ +
+ + + + + + + + slack_icon + Join Slack + + + + +
+ + + + + + + + + + Netris VPC — Netris Documentation + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + +
+ + + + + +
+ +
+ + + + + + + + + + + + + + + + + + + +
+ + + + + + +
+
+
+
+ +
+

Netris VPC

+

The Netris VPC offers you the ability to operate your resources within a logically segregated virtual network. You can create, edit, and remove VPCs as needed. The VPC acts as a VRF in traditional networking, providing the flexibility to employ overlapping IP ranges across various VPCs while maintaining secure management and operation of resources.

+

Netris Controller is preconfigured with a default system VPC-1. Use the default VPC, and create additional VPCs as needed in the future.

+

The following diagram shows a VPC concept in the Netris Controller.

+VPC diagram +

VPC is the highest entity in the hierarchy and it spreads over all Sites. +Take a look at the VPC features and services.

+
+

Sites

+

For each individual deployment/region, you should define a Site. All network components and resources should be associated with their respective Site and VPC.

+

Physically connected sites function like an availability zone. This means that two V-Nets (Subnets) will communicate using the direct link, and even a single V-Net can span within an availability zone. Sites that are not physically connected function like regions. And Netris SiteMesh, a Wireguard-based site-to-site VPN, can be used to enable communication between these regions.

+
+
+

IPAM

+

You have the ability to create IP Allocations and Subnet assignments for a VPC, and these may overlap between different VPCs. A Subnet can be assigned to multiple sites if you aim to extend your V-Net across various locations.

+
+
+

V-Nets

+

V-Net is a virtual networking service that provides Layer-2 (unrouted) or Layer-3 (routed) virtual network segments in VPC. V-Net is assigned to one VPC and one or multiple sites. Your endpoints (servers, VMs) are connected to V-Nets.

+

Note: multisite feature requires a direct physical link between Sites.

+
+
+

External connections

+

You can connect your VPC to ISP providers or other segments of your network using Netris E-BGP service, or statically by defining a V-Net and using Net->Routes (for natively integrated Bare Metal Cloud Providers please refer to the provider-specific tutorial, as external connections usually establish automatically

+
+
+

SiteMesh connections

+

Connect your VPCs over an automatic Wireguard-based Site-to-Site VPN across multiple Sites if you don’t have a direct link between those Sites.

+

Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).

+
+
+

NAT services

+

SNAT allows your endpoints to communicate with the Internet. DNAT is also available.

+

Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).

+
+
+

Load-balancing service

+

Use L4LB service to share the load between your endpoints.

+

Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).

+
+
+

Access lists

+

ACLs provide a layer of security that acts as a firewall for controlling traffic in and out of one or more subnets.

+

Note: works only in the system default VPC (limitation is planned to be lifted in Netris v. 4.1.0).

+
+
+

Working with Netris VPC

+

You have the flexibility to create and manage your VPCs using any of the interfaces listed below:

+
    +
  • Web interface of the Netris Controller

  • +
  • Terraform Netris provider

  • +
  • Kubernetes Integration

  • +
  • REST API

  • +
+
+
+ + +
+ +
+ +
+
+ +
+ +
+ + +
+ + Alternate Versions + v: latest + + +
+ + +
+
Versions
+ + +
2.9
+ + + +
3.0
+ + + +
3.4
+ + + +
latest
+
+ +
+ + +
+
Downloads
+ +
pdf
+ +
epub
+ +
+ + +
+ Free theme by Read the Docs. + +
+
+ + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/index.html b/index.html new file mode 100644 index 0000000000..d184e0ff16 --- /dev/null +++ b/index.html @@ -0,0 +1,10 @@ + + + + Netris Docs + + + +

Please wait while you're redirected to our netris documentation page.

+ +